youtubebeat/vendor/github.com/elastic/beats/filebeat/docs/migration.asciidoc

[[migrating-from-logstash-forwarder]]
= Migrating from Logstash Forwarder to {beatname_uc}

[partintro]
--
{beatname_uc} is based on the Logstash Forwarder source code and replaces Logstash Forwarder as the method
to use for tailing log files and forwarding them to Logstash.

{beatname_uc} introduces the following major changes:

* The config file was restructured and converted from JSON to YAML.
* The registry file, which stores the state of the currently read files, was
  changed.
* Command line options were removed and moved to the configuration file.
* Configuration options for outputs are now inherited from libbeat. For details, see the {libbeat}/index.html[Beats Platform Reference].
* The {logstash-ref}/plugins-inputs-beats.html[Beats input plugin for Logstash] is required.

The following topics describe how to migrate from
https://github.com/elastic/logstash-forwarder[Logstash Forwarder] to {beatname_uc}:

* <<migration-input-plugin>>
* <<migration-registry-file>>
* <<migration-configuration>>
* <<migration-changed-cli>>
* <<migration-changed-fields>>
* <<migration-other-changes>>
--

[[migration-input-plugin]]
== Migrate to the Beats input plugin for Logstash

{beatname_uc} requires the {logstash-ref}/plugins-inputs-beats.html[Beats input
plugin for Logstash]. For information about getting started with this plugin,
see {stack-gs}/get-started-elastic-stack.html#logstash-setup[Configure Logstash to
listen for Beats input] in the {stack} getting started tutorial.

In both the 1.5.x and 2.x versions of Logstash, this plugin can be loaded in
parallel with the
https://github.com/logstash-plugins/logstash-input-lumberjack[Lumberjack]
plugin used by the Logstash Forwarder.

If you have a large number of servers that you want to migrate from
Logstash Forwarder to {beatname_uc}, we recommend that you keep the Lumberjack plugin and load the
Beats input plugin on the same Logstash instances, but set up the Beats input plugin to use a different port. After you have migrated
all the machines to {beatname_uc}, you can remove the Lumberjack plugin.

We realize that opening additional ports may not be feasible in your organization. Another option for phased migration
to {beatname_uc} is to ship data from Logstash Forwarder directly to the Beats input plugin.

IMPORTANT: This data shipping path is only supported for migrating to {beatname_uc} and will no longer be supported when Logstash Forwarder reaches https://www.elastic.co/support/eol[End of Life].

What's required?

* The https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash]
version 2.2.8 or later.
* SSL must be explicitly enabled in the Beats input plugin (`ssl => true`) because SSL is on by default with Logstash Forwarder. The SSL/TLS configs should be the same for both the Logstash Forwarder and {beatname_uc} instances.

[[migration-registry-file]]
==  Update the registry file

The registry file stores the state and location information that {beatname_uc} uses to track
where it was last reading. Under Logstash Forwarder, this file was called `.logstash-forwarder`. For {beatname_uc},
the file was renamed. The name varies depending on the package type:

 * `data/registry` for `.tar.gz` and `.tgz` archives
 * +/var/lib/{beatname_lc}/registry+ for DEB and RPM packages
 * +c:\ProgramData{backslash}{beatname_lc}{backslash}registry+ for the Windows zip file

For enhancement reasons, especially for Windows,
the structure of the registry file has changed. This makes migrating the file
complex and leads to potential errors.

Instead of migrating the registry file, we recommend that you start {beatname_uc} on
the same host where Logstash Forwarder is running, and send the log files to a
different index.  This will start indexing from scratch. If you want to start
reading at the end of all files, you can set the `tail_files` option in the
{beatname_uc} configuration file to true.

Using this approach allows you to keep the old Logstash Forwarder running and then
slowly migrate over to {beatname_uc}.

[[migration-configuration]]
== Migrate your configuration

Although {beatname_uc} is based on Logstash Forwarder, {beatname_uc} uses YAML for its configuration
file, rather than the JSON+comments language used by Logstash Forwarder. This means that you
will need to migrate your existing configuration files to use the YAML syntax. {beatname_uc} has a main
configuration file called +{beatname_lc}.yml+, but {beatname_uc} also accepts reading
multiple configuration files from a `conf.d` directory and has similar restrictions to Logstash Forwarder.
If you specify additional config files, you need to place them in a directory other than the directory
where the main {beatname_uc} config file resides. You specify the location of the config files by using the
`config_dir` option to configure the path to the directory. In most cases, you can do a one-to-one
conversion to create a {beatname_uc} config file for each Logstash Forwarder config file.

Before migrating your config files, we recommend that you first read the <<configuring-howto-{beatname_lc}>>
section to understand the {beatname_uc} options.

[float]
=== Migrate the "files" section

To migrate the `files` section from the Logstash Forwarder configuration, create an `inputs` section in the {beatname_uc} config file. For example, assuming that you start
with this configuration in Logstash Forwarder:

[source,json]
-------------------------------------------------------------------------------------
  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": {
        "type": "syslog",
        "service": "apache",
        "zone": "us-east-1"
      }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]
-------------------------------------------------------------------------------------

The equivalent `inputs` section would look like this:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
{beatname_lc}.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
  fields:
    service: apache
    zone: us-east-1
  fields_under_root: true
- type: stdin <1>
- type: log
  paths:
    - /var/log/apache2/httpd-*.log
-------------------------------------------------------------------------------------

<1> The explicit `type` option was introduced to differentiate between normal files and
    stdin. In the future, additional types might be supported.

As you can see, apart from the new `type` options,
which were before implicitly defined via the `type` custom field, the remaining
options can be migrated mechanically.

The {beatname_uc} configuration gives you more control over how each input behaves
by allowing you to configure options that were previously global in Logstash Forwarder
and set them separately for each input. See <<configuring-howto-{beatname_lc}>>.

[float]
=== Migrate the "network" section

Like Logstash Forwarder, {beatname_uc} can communicate directly with Logstash.
{beatname_uc} can also insert log entries directly
into Elasticsearch. This results in an `output` section that is a bit more complex, as
you can see in the following example. You'll find, however, that you can easily
translate the Logstash part of the configuration from the equivalent Logstash Forwarder
configuration.

The following snippet shows the `network` section of the Logstash Forwarder configuration:

[source,json]
-------------------------------------------------------------------------------------
  # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }
-------------------------------------------------------------------------------------

The equivalent in {beatname_uc} would look like this:


[source,yaml]
-------------------------------------------------------------------------------------
output.logstash:
  hosts: <1>
    - localhost:5043
  timeout: 15
  ssl.certificate_authorities: <2>
    - ./logstash-forwarder.crt
  ssl.certificate: ./logstash-forwarder.crt
  ssl.key: ./logstash-forwarder.key
-------------------------------------------------------------------------------------

<1> When multiple hosts are defined, the default behavior in {beatname_uc} is to
    pick a random host for new connections, similar to the Logstash Forwarder
    behavior. {beatname_uc} can optionally do load balancing. For more details, see the
    <<loadbalance>> configuration option.
<2> Note that if the `ssl` settings are missing, then SSL is disabled. SSL is
    automatically enabled when you add any of the `ssl` options. For more information about
    specific configuration options, see <<configuration-ssl>>.


[[changed-configuration-options]]
[float]
=== Changed configuration file options

With the refactoring of the configuration file, the following options were removed or renamed:

[cols="2*", options="header"]
|===
|Config Option
|Action

|`deadTime`
|`deadTime` was renamed to `ignore_older`. {beatname_uc} keeps the files that it’s reading open until they are older than the timespan specified by `ignore_older`. If a file is changed, {beatname_uc} reopens it.

|`netTimeout`
|`netTimeout` was removed and is replaced by the `timeout` option in libbeat.

|`log-to-syslog` and `syslog`
|Both options were removed and replaced by logging options in libbeat.
|===

For more information about these options, see <<configuring-howto-{beatname_lc}>>.

[float]
=== A complete example

Let's see a simple, but complete example of a Logstash Forwarder configuration
and its equivalent for {beatname_uc}.

Logstash Forwarder configuration:

[source,json]
-------------------------------------------------------------------------------------
{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": {
        "type": "syslog",
        "service": "test01"
      }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}
-------------------------------------------------------------------------------------

{beatname_uc} configuration:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
{beatname_lc}.inputs:
- type: log
  paths:
    - /var/log/*.log
  fields:
    service: test01
output.elasticsearch:
  hosts: ["http://localhost:5043"]
-------------------------------------------------------------------------------------

[[migration-changed-cli]]
== Changes to command line options

Most command line options available in Logstash Forwarder have been removed and
migrated to config file options. The only mandatory command line option for
running  {beatname_uc} is  `-c` followed by the path to the config file. If you used command line
options with Logstash Forwarder, make sure that you add your options to the
configuration file. For naming changes, see <<renamed-options>>.

{beatname_uc} does provide command line options that are common to all Beats. For more details about
these options, see <<command-line-options>>.

[[renamed-options]]
[float]
=== Renamed options

The following command line options have been renamed and moved to the config file. Also see
<<changed-configuration-options>> for a list of configuration file options that were completely removed or
replaced by options specified in libbeat.

[cols="3*", options="header"]
|===
|Command Line Option
|Config File Option
|Description

|`-config`
|`-c` command line option and `config_dir`
|The config option was split into two parts. You use the `-c` command line option to specify the location of the base (required) config file when you start {beatname_uc}. To use additional config files, you specify the `config_dir` configuration option.

The `config_dir` option specifies the path to the directory that contains additional configuration files. This option MUST point to a directory other than the directory where the main {beatname_uc} config file resides.

|`-idle-timeout`
|
|`idle_timeout` was removed. Libbeat is used for publishing logs.

|`-spool-size`
|`queue.mem.events`
|`spool_size` was moved to the config file and removed as a flag.

|`-harvester-buff-size`
|`harvester_buffer_size`
|`harvester_buffer_size` was moved to the config file and removed as a flag. You can now configure the buffer size separately for each harvester.

|`-tail`
|`tail_files`
|`tail_files` was moved to the config file and removed as a flag. You can now configure this option separately for each input.

|`-cpuProfileFile`
|
|`cpuProfileFile` was removed. You can use the profiling options of libbeat instead. For more details on profiling, see https://github.com/elastic/libbeat/issues/122.

|`-quiet`
|
|`quiet` was removed. Libbeat is now used for logging, so you must use the libbeat <<configuration-logging, logging options>> instead.


|===

[[migration-changed-fields]]
== Changes to the output fields

In the default configuration, {beatname_uc} structures its output documents a little
differently from the Logstash Forwarder. This section discusses the differences
and the options you have in case you want compatibility with the Logstash
Forwarder.

[float]
=== Custom fields are grouped under a "fields" dictionary

The custom fields (added from the configuration file) are set as top-level
fields in Logstash Forwarder but are grouped together under a `fields`
dictionary in {beatname_uc}. If you need the old behavior during the migration phase,
you can use the <<fields-under-root-log>> configuration option:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
{beatname_lc}.inputs:
- type: log
  paths:
    - /var/log/*.log
  fields:
    service: test01
  fields_under_root: true
-------------------------------------------------------------------------------------

[float]
=== {beatname_uc} uses "beat.hostname" for sending the hostname of the server

While the Logstash Forwarder sends the hostname of the server it's running on in
the `host` field, {beatname_uc} uses the `beat.hostname` field for the same purpose.
Because `host` is commonly used in the Logstash plugin ecosystem, the Beats
input plugin automatically copies `beat.hostname` into `host`.

[float]
=== The "file" field was renamed to "source"

The `file` field was renamed to `source`. If you rely on this field being
named `file`, you can rename it by using the mutate filter in Logstash. For
example:

[source,plain]
-------------------------------------------------------------------------------------
filter {
    mutate {
        rename => {
            "source" => "file"
        }
    }
}
-------------------------------------------------------------------------------------

[float]
=== The "line" field was removed

The `line` field was removed. This field wasn't correct after restarts, and
making it correct would have resulted in a performance penalty. We recommend
using the `offset` field instead.

[[migration-other-changes]]
== Other changes

The following list of implementation changes should not affect your experience migrating
from Logstash Forwarder, but you should be aware of the changes. Please post GitHub issues
if you notice any regressions from Logstash Forwarder.

[float]
=== Packaging

The packaging process for {beatname_uc} uses the Beats infrastructure, so some
aspects of packaging, such as the init scripts, are different from
Logstash Forwarder. Please post GitHub issues if you hit any issues
with the new packages.

One notable change is the name of the registry file. The name varies depending on the package
type:

 * `registry` for `.tar.gz` and `.tgz` archives
 * +/usr/lib/{beatname_lc}/registry+ for DEB and RPM packages
 * +c:\ProgramData{backslash}{beatname_lc}{backslash}registry+ for the Windows zip file

[float]
=== Publisher improvements

Behind the scenes, {beatname_uc} uses a slightly improved protocol for communicating
with Logstash.

[float]
=== SSL is off by default

If you follow the section on migrating the configuration, you will have SSL
enabled. However, you must be aware that if the `ssl` section is missing from the
configuration file, {beatname_uc} uses an unencrypted connection to talk to Logstash.

[float]
=== Logging

{beatname_uc} uses libbeat logging and can also log to rotating files instead of syslog.