[[migrating-from-logstash-forwarder]] = Migrating from Logstash Forwarder to {beatname_uc} [partintro] -- {beatname_uc} is based on the Logstash Forwarder source code and replaces Logstash Forwarder as the method to use for tailing log files and forwarding them to Logstash. {beatname_uc} introduces the following major changes: * The config file was restructured and converted from JSON to YAML. * The registry file, which stores the state of the currently read files, was changed. * Command line options were removed and moved to the configuration file. * Configuration options for outputs are now inherited from libbeat. For details, see the {libbeat}/index.html[Beats Platform Reference]. * The {logstash-ref}/plugins-inputs-beats.html[Beats input plugin for Logstash] is required. The following topics describe how to migrate from https://github.com/elastic/logstash-forwarder[Logstash Forwarder] to {beatname_uc}: * <> * <> * <> * <> * <> * <> -- [[migration-input-plugin]] == Migrate to the Beats input plugin for Logstash {beatname_uc} requires the {logstash-ref}/plugins-inputs-beats.html[Beats input plugin for Logstash]. For information about getting started with this plugin, see {stack-gs}/get-started-elastic-stack.html#logstash-setup[Configure Logstash to listen for Beats input] in the {stack} getting started tutorial. In both the 1.5.x and 2.x versions of Logstash, this plugin can be loaded in parallel with the https://github.com/logstash-plugins/logstash-input-lumberjack[Lumberjack] plugin used by the Logstash Forwarder. If you have a large number of servers that you want to migrate from Logstash Forwarder to {beatname_uc}, we recommend that you keep the Lumberjack plugin and load the Beats input plugin on the same Logstash instances, but set up the Beats input plugin to use a different port. After you have migrated all the machines to {beatname_uc}, you can remove the Lumberjack plugin. We realize that opening additional ports may not be feasible in your organization. Another option for phased migration to {beatname_uc} is to ship data from Logstash Forwarder directly to the Beats input plugin. IMPORTANT: This data shipping path is only supported for migrating to {beatname_uc} and will no longer be supported when Logstash Forwarder reaches https://www.elastic.co/support/eol[End of Life]. What's required? * The https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash] version 2.2.8 or later. * SSL must be explicitly enabled in the Beats input plugin (`ssl => true`) because SSL is on by default with Logstash Forwarder. The SSL/TLS configs should be the same for both the Logstash Forwarder and {beatname_uc} instances. [[migration-registry-file]] == Update the registry file The registry file stores the state and location information that {beatname_uc} uses to track where it was last reading. Under Logstash Forwarder, this file was called `.logstash-forwarder`. For {beatname_uc}, the file was renamed. The name varies depending on the package type: * `data/registry` for `.tar.gz` and `.tgz` archives * +/var/lib/{beatname_lc}/registry+ for DEB and RPM packages * +c:\ProgramData{backslash}{beatname_lc}{backslash}registry+ for the Windows zip file For enhancement reasons, especially for Windows, the structure of the registry file has changed. This makes migrating the file complex and leads to potential errors. Instead of migrating the registry file, we recommend that you start {beatname_uc} on the same host where Logstash Forwarder is running, and send the log files to a different index. This will start indexing from scratch. If you want to start reading at the end of all files, you can set the `tail_files` option in the {beatname_uc} configuration file to true. Using this approach allows you to keep the old Logstash Forwarder running and then slowly migrate over to {beatname_uc}. [[migration-configuration]] == Migrate your configuration Although {beatname_uc} is based on Logstash Forwarder, {beatname_uc} uses YAML for its configuration file, rather than the JSON+comments language used by Logstash Forwarder. This means that you will need to migrate your existing configuration files to use the YAML syntax. {beatname_uc} has a main configuration file called +{beatname_lc}.yml+, but {beatname_uc} also accepts reading multiple configuration files from a `conf.d` directory and has similar restrictions to Logstash Forwarder. If you specify additional config files, you need to place them in a directory other than the directory where the main {beatname_uc} config file resides. You specify the location of the config files by using the `config_dir` option to configure the path to the directory. In most cases, you can do a one-to-one conversion to create a {beatname_uc} config file for each Logstash Forwarder config file. Before migrating your config files, we recommend that you first read the <> section to understand the {beatname_uc} options. [float] === Migrate the "files" section To migrate the `files` section from the Logstash Forwarder configuration, create an `inputs` section in the {beatname_uc} config file. For example, assuming that you start with this configuration in Logstash Forwarder: [source,json] ------------------------------------------------------------------------------------- # The list of files configurations "files": [ # An array of hashes. Each hash tells what paths to watch and # what fields to annotate on events from those paths. { "paths": [ "/var/log/messages", "/var/log/*.log" ], # A dictionary of fields to annotate on each event. "fields": { "type": "syslog", "service": "apache", "zone": "us-east-1" } }, { # A path of "-" means stdin. "paths": [ "-" ], "fields": { "type": "stdin" } }, { "paths": [ "/var/log/apache/httpd-*.log" ], "fields": { "type": "apache" } } ] ------------------------------------------------------------------------------------- The equivalent `inputs` section would look like this: ["source","yaml",subs="attributes"] ------------------------------------------------------------------------------------- {beatname_lc}.inputs: - type: log paths: - /var/log/messages - /var/log/*.log fields: service: apache zone: us-east-1 fields_under_root: true - type: stdin <1> - type: log paths: - /var/log/apache2/httpd-*.log ------------------------------------------------------------------------------------- <1> The explicit `type` option was introduced to differentiate between normal files and stdin. In the future, additional types might be supported. As you can see, apart from the new `type` options, which were before implicitly defined via the `type` custom field, the remaining options can be migrated mechanically. The {beatname_uc} configuration gives you more control over how each input behaves by allowing you to configure options that were previously global in Logstash Forwarder and set them separately for each input. See <>. [float] === Migrate the "network" section Like Logstash Forwarder, {beatname_uc} can communicate directly with Logstash. {beatname_uc} can also insert log entries directly into Elasticsearch. This results in an `output` section that is a bit more complex, as you can see in the following example. You'll find, however, that you can easily translate the Logstash part of the configuration from the equivalent Logstash Forwarder configuration. The following snippet shows the `network` section of the Logstash Forwarder configuration: [source,json] ------------------------------------------------------------------------------------- # The network section covers network configuration :) "network": { # A list of downstream servers listening for our messages. # logstash-forwarder will pick one at random and only switch if # the selected one appears to be dead or unresponsive "servers": [ "localhost:5043" ], # The path to your client ssl certificate (optional) "ssl certificate": "./logstash-forwarder.crt", # The path to your client ssl key (optional) "ssl key": "./logstash-forwarder.key", # The path to your trusted ssl CA file. This is used # to authenticate your downstream server. "ssl ca": "./logstash-forwarder.crt", # Network timeout in seconds. This is most important for # logstash-forwarder determining whether to stop waiting for an # acknowledgement from the downstream server. If an timeout is reached, # logstash-forwarder will assume the connection or server is bad and # will connect to a server chosen at random from the servers list. "timeout": 15 } ------------------------------------------------------------------------------------- The equivalent in {beatname_uc} would look like this: [source,yaml] ------------------------------------------------------------------------------------- output.logstash: hosts: <1> - localhost:5043 timeout: 15 ssl.certificate_authorities: <2> - ./logstash-forwarder.crt ssl.certificate: ./logstash-forwarder.crt ssl.key: ./logstash-forwarder.key ------------------------------------------------------------------------------------- <1> When multiple hosts are defined, the default behavior in {beatname_uc} is to pick a random host for new connections, similar to the Logstash Forwarder behavior. {beatname_uc} can optionally do load balancing. For more details, see the <> configuration option. <2> Note that if the `ssl` settings are missing, then SSL is disabled. SSL is automatically enabled when you add any of the `ssl` options. For more information about specific configuration options, see <>. [[changed-configuration-options]] [float] === Changed configuration file options With the refactoring of the configuration file, the following options were removed or renamed: [cols="2*", options="header"] |=== |Config Option |Action |`deadTime` |`deadTime` was renamed to `ignore_older`. {beatname_uc} keeps the files that it’s reading open until they are older than the timespan specified by `ignore_older`. If a file is changed, {beatname_uc} reopens it. |`netTimeout` |`netTimeout` was removed and is replaced by the `timeout` option in libbeat. |`log-to-syslog` and `syslog` |Both options were removed and replaced by logging options in libbeat. |=== For more information about these options, see <>. [float] === A complete example Let's see a simple, but complete example of a Logstash Forwarder configuration and its equivalent for {beatname_uc}. Logstash Forwarder configuration: [source,json] ------------------------------------------------------------------------------------- { "files": [ { "paths": [ "/var/log/*.log" ], "fields": { "type": "syslog", "service": "test01" } } ], "network": { "servers": [ "localhost:5043" ], } } ------------------------------------------------------------------------------------- {beatname_uc} configuration: ["source","yaml",subs="attributes"] ------------------------------------------------------------------------------------- {beatname_lc}.inputs: - type: log paths: - /var/log/*.log fields: service: test01 output.elasticsearch: hosts: ["http://localhost:5043"] ------------------------------------------------------------------------------------- [[migration-changed-cli]] == Changes to command line options Most command line options available in Logstash Forwarder have been removed and migrated to config file options. The only mandatory command line option for running {beatname_uc} is `-c` followed by the path to the config file. If you used command line options with Logstash Forwarder, make sure that you add your options to the configuration file. For naming changes, see <>. {beatname_uc} does provide command line options that are common to all Beats. For more details about these options, see <>. [[renamed-options]] [float] === Renamed options The following command line options have been renamed and moved to the config file. Also see <> for a list of configuration file options that were completely removed or replaced by options specified in libbeat. [cols="3*", options="header"] |=== |Command Line Option |Config File Option |Description |`-config` |`-c` command line option and `config_dir` |The config option was split into two parts. You use the `-c` command line option to specify the location of the base (required) config file when you start {beatname_uc}. To use additional config files, you specify the `config_dir` configuration option. The `config_dir` option specifies the path to the directory that contains additional configuration files. This option MUST point to a directory other than the directory where the main {beatname_uc} config file resides. |`-idle-timeout` | |`idle_timeout` was removed. Libbeat is used for publishing logs. |`-spool-size` |`queue.mem.events` |`spool_size` was moved to the config file and removed as a flag. |`-harvester-buff-size` |`harvester_buffer_size` |`harvester_buffer_size` was moved to the config file and removed as a flag. You can now configure the buffer size separately for each harvester. |`-tail` |`tail_files` |`tail_files` was moved to the config file and removed as a flag. You can now configure this option separately for each input. |`-cpuProfileFile` | |`cpuProfileFile` was removed. You can use the profiling options of libbeat instead. For more details on profiling, see https://github.com/elastic/libbeat/issues/122. |`-quiet` | |`quiet` was removed. Libbeat is now used for logging, so you must use the libbeat <> instead. |=== [[migration-changed-fields]] == Changes to the output fields In the default configuration, {beatname_uc} structures its output documents a little differently from the Logstash Forwarder. This section discusses the differences and the options you have in case you want compatibility with the Logstash Forwarder. [float] === Custom fields are grouped under a "fields" dictionary The custom fields (added from the configuration file) are set as top-level fields in Logstash Forwarder but are grouped together under a `fields` dictionary in {beatname_uc}. If you need the old behavior during the migration phase, you can use the <> configuration option: ["source","yaml",subs="attributes"] ------------------------------------------------------------------------------------- {beatname_lc}.inputs: - type: log paths: - /var/log/*.log fields: service: test01 fields_under_root: true ------------------------------------------------------------------------------------- [float] === {beatname_uc} uses "beat.hostname" for sending the hostname of the server While the Logstash Forwarder sends the hostname of the server it's running on in the `host` field, {beatname_uc} uses the `beat.hostname` field for the same purpose. Because `host` is commonly used in the Logstash plugin ecosystem, the Beats input plugin automatically copies `beat.hostname` into `host`. [float] === The "file" field was renamed to "source" The `file` field was renamed to `source`. If you rely on this field being named `file`, you can rename it by using the mutate filter in Logstash. For example: [source,plain] ------------------------------------------------------------------------------------- filter { mutate { rename => { "source" => "file" } } } ------------------------------------------------------------------------------------- [float] === The "line" field was removed The `line` field was removed. This field wasn't correct after restarts, and making it correct would have resulted in a performance penalty. We recommend using the `offset` field instead. [[migration-other-changes]] == Other changes The following list of implementation changes should not affect your experience migrating from Logstash Forwarder, but you should be aware of the changes. Please post GitHub issues if you notice any regressions from Logstash Forwarder. [float] === Packaging The packaging process for {beatname_uc} uses the Beats infrastructure, so some aspects of packaging, such as the init scripts, are different from Logstash Forwarder. Please post GitHub issues if you hit any issues with the new packages. One notable change is the name of the registry file. The name varies depending on the package type: * `registry` for `.tar.gz` and `.tgz` archives * +/usr/lib/{beatname_lc}/registry+ for DEB and RPM packages * +c:\ProgramData{backslash}{beatname_lc}{backslash}registry+ for the Windows zip file [float] === Publisher improvements Behind the scenes, {beatname_uc} uses a slightly improved protocol for communicating with Logstash. [float] === SSL is off by default If you follow the section on migrating the configuration, you will have SSL enabled. However, you must be aware that if the `ssl` section is missing from the configuration file, {beatname_uc} uses an unencrypted connection to talk to Logstash. [float] === Logging {beatname_uc} uses libbeat logging and can also log to rotating files instead of syslog.