youtubebeat/vendor/github.com/elastic/beats/libbeat/docs/breaking.asciidoc

:see-relnotes: See the <<release-notes,release notes>> for a complete list of breaking changes, including changes to beta or experimental functionality.

[[breaking-changes]]
== Breaking changes

As a general rule, we strive to keep backwards compatibility between minor
versions (e.g.  6.x to 6.y) so you can upgrade without any configuration file
changes, but there are breaking changes between major versions (e.g. 5.x to
6.y). Migrating directly between non consecutive major versions (e.g. 1.x to
6.x) is not recommended.

See the following topics for a description of breaking changes:

* <<breaking-changes-6.3>>
* <<breaking-changes-6.2>>
* <<breaking-changes-6.1>>
* <<breaking-changes-6.0>>
* {auditbeat}/auditbeat-breaking-changes.html[Breaking changes in Auditbeat 6.2]

[[breaking-changes-6.3]]
=== Breaking changes in 6.3

This section discusses the main changes that you should be aware of if you
upgrade the Beats to version 6.3. {see-relnotes}

[[breaking-changes-monitoring]]
==== Beats monitoring schema changes

Starting with version 6.3, the monitoring field `beat.cpu.*.time.metrics` is
renamed to `beat.cpu.*.time.ms`. As a result of this change, Beats shippers
released prior to version 6.3 are unable to send monitoring data to clusters
running on Elasticsearch 6.3 and later. {see-relnotes}

[[breaking-changes-mapping-conflict]]
==== New `host` namespace may cause mapping conflicts for Logstash

This breaking change applies only to users who send Beats events to Logstash.

Starting with version 6.3, Beats provides an `add_host_metadata` processor for
adding fields, such as `host.name` and `host.id`, to Beats events. These fields
are grouped under a `host` prefix and conform to the
https://github.com/elastic/ecs[Elastic Common Schema (ECS)]. The `host` object
is defined in the Elasticsearch index template even if the processor is not
used.

We've also added a `host.name` field to all events sent by Beats. This field
prevents the Beats input plugin in Logstash from adding a default `host` field.
(By default, the plugin adds a `host` field if the event doesn't already have
one. We don't want the plugin to add this field because it causes a mapping
conflict with the `host` object defined in the index template.)

*What does this mean to you?*

See the info for your particular use case:

* <<beats-template-versioned-indices>>
* <<custom-template-non-versioned-indices>>
* <<beats-template-non-versioned-indices>>

[[beats-template-versioned-indices]]
===== Use case: You use the Beats index template and versioned indices

In this use case, you load the versioned template manually and use the Beats
versioned index pattern, `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}`,
as recommended in the {logstash-ref}/plugins-inputs-beats.html[Beats input
plugin] documentation. This results in a `host` field in 6.2 indices, and a
`host.name` field in 6.3 indices. There are no mapping conflicts, but
any visualizations or searches that use `host` will no longer show results for
6.3 indices. 

*What do you need to change?*

If you searched for the `host` field previously, modify your searches to use the
`beat.hostname` field instead. The `beat.hostname` field existed prior to 6.3
and contains the same information as `host.name`. Using this field ensures that
your queries and aggregations will work as expected in earlier releases and 6.3.

To save time when you have a large number of objects to update, you can batch
this process. Use either the Kibana UI or API to export the objects to JSON,
make the JSON modification, and then re-import the objects. For more
information, see:

* {kibana-ref}/managing-saved-objects.html[Managing Saved Objects]
* {kibana-ref}/saved-objects-api.html[Kibana Saved Objects API]


[[custom-template-non-versioned-indices]]
===== Use case: You use a custom template and your indices are not versioned

Mapping conflicts are likely in this use case because two different Beats
versions (6.2 and 6.3) are sending data to the same index. For 6.2, Logstash
adds the default `host` field, and for 6.3, Beats adds the `host.name` field,
which results in a mapping conflict.

*What do you need to change?*

To resolve the mapping issue, do *one* of the following:

* Use versioned indices to prevent the mapping conflict. In the Logstash
pipeline configuration, set `manage_template => false` and use an index naming
pattern that includes `[version]` metadata:
+
[source,yaml]
----
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
----
+
For more information, see the documentation for the
{logstash-ref}/plugins-inputs-beats.html[Beats input plugin].

* Or, in the Beats config file, configure Beats to drop all `host.*` fields: 
+
[source,yaml]
----
  processors:
   - drop_fields:
      fields: ["host"]
----
+
--
IMPORTANT: If you drop the `host` fields, you cannot use the `add_host_metadata`
processor.

--
+
With this configuration, Beats drops the `host` fields before sending the
event to Logstash, and Logstash adds a default `host` field, as it did with
previous Beats versions. This approach resolves the mapping conflict, but you
should plan to migrate your Logstash configurations to use `host.name` in
future releases. 

[[beats-template-non-versioned-indices]]
===== Use case: You use the Beats index template and your indices are not versioned

In this use case, you load the Beats index template manually into Elasticsearch,
and send your data through Logstash, but you don’t use the versioned index
pattern to create versioned indices.

You cannot resolve the problem by dropping the `host.*` fields, because Logstash
will add a default `host` field, resulting in a mapping conflict with the
`host` field defined as an object in the index template. 

*What do you need to change?*

To resolve the mapping issue, do *one* of the following:

* Drop the `host.*` fields in a Logstash filter. For example:
+
[source,yaml]
----
filter {
  mutate {
    remove_field => [ "[host]" ]
  }
}
----
+
With this configuration, there will be no `host` field in the final event at
ingestion time, and the mapping conflict is avoided.

* Or:
** Modify the Beats index template by removing the `host.*` fields, and
** Configure Beats to drop all `host.*` fields, as described in
<<custom-template-non-versioned-indices,this section>>.
+
This solution prevents a mapping conflict because the fields are no longer
defined in the Elasticsearch template. Elasticsearch can use the `host`
mapping created when Logstash added a default `host` field.

The difference between these two approaches is that the first approach, using
a Logstash filter, drops the `host` fields completely. There will be no `host`
field in the final event. The second approach drops the `host` fields from the
Beats event, but because Logstash adds a default `host` field, there will be a
`host` field in the final event.

[[breaking-changes-6.2]]
=== Breaking changes in 6.2

{see-relnotes}

[[breaking-changes-6.1]]
=== Breaking changes in 6.1

{see-relnotes}

[[breaking-changes-6.0]]
=== Breaking changes in 6.0

This section discusses the main changes that you should be aware of if you
upgrade the Beats from version 5.x to 6.x. {see-relnotes}

// TODO: better link to the consolidated release notes for 6.0.0.


[[breaking-changes-spooler-removed]]
==== Filebeat spooler removed

Version 6.0 comes with a new architecture for the internal pipeline of all
Beats. This architecture refactoring is mostly internal, but one of the more
visible effects is that the Spooler component of Filebeat is removed. The
functionality of the Spooler was similar to the one of the publisher queue from
libbeat (the code shared by all Beats), and the presence of multiple queues
made the performance tuning of Filebeat more complex than it needed to be.

As a result, the following options are removed:

- `filebeat.spool_size`
- `filebeat.publish_async`
- `filebeat.idle_timeout`
- `queue_size`
- `bulk_queue_size`

The first three are specific to Filebeat, while `queue_size` and
`bulk_queue_size` exist in all Beats. If any of these options is set, Filebeat
6.0 will refuse to start.

Instead of the settings above, the `queue.mem` settings are introduced. If you
had to tune the `spool_size` or the `queue_size` before, you might want to tune
the `queue.mem.events` when upgrading. However, it is best to leave the rest of
the `queue.mem` settings to their default values, as they are appropriate for
all loads.

The `publish_async` option (which was deprecated since 5.3) is removed because
the new pipeline already works asynchronously by default.

// TODO: for the above new settings, link to their configuration settings.

[[breaking-changes-single-output]]
==== Only one enabled output

In versions prior to 6.0, you could enabled multiple outputs at the same time,
but only of different types. For example, you were able to enable the
Elasticsearch and Logstash outputs, but not two Logstash outputs. The drawback
of enabling multiple outputs was that the Beats that wait for acknowledgments
(Filebeat and Winlogbeat) before proceeding slowed down to the slowest output.
This implication was not obvious and hindered the use cases where multiple
outputs would have been useful.

As part of the pipeline re-architecture that we did for 6.0, we removed the
option to enable multiple outputs at the same time. This helps with keeping the
pipeline simple and with clarifying the scope of outputs in Beats.

If you require multiple outputs, you have the following options:

* use the Logstash output and then use Logstash to pipe the events to multiple
  outputs
* run multiple instances of the same Beat

If you used the `file` or `console` outputs for debugging purposes, in addition
to the main output, we recommend using the `-d "publish"` option which logs the
published events in the Filebeat logs.

[[breaking-changes-ls-index]]
==== Logstash index setting now requires version

If you use the Logstash output to send data from Beats to Logstash, you need to
update the `index` setting in your Logstash configuration to include the Beat
version:

[source,json]
----
output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
----


Prior to 6.0, the recommended setting was:

[source,yaml]
----
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
----


The index templates that ship with 6.0 are applied to new indices that match the
pattern `[beat]-[version]-*`. You must update your Logstash config, or the
templates will not be applied.

[[breaking-changes-types]]
==== Filebeat prospector type and document type changes

The `document_type` setting, from the prospector configuration, was removed
because the `_type` concept is being
{elasticsearch}/removal-of-types.html[removed from Elasticsearch]. Instead of
the `document_type` setting, you can use a custom field.

This has led also to the rename of the `input_type` configuration setting to
`type`. This change is backwards compatible because the old setting still
works. However, the `input_type` output field was renamed to `prospector.type`.

[[breaking-changes-default-config]]
==== Filebeat default prospector disabled in the configuration file

The default startup behaviour (based on the included sample configuration) of
Filebeat was to read all the files matching the `/var/log/*.log` pattern.
Starting with version 6.0, Filebeat doesn't read any files in its default
configuration. However, you can easily enable the `system` module, for example
with a CLI flag:

["source","sh",subs="attributes"]
----
filebeat --modules=system
----

==== Other settings changed or moved

The `outputs.elasticsearch.template.*` settings have been moved under
`setup.template.*`, but are otherwise unchanged.

The `dashboards.*` settings have been moved under `setup.dashboards.*`.

The Filebeat deprecated options	`force_close_files` and `close_older` are
removed.

[[breaking-changes-import-dashboards]]
==== Changes for importing the Kibana dashboards

The `import_dashboards` program, used to load the Kibana dashboards in previous
versions of Beats, is replaced by the `setup` command. For example, the
following command:

["source","sh",subs="attributes"]
----
./scripts/import_dashboards -user elastic -pass {pwd}
----

Can be replaced with:

["source","sh",subs="attributes"]
----
./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password={pwd}"
----

Note that the `-E` flags are only required if the Elasticsearch output is not
already configured in the configuration file.

Besides the change in the commands, it's important to note that loading the
Kibana dashboards works differently in the 6.0 version of the stack. Prior to
6.0, the dashboards were inserted directly in the `.kibana` Elasticsearch
index. Starting with 6.0, the Beats use a Kibana server API. This means that
the Beat that loads the dashboards needs direct access to Kibana and that the
Kibana URL needs to be set. The option to set the Kibana URL is
`setup.kibana.host`, which you can set in the configuration file or via the
`-E` CLI flag:


["source","sh",subs="attributes"]
----
./filebeat setup -E "setup.kibana.host=http://kibana-host:5601"
----

The default value for the Kibana host is `localhost:5601`.

[[breaking-changes-filters]]
==== Metricbeat filters renamed to processors

The "local" processors, which are configured at the module level, used to be
called `filters` in Metricbeat, but were offering similar functionality with
the global `processors`. A notable difference between the two was that the
filters accessed fields relatively to the metricset (for example,
`mount_point`), while the processors referred to fields by their fully
qualified name (for example, `system.filesystem.mount_point`).

Starting with version 6.0, the `filters` are renamed to `processors` and they
can access the fields only by using the fully qualified names.

[[breaking-changes-cgo]]
==== Binaries are dynamically compiled against libc

Prior to 6.0, Metricbeat and Packetbeat were compiled using
https://golang.org/cmd/cgo/[Cgo], while Filebeat, Winlogbeat, and Heartbeat
were compiled using the pure Go compiler. One of the side-effects of compiling
with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats
are compiled using Cgo and therefore dynamically compiled against libc. This
can reduce the portability of the binaries, but none of the supported platforms
is affected.