383 lines
14 KiB
Text
383 lines
14 KiB
Text
:see-relnotes: See the <<release-notes,release notes>> for a complete list of breaking changes, including changes to beta or experimental functionality.
|
||
|
||
[[breaking-changes]]
|
||
== Breaking changes
|
||
|
||
As a general rule, we strive to keep backwards compatibility between minor
|
||
versions (e.g. 6.x to 6.y) so you can upgrade without any configuration file
|
||
changes, but there are breaking changes between major versions (e.g. 5.x to
|
||
6.y). Migrating directly between non consecutive major versions (e.g. 1.x to
|
||
6.x) is not recommended.
|
||
|
||
See the following topics for a description of breaking changes:
|
||
|
||
* <<breaking-changes-6.3>>
|
||
* <<breaking-changes-6.2>>
|
||
* <<breaking-changes-6.1>>
|
||
* <<breaking-changes-6.0>>
|
||
* {auditbeat}/auditbeat-breaking-changes.html[Breaking changes in Auditbeat 6.2]
|
||
|
||
[[breaking-changes-6.3]]
|
||
=== Breaking changes in 6.3
|
||
|
||
This section discusses the main changes that you should be aware of if you
|
||
upgrade the Beats to version 6.3. {see-relnotes}
|
||
|
||
[[breaking-changes-monitoring]]
|
||
==== Beats monitoring schema changes
|
||
|
||
Starting with version 6.3, the monitoring field `beat.cpu.*.time.metrics` is
|
||
renamed to `beat.cpu.*.time.ms`. As a result of this change, Beats shippers
|
||
released prior to version 6.3 are unable to send monitoring data to clusters
|
||
running on Elasticsearch 6.3 and later. {see-relnotes}
|
||
|
||
[[breaking-changes-mapping-conflict]]
|
||
==== New `host` namespace may cause mapping conflicts for Logstash
|
||
|
||
This breaking change applies only to users who send Beats events to Logstash.
|
||
|
||
Starting with version 6.3, Beats provides an `add_host_metadata` processor for
|
||
adding fields, such as `host.name` and `host.id`, to Beats events. These fields
|
||
are grouped under a `host` prefix and conform to the
|
||
https://github.com/elastic/ecs[Elastic Common Schema (ECS)]. The `host` object
|
||
is defined in the Elasticsearch index template even if the processor is not
|
||
used.
|
||
|
||
We've also added a `host.name` field to all events sent by Beats. This field
|
||
prevents the Beats input plugin in Logstash from adding a default `host` field.
|
||
(By default, the plugin adds a `host` field if the event doesn't already have
|
||
one. We don't want the plugin to add this field because it causes a mapping
|
||
conflict with the `host` object defined in the index template.)
|
||
|
||
*What does this mean to you?*
|
||
|
||
See the info for your particular use case:
|
||
|
||
* <<beats-template-versioned-indices>>
|
||
* <<custom-template-non-versioned-indices>>
|
||
* <<beats-template-non-versioned-indices>>
|
||
|
||
[[beats-template-versioned-indices]]
|
||
===== Use case: You use the Beats index template and versioned indices
|
||
|
||
In this use case, you load the versioned template manually and use the Beats
|
||
versioned index pattern, `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}`,
|
||
as recommended in the {logstash-ref}/plugins-inputs-beats.html[Beats input
|
||
plugin] documentation. This results in a `host` field in 6.2 indices, and a
|
||
`host.name` field in 6.3 indices. There are no mapping conflicts, but
|
||
any visualizations or searches that use `host` will no longer show results for
|
||
6.3 indices.
|
||
|
||
*What do you need to change?*
|
||
|
||
If you searched for the `host` field previously, modify your searches to use the
|
||
`beat.hostname` field instead. The `beat.hostname` field existed prior to 6.3
|
||
and contains the same information as `host.name`. Using this field ensures that
|
||
your queries and aggregations will work as expected in earlier releases and 6.3.
|
||
|
||
To save time when you have a large number of objects to update, you can batch
|
||
this process. Use either the Kibana UI or API to export the objects to JSON,
|
||
make the JSON modification, and then re-import the objects. For more
|
||
information, see:
|
||
|
||
* {kibana-ref}/managing-saved-objects.html[Managing Saved Objects]
|
||
* {kibana-ref}/saved-objects-api.html[Kibana Saved Objects API]
|
||
|
||
|
||
[[custom-template-non-versioned-indices]]
|
||
===== Use case: You use a custom template and your indices are not versioned
|
||
|
||
Mapping conflicts are likely in this use case because two different Beats
|
||
versions (6.2 and 6.3) are sending data to the same index. For 6.2, Logstash
|
||
adds the default `host` field, and for 6.3, Beats adds the `host.name` field,
|
||
which results in a mapping conflict.
|
||
|
||
*What do you need to change?*
|
||
|
||
To resolve the mapping issue, do *one* of the following:
|
||
|
||
* Use versioned indices to prevent the mapping conflict. In the Logstash
|
||
pipeline configuration, set `manage_template => false` and use an index naming
|
||
pattern that includes `[version]` metadata:
|
||
+
|
||
[source,yaml]
|
||
----
|
||
manage_template => false
|
||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||
----
|
||
+
|
||
For more information, see the documentation for the
|
||
{logstash-ref}/plugins-inputs-beats.html[Beats input plugin].
|
||
|
||
* Or, in the Beats config file, configure Beats to drop all `host.*` fields:
|
||
+
|
||
[source,yaml]
|
||
----
|
||
processors:
|
||
- drop_fields:
|
||
fields: ["host"]
|
||
----
|
||
+
|
||
--
|
||
IMPORTANT: If you drop the `host` fields, you cannot use the `add_host_metadata`
|
||
processor.
|
||
|
||
--
|
||
+
|
||
With this configuration, Beats drops the `host` fields before sending the
|
||
event to Logstash, and Logstash adds a default `host` field, as it did with
|
||
previous Beats versions. This approach resolves the mapping conflict, but you
|
||
should plan to migrate your Logstash configurations to use `host.name` in
|
||
future releases.
|
||
|
||
[[beats-template-non-versioned-indices]]
|
||
===== Use case: You use the Beats index template and your indices are not versioned
|
||
|
||
In this use case, you load the Beats index template manually into Elasticsearch,
|
||
and send your data through Logstash, but you don’t use the versioned index
|
||
pattern to create versioned indices.
|
||
|
||
You cannot resolve the problem by dropping the `host.*` fields, because Logstash
|
||
will add a default `host` field, resulting in a mapping conflict with the
|
||
`host` field defined as an object in the index template.
|
||
|
||
*What do you need to change?*
|
||
|
||
To resolve the mapping issue, do *one* of the following:
|
||
|
||
* Drop the `host.*` fields in a Logstash filter. For example:
|
||
+
|
||
[source,yaml]
|
||
----
|
||
filter {
|
||
mutate {
|
||
remove_field => [ "[host]" ]
|
||
}
|
||
}
|
||
----
|
||
+
|
||
With this configuration, there will be no `host` field in the final event at
|
||
ingestion time, and the mapping conflict is avoided.
|
||
|
||
* Or:
|
||
** Modify the Beats index template by removing the `host.*` fields, and
|
||
** Configure Beats to drop all `host.*` fields, as described in
|
||
<<custom-template-non-versioned-indices,this section>>.
|
||
+
|
||
This solution prevents a mapping conflict because the fields are no longer
|
||
defined in the Elasticsearch template. Elasticsearch can use the `host`
|
||
mapping created when Logstash added a default `host` field.
|
||
|
||
The difference between these two approaches is that the first approach, using
|
||
a Logstash filter, drops the `host` fields completely. There will be no `host`
|
||
field in the final event. The second approach drops the `host` fields from the
|
||
Beats event, but because Logstash adds a default `host` field, there will be a
|
||
`host` field in the final event.
|
||
|
||
[[breaking-changes-6.2]]
|
||
=== Breaking changes in 6.2
|
||
|
||
{see-relnotes}
|
||
|
||
[[breaking-changes-6.1]]
|
||
=== Breaking changes in 6.1
|
||
|
||
{see-relnotes}
|
||
|
||
[[breaking-changes-6.0]]
|
||
=== Breaking changes in 6.0
|
||
|
||
This section discusses the main changes that you should be aware of if you
|
||
upgrade the Beats from version 5.x to 6.x. {see-relnotes}
|
||
|
||
// TODO: better link to the consolidated release notes for 6.0.0.
|
||
|
||
|
||
[[breaking-changes-spooler-removed]]
|
||
==== Filebeat spooler removed
|
||
|
||
Version 6.0 comes with a new architecture for the internal pipeline of all
|
||
Beats. This architecture refactoring is mostly internal, but one of the more
|
||
visible effects is that the Spooler component of Filebeat is removed. The
|
||
functionality of the Spooler was similar to the one of the publisher queue from
|
||
libbeat (the code shared by all Beats), and the presence of multiple queues
|
||
made the performance tuning of Filebeat more complex than it needed to be.
|
||
|
||
As a result, the following options are removed:
|
||
|
||
- `filebeat.spool_size`
|
||
- `filebeat.publish_async`
|
||
- `filebeat.idle_timeout`
|
||
- `queue_size`
|
||
- `bulk_queue_size`
|
||
|
||
The first three are specific to Filebeat, while `queue_size` and
|
||
`bulk_queue_size` exist in all Beats. If any of these options is set, Filebeat
|
||
6.0 will refuse to start.
|
||
|
||
Instead of the settings above, the `queue.mem` settings are introduced. If you
|
||
had to tune the `spool_size` or the `queue_size` before, you might want to tune
|
||
the `queue.mem.events` when upgrading. However, it is best to leave the rest of
|
||
the `queue.mem` settings to their default values, as they are appropriate for
|
||
all loads.
|
||
|
||
The `publish_async` option (which was deprecated since 5.3) is removed because
|
||
the new pipeline already works asynchronously by default.
|
||
|
||
// TODO: for the above new settings, link to their configuration settings.
|
||
|
||
[[breaking-changes-single-output]]
|
||
==== Only one enabled output
|
||
|
||
In versions prior to 6.0, you could enabled multiple outputs at the same time,
|
||
but only of different types. For example, you were able to enable the
|
||
Elasticsearch and Logstash outputs, but not two Logstash outputs. The drawback
|
||
of enabling multiple outputs was that the Beats that wait for acknowledgments
|
||
(Filebeat and Winlogbeat) before proceeding slowed down to the slowest output.
|
||
This implication was not obvious and hindered the use cases where multiple
|
||
outputs would have been useful.
|
||
|
||
As part of the pipeline re-architecture that we did for 6.0, we removed the
|
||
option to enable multiple outputs at the same time. This helps with keeping the
|
||
pipeline simple and with clarifying the scope of outputs in Beats.
|
||
|
||
If you require multiple outputs, you have the following options:
|
||
|
||
* use the Logstash output and then use Logstash to pipe the events to multiple
|
||
outputs
|
||
* run multiple instances of the same Beat
|
||
|
||
If you used the `file` or `console` outputs for debugging purposes, in addition
|
||
to the main output, we recommend using the `-d "publish"` option which logs the
|
||
published events in the Filebeat logs.
|
||
|
||
[[breaking-changes-ls-index]]
|
||
==== Logstash index setting now requires version
|
||
|
||
If you use the Logstash output to send data from Beats to Logstash, you need to
|
||
update the `index` setting in your Logstash configuration to include the Beat
|
||
version:
|
||
|
||
[source,json]
|
||
----
|
||
output {
|
||
elasticsearch {
|
||
hosts => "localhost:9200"
|
||
manage_template => false
|
||
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
|
||
}
|
||
}
|
||
----
|
||
|
||
|
||
Prior to 6.0, the recommended setting was:
|
||
|
||
[source,yaml]
|
||
----
|
||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||
----
|
||
|
||
|
||
The index templates that ship with 6.0 are applied to new indices that match the
|
||
pattern `[beat]-[version]-*`. You must update your Logstash config, or the
|
||
templates will not be applied.
|
||
|
||
[[breaking-changes-types]]
|
||
==== Filebeat prospector type and document type changes
|
||
|
||
The `document_type` setting, from the prospector configuration, was removed
|
||
because the `_type` concept is being
|
||
{elasticsearch}/removal-of-types.html[removed from Elasticsearch]. Instead of
|
||
the `document_type` setting, you can use a custom field.
|
||
|
||
This has led also to the rename of the `input_type` configuration setting to
|
||
`type`. This change is backwards compatible because the old setting still
|
||
works. However, the `input_type` output field was renamed to `prospector.type`.
|
||
|
||
[[breaking-changes-default-config]]
|
||
==== Filebeat default prospector disabled in the configuration file
|
||
|
||
The default startup behaviour (based on the included sample configuration) of
|
||
Filebeat was to read all the files matching the `/var/log/*.log` pattern.
|
||
Starting with version 6.0, Filebeat doesn't read any files in its default
|
||
configuration. However, you can easily enable the `system` module, for example
|
||
with a CLI flag:
|
||
|
||
["source","sh",subs="attributes"]
|
||
----
|
||
filebeat --modules=system
|
||
----
|
||
|
||
==== Other settings changed or moved
|
||
|
||
The `outputs.elasticsearch.template.*` settings have been moved under
|
||
`setup.template.*`, but are otherwise unchanged.
|
||
|
||
The `dashboards.*` settings have been moved under `setup.dashboards.*`.
|
||
|
||
The Filebeat deprecated options `force_close_files` and `close_older` are
|
||
removed.
|
||
|
||
[[breaking-changes-import-dashboards]]
|
||
==== Changes for importing the Kibana dashboards
|
||
|
||
The `import_dashboards` program, used to load the Kibana dashboards in previous
|
||
versions of Beats, is replaced by the `setup` command. For example, the
|
||
following command:
|
||
|
||
["source","sh",subs="attributes"]
|
||
----
|
||
./scripts/import_dashboards -user elastic -pass {pwd}
|
||
----
|
||
|
||
Can be replaced with:
|
||
|
||
["source","sh",subs="attributes"]
|
||
----
|
||
./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password={pwd}"
|
||
----
|
||
|
||
Note that the `-E` flags are only required if the Elasticsearch output is not
|
||
already configured in the configuration file.
|
||
|
||
Besides the change in the commands, it's important to note that loading the
|
||
Kibana dashboards works differently in the 6.0 version of the stack. Prior to
|
||
6.0, the dashboards were inserted directly in the `.kibana` Elasticsearch
|
||
index. Starting with 6.0, the Beats use a Kibana server API. This means that
|
||
the Beat that loads the dashboards needs direct access to Kibana and that the
|
||
Kibana URL needs to be set. The option to set the Kibana URL is
|
||
`setup.kibana.host`, which you can set in the configuration file or via the
|
||
`-E` CLI flag:
|
||
|
||
|
||
["source","sh",subs="attributes"]
|
||
----
|
||
./filebeat setup -E "setup.kibana.host=http://kibana-host:5601"
|
||
----
|
||
|
||
The default value for the Kibana host is `localhost:5601`.
|
||
|
||
[[breaking-changes-filters]]
|
||
==== Metricbeat filters renamed to processors
|
||
|
||
The "local" processors, which are configured at the module level, used to be
|
||
called `filters` in Metricbeat, but were offering similar functionality with
|
||
the global `processors`. A notable difference between the two was that the
|
||
filters accessed fields relatively to the metricset (for example,
|
||
`mount_point`), while the processors referred to fields by their fully
|
||
qualified name (for example, `system.filesystem.mount_point`).
|
||
|
||
Starting with version 6.0, the `filters` are renamed to `processors` and they
|
||
can access the fields only by using the fully qualified names.
|
||
|
||
[[breaking-changes-cgo]]
|
||
==== Binaries are dynamically compiled against libc
|
||
|
||
Prior to 6.0, Metricbeat and Packetbeat were compiled using
|
||
https://golang.org/cmd/cgo/[Cgo], while Filebeat, Winlogbeat, and Heartbeat
|
||
were compiled using the pure Go compiler. One of the side-effects of compiling
|
||
with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats
|
||
are compiled using Cgo and therefore dynamically compiled against libc. This
|
||
can reduce the portability of the binaries, but none of the supported platforms
|
||
is affected.
|
||
|