241 lines
15 KiB
Plaintext
241 lines
15 KiB
Plaintext
[[release-notes-5.0.0]]
|
|
=== Beats version 5.0.0
|
|
|
|
The list below covers changes between 1.x to 5.0.0 releases.
|
|
|
|
|
|
==== Breaking changes
|
|
|
|
*Affecting all Beats*
|
|
|
|
- Rename the `filters` section to `processors`. {pull}1944[1944]
|
|
- Require braces for environment variable expansion in config files {pull}1304[1304]
|
|
- On DEB/RPM installations, the binary files are now found under `/usr/share/{{beat_name}}/bin`, not in `/usr/bin`. {pull}1385[1385]
|
|
- The logs are written by default to self rotating files, instead of syslog. {pull}1371[1371]
|
|
- Remove deprecated `host` option from elasticsearch, logstash and redis outputs. {pull}1474[1474]
|
|
- All configuration settings under `shipper:` are moved to be top level configuration settings. I.e.
|
|
`shipper.name:` becomes `name:` in the configuration file. {pull}1570[1570]
|
|
- The `topology_expire` option of the Elasticsearch output was removed. {pull}1907[1907]
|
|
- The Elasticsearch template is now loaded by default. {pull}1993[1993]
|
|
- The Redis output `index` setting is renamed to `key`. `index` still works but it's deprecated. {pull}2077[2077]
|
|
- The undocumented file output `index` setting was removed. Use `filename` instead. {pull}2077[2077]
|
|
- Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. {pull}2119[2119]
|
|
- If the path specified by the `-c` flag is not absolute and `-path.config` is not specified, it
|
|
is considered relative to the current working directory. {pull}2245[2245]
|
|
- Rename `tls` configurations section to `ssl`. {pull}2330[2330]
|
|
- Rename `certificate_key` configuration to `key`. {pull}2330[2330]
|
|
- Replace `tls.insecure` with `ssl.verification_mode` setting. {pull}2330[2330]
|
|
- Replace `tls.min/max_version` with `ssl.supported_protocols` setting requiring full protocol name. {pull}2330[2330]
|
|
- A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. {pull}2688[2688]
|
|
|
|
|
|
*Filebeat*
|
|
|
|
- Scalar values in used in the `fields` configuration setting are no longer automatically converted to strings. {pull}1092[1092]
|
|
- Count field was removed from event as not used in filebeat {issue}778[778]
|
|
- Default location for the registry file was changed to be `data/registry` from the binary directory,
|
|
rather than `.filebeat` in the current working directory. This affects installations for zip/tar.gz/source,
|
|
the location for DEB and RPM packages stays the same. {pull}1373[1373]
|
|
- The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled.
|
|
- Ignore symlinks by default, but they can be enabled with `symlinks` config {pull}1686[1686] {pull}2478[2478]
|
|
- Set `close_inactive` default to 5 minutes (was 1 hour before)
|
|
- Set `clean_removed` and `close_removed` to true by default
|
|
|
|
|
|
*Packetbeat*
|
|
|
|
- Rename output fields in the dns package. Former flag `recursion_allowed` becomes `recursion_available`. {pull}803[803]
|
|
Former SOA field `ttl` becomes `minimum`. {pull}803[803]
|
|
- The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. {pull}803[803]
|
|
- Remove the `count` field from the exported event {pull}1210[1210]
|
|
- Configuration of redis topology support changed. {pull}1353[1353]
|
|
- Move all Packetbeat configuration options under the packetbeat namespace {issue}1417[1417]
|
|
- Set `enabled` ` in `packetbeat.protocols.icmp` configuration to `true` by default. {pull}1988[1988]
|
|
- Group HTTP fields under `http.request` and `http.response` {pull}2167[2167]
|
|
- Export `http.request.body` and `http.response.body` when configured under `include_body_for` {pull}2167[2167]
|
|
- Move `ignore_outgoing` config to `packetbeat.ignore_outgoing` {pull}2393[2393]
|
|
|
|
*Winlogbeat*
|
|
|
|
- The `message_inserts` field was replaced with the `event_data` field {issue}1053[1053]
|
|
- The `category` field was renamed to `task` to better align with the Windows Event Log API naming {issue}1053[1053]
|
|
- Remove the `count` field from the exported event {pull}1218[1218]
|
|
|
|
|
|
==== Bugfixes
|
|
|
|
*Affecting all Beats*
|
|
|
|
- Logstash output will not retry events that are not JSON-encodable {pull}927[927]
|
|
- Drain response buffers when pipelining is used by Redis output. {pull}1353[1353]
|
|
- Unterminated environment variable expressions in config files will now cause an error {pull}1389[1389]
|
|
- Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. {issue}1321[1321]
|
|
- Fix bug affecting `-cpuprofile`, `-memprofile`, and `-httpprof` CLI flags {pull}1415[1415]
|
|
- Fix race when multiple outputs access the same event with logstash output manipulating event {issue}1410[1410] {pull}1428[1428]
|
|
- Seed random number generator using crypto.rand package. {pull}1503{1503]
|
|
- Fix beats hanging in `-configtest` {issue}1213[1213]
|
|
- Reset backoff factor on partial ACK. {issue}1803[1803]
|
|
- Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. {issue}1829[1829]
|
|
- Fix logstash output with pipelining mode enabled not reconnecting. {issue}1876[1876]
|
|
- Empty configuration sections become merge-able with variables containing full path. {pull}1900[1900]
|
|
- Fix error message about required fields missing not printing the missing field name. {pull}1900[1900]
|
|
- Fix sync publisher `PublishEvents` return value if client is closed concurrently. {pull}2046[2046]
|
|
- Fix Logstash output handles error twice when asynchronous sending fails. {pull}2441[2441]
|
|
- Fix Elasticsearch structured error response parsing error. {issue}2229[2229]
|
|
- Fixed the run script to allow the overriding of the configuration file. {issue}2171[2171]
|
|
- Fix logstash output crash if no hosts are configured. {issue}2325[2325]
|
|
- Fix beats failing to start due to invalid duplicate key error in configuration file. {pull}2521[2521]
|
|
- Fix panic on non writable logging directory. {pull}2571[2571]
|
|
- Fix ignoring all fields from drop_fields in case the first field is unknown. {pull}2685[2685]
|
|
- Fix dynamic configuration int/uint` to float type conversion. {pull}2698[2698]
|
|
- Fix primitive types conversion if values are read from environment variables. {pull}2698[2698]
|
|
|
|
|
|
*Filebeat*
|
|
|
|
- Stop filebeat if started without any prospectors defined or empty prospectors {pull}644[644] {pull}647[647]
|
|
- Improve shutdown of crawler and prospector to wait for clean completion {pull}720[720]
|
|
- Omit `fields` from Filebeat events when null {issue}899[899]
|
|
- Improvements in registrar dealing with file rotation. {pull}1281[1281]
|
|
- Multiline reader normalizing newline to use `\n`. {pull}1552[1552]
|
|
- Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. {issue}2041[2041]
|
|
- Fix open file handler issue. {issue}2028[2028] {pull}2020[2020]
|
|
- Fix async publisher sending empty events {pull}2455[2455]
|
|
- Fix potential issue with multiple harvester per file on large file numbers or slow output {pull}2541[2541]
|
|
- Fix input buffer on encoding problem. {pull}2416[2416]
|
|
- Fix issue when `clean_removed` and `clean_inactive` were used together that states were not directly removed from the registry.
|
|
- Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. {pull}2792[2792]
|
|
|
|
|
|
*Packetbeat*
|
|
|
|
- Create a proper BPF filter when ICMP is the only enabled protocol {issue}757[757]
|
|
- Check column length in pgsql parser. {issue}565[565]
|
|
- Harden pgsql parser. {issue}565[565]
|
|
- Add missing nil-check to memcached GapInStream handler. {issue}1162[1162]
|
|
- Fix NFSv4 Operation returning the first found first-class operation available in compound requests. {pull}1821[1821]
|
|
- Fix TCP overlapping segments not being handled correctly. {pull}1898[1898]
|
|
- Fix mapping for some Packetbeat flow metrics that were not marked as being longs. {issue}2177[2177]
|
|
- Fix handling of messages larger than the maximum message size (10MB). {pull}2470[2470]
|
|
- Fix the `bpf_filter` setting. {issue}2660[2660]
|
|
- Fix compile issues for OpenBSD. {pull}1347[1347]
|
|
|
|
|
|
*Winlogbeat*
|
|
|
|
- Fix panic when reading messages larger than 32K characters on Windows XP and 2003. {pull}1498[1498]
|
|
- Fix panic that occurs when reading a large events on Windows Vista and newer. {pull}1499[1499]
|
|
- Adding missing argument to the "Stop processing" log message. {pull}1590[1590]
|
|
- Fix issue with rendering forwarded event log records. {pull}1891[1891]
|
|
- Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. {issue}2041[2041]
|
|
- Fix corrupt registry file that occurs on power loss by disabling file write caching. {issue}2313[2313]
|
|
|
|
|
|
|
|
==== Added
|
|
|
|
*Affecting all Beats*
|
|
|
|
- Update to Go 1.7. {pull}2306[2306]
|
|
- Add option to Elasticsearch output to pass http parameters in index operations {issue}805[805]
|
|
- Improve Logstash and Elasticsearch backoff behavior. {pull}927[927]
|
|
- Add Kafka output. {pull}942[942] {pull}2188[2188] {pull}2190[2190] {pull}2284[2284]
|
|
- Add config file option to configure GOMAXPROCS. {pull}969[969]
|
|
- Improve shutdown handling in libbeat. {pull}1075[1075]
|
|
- Log total non-zero internal metrics on shutdown. {pull}2349[2349]
|
|
- Add `fields` and `fields_under_root` options under the `shipper` configuration {pull}1092[1092]
|
|
- Add the ability to use a SOCKS5 proxy with the Logstash output {issue}823[823]
|
|
- The `-configtest` flag will now print "Config OK" to stdout on success {pull}1249[1249]
|
|
- Add support for TLS to Redis output. {pull}1353[1353]
|
|
- Add SOCKS5 proxy support to Redis output. {pull}1353[1353]
|
|
- Failover and load balancing support in Redis output. {pull}1353[1353]
|
|
- Multiple-worker per host support for Redis output. {pull}1353[1353]
|
|
- Added ability to escape `${x}` in config files to avoid environment variable expansion {pull}1389[1389]
|
|
- Configuration options and CLI flags for setting the home, data and config paths. {pull}1373[1373]
|
|
- Configuration options and CLI flags for setting the default logs path. {pull}1437[1437]
|
|
- Add Elasticsearch template files compatible with Elasticsearch 2.x. {pull}1501[1501]
|
|
- Load the mapping template depending on the Elasticsearch version. {pull}1993[1993]
|
|
- Add conditions to processors. {pull}1623[1623]
|
|
- Enhance `contains` condition to work on fields that are arrays of strings. {issue}2237[2237]
|
|
- Add OR/AND/NOT to the condition associated with the processors. {pull}1983[1983]
|
|
- Improve error message if compiling regular expression from config files fails. {pull}1900[1900]
|
|
- Compression support in the Elasticsearch output. {pull}1835[1835]
|
|
- Periodically log internal metrics. {pull}1955[1955]
|
|
- Add `enabled` setting to all output modules. {pull}1987[1987]
|
|
- Command line flag `-c` can be used multiple times. {pull}1985[1985]
|
|
- Add `-E` CLI flag for overwriting single config options via command line. {pull}1986[1986]
|
|
- Check stdout being available when console output is configured. {issue}2035[2035]
|
|
- Add script to generate the Kibana index-pattern from fields.yml. {pull}2122[2122]
|
|
- Enhance Redis output key selection based on format string. {pull}2169[2169]
|
|
- Configurable Redis `keys` using filters and format strings. {pull}2169[2169]
|
|
- Lookup the configuration file relative to the `-path.config` CLI flag. {pull}2245[2245]
|
|
- Re-write `import_dashboards` script in Golang. {pull}2155[2155]
|
|
- Add support for encrypted private key files by introducing `ssl.key_passphrase` setting. {pull}2330[2330]
|
|
- Add `beat.version` fields to all events.
|
|
- Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. {pull}2627[2627]
|
|
|
|
|
|
*Filebeat*
|
|
|
|
- Add the ability to set a list of tags for each prospector {pull}1092[1092]
|
|
- Add JSON decoding support {pull}1143[1143]
|
|
- The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. {pull}1703[1703]
|
|
- Introduce `close_removed` and `close_renamed` harvester options. {issue}1600[1600]
|
|
- Introduce `close_eof` harvester option. {issue}1600[1600]
|
|
- Add `clean_removed` and `clean_inactive` config option. {issue}1600[1600]
|
|
- Introduce `close_timeout` harvester options {issue}1926[1926]
|
|
- Strip BOM from first message in case of BOM files {issue}2351[2351]
|
|
- Add `harvester_limit` option {pull}2417[2417]
|
|
|
|
|
|
*Packetbeat*
|
|
|
|
- Change the DNS library used throughout the dns package to github.com/miekg/dns. {pull}803[803]
|
|
- Add support for NFS v3 and v4. {pull}1231[1231]
|
|
- Add support for EDNS and DNSSEC. {pull}1292[1292]
|
|
- Add `enabled` setting to Packetbeat protocols. {pull}1988[1988]
|
|
- Add `enabled` setting to Packetbeat network flows configuration. {pull}1988[1988]
|
|
- Add Cassandra protocol analyzer to Packetbeat. {pull}1959[1959]
|
|
- Match connections with IPv6 addresses to processes {pull}2254[2254]
|
|
- Add IP address to `-devices` command output {pull}2327[2327]
|
|
- Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. {pull}2470[2470]
|
|
|
|
*Metricbeat*
|
|
|
|
- First public release, containing the following modules: system, Apache, MySQL, PostgreSQL, Nginx, Redis, MongoDB, and Zookeeper.
|
|
Metricbeat is replacing Topbeat in 5.0, and its default behaviour is to export system statistics like CPU usage,
|
|
memory usage, Disk IO similar with what Topbeat 1.x is doing.
|
|
- Add `pgid` field to process information. {pull} 2021[2021]
|
|
- Use the new `scaled_float` Elasticsearch type for the percentage values. {pull}2156[2156]
|
|
- Improve mapping by converting `half_float` to `scaled_float` and integers to long. {pull}2430[2430]
|
|
- Add experimental `cgroup` metrics to the system/process MetricSet. {pull}2184[2184]
|
|
- Add experimental `haproxy` module. {pull}2384[2384]
|
|
|
|
|
|
*Winlogbeat*
|
|
|
|
- Add caching of event metadata handles and the system render context for the wineventlog API {pull}888[888]
|
|
- Improve config validation by checking for unknown top-level YAML keys. {pull}1100[1100]
|
|
- Add the ability to set `tags`, `fields`, and `fields_under_root` as options for each event log {pull}1092[1092]
|
|
- Add additional data to the events published by Winlogbeat. The new fields are `activity_id`,
|
|
`event_data`, `keywords`, `opcode`, `process_id`, `provider_guid`, `related_activity_id`,
|
|
`task`, `thread_id`, `user_data`, and `version`. {issue}1053[1053]
|
|
- Add `event_id`, `level`, and `provider` configuration options for filtering events {pull}1218[1218]
|
|
- Add `include_xml` configuration option for including the raw XML with the event {pull}1218[1218]
|
|
|
|
|
|
==== Deprecated
|
|
|
|
*Affecting all Beats*
|
|
|
|
- The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. {pull}1601[1601]
|
|
- Topology map is deprecated. This applies to the settings: `refresh_topology_freq`, `topology_expire`, `save_topology`,
|
|
`host_topology`, `password_topology`, `db_topology`.
|
|
- Setting `port` has been deprecated in Redis and Logstash outputs. {pull}2620[2620]
|
|
|
|
*Filebeat*
|
|
|
|
- Deprecate `close_older` option and replace it with `close_inactive`. {issue}2051[2051]
|
|
- Deprecate `force_close_files` option and replace it with `close_removed` and `close_renamed`. {issue}1600[1600]
|
|
|