gaugendre/youtubebeat
1
0
Fork 0
Code Issues Pull requests Releases Activity
youtubebeat/vendor/github.com/elastic/beats/libbeat/docs/outputconfig.asciidoc

1440 lines
49 KiB
Text
Raw Blame History

//////////////////////////////////////////////////////////////////////////
//// This content is shared by all Elastic Beats. Make sure you keep the
//// descriptions here generic enough to work for all Beats that include
//// this file. When using cross references, make sure that the cross
//// references resolve correctly for any files that include this one.
//// Use the appropriate variables defined in the index.asciidoc file to
//// resolve Beat names: beatname_uc and beatname_lc.
//// Use the following include to pull this content into a doc file:
//// include::../../libbeat/docs/outputconfig.asciidoc[]
//// Make sure this content appears below a level 2 heading.
//////////////////////////////////////////////////////////////////////////
[[configuring-output]]
== Configure the output
ifdef::only-elasticsearch[]
You configure {beatname_uc} to write to Elasticsearch by setting options
in the `output.elasticsearch` section of the +{beatname_lc}.yml+ config file
endif::[]
ifndef::only-elasticsearch[]
You configure {beatname_uc} to write to a specific output by setting options
in the `output` section of the +{beatname_lc}.yml+ config file. Only a single
output may be defined.
The following topics describe how to configure each supported output:
* <<elasticsearch-output>>
* <<logstash-output>>
* <<kafka-output>>
ifndef::no-redis-output[]
* <<redis-output>>
endif::[]
* <<file-output>>
* <<console-output>>
* <<configure-cloud-id>>
If you've secured the {stack}, also read <<securing-{beatname_lc}>> for more about
security-related configuration options.
endif::[]
ifdef::beat-specific-output-config[]
include::{beat-specific-output-config}[]
endif::[]
[[elasticsearch-output]]
=== Configure the Elasticsearch output
++++
<titleabbrev>Elasticsearch</titleabbrev>
++++
When you specify Elasticsearch for the output, {beatname_uc} sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.
Example configuration:
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["https://localhost:9200"]
index: "{beatname_lc}-%{[beat.version]}-%{+yyyy.MM.dd}"
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
------------------------------------------------------------------------------
To enable SSL, just add `https` to all URLs defined under __hosts__.
["source","yaml",subs="attributes,callouts"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["https://localhost:9200"]
username: "{beatname_lc}_internal"
password: "{pwd}"
------------------------------------------------------------------------------
If the Elasticsearch nodes are defined by `IP:PORT`, then add `protocol: https` to the yaml file.
[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["localhost"]
protocol: "https"
username: "{beatname_lc}_internal"
password: "{pwd}"
------------------------------------------------------------------------------
For more information about securing {beatname_uc}, see
<<securing-{beatname_lc}>>.
==== Compatibility
This output works with all compatible versions of Elasticsearch. See the
https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support
Matrix].
==== Configuration options
You can specify the following options in the `elasticsearch` section of the +{beatname_lc}.yml+ config file:
===== `enabled`
The enabled config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true.
[[hosts-option]]
===== `hosts`
The list of Elasticsearch nodes to connect to. The events are distributed to
these nodes in round robin order. If one node becomes unreachable, the event is
automatically sent to another node. Each Elasticsearch node can be defined as a `URL` or `IP:PORT`.
For example: `http://192.15.3.2`, `https://es.found.io:9230` or `192.24.3.2:9300`.
If no port is specified, `9200` is used.
NOTE: When a node is defined as an `IP:PORT`, the _scheme_ and _path_ are taken from the
<<protocol-option,`protocol`>> and <<path-option,`path`>> config options.
[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["10.45.3.2:9220", "10.45.3.1:9230"]
protocol: https
path: /elasticsearch
------------------------------------------------------------------------------
In the previous example, the Elasticsearch nodes are available at `https://10.45.3.2:9220/elasticsearch` and
`https://10.45.3.1:9230/elasticsearch`.
===== `compression_level`
The gzip compression level. Setting this value to 0 disables compression.
The compression level must be in the range of 1 (best speed) to 9 (best compression).
Increasing the compression level will reduce the network usage but will increase the cpu usage.
The default value is 0.
===== `escape_html`
Configure escaping of HTML in strings. Set to `false` to disable escaping.
The default value is `true`.
===== `worker`
The number of workers per configured host publishing events to Elasticsearch. This
is best used with load balancing mode enabled. Example: If you have 2 hosts and
3 workers, in total 6 workers are started (3 for each host).
The default value is 1.
===== `username`
The basic authentication username for connecting to Elasticsearch.
===== `password`
The basic authentication password for connecting to Elasticsearch.
===== `parameters`
Dictionary of HTTP parameters to pass within the url with index operations.
[[protocol-option]]
===== `protocol`
The name of the protocol Elasticsearch is reachable on. The options are:
`http` or `https`. The default is `http`. However, if you specify a URL for
<<hosts-option,`hosts`>>, the value of `protocol` is overridden by whatever scheme you
specify in the URL.
[[path-option]]
===== `path`
An HTTP path prefix that is prepended to the HTTP API calls. This is useful for
the cases where Elasticsearch listens behind an HTTP reverse proxy that exports
the API under a custom prefix.
===== `headers`
Custom HTTP headers to add to each request created by the Elasticsearch output.
Example:
[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch.headers:
X-My-Header: Header contents
------------------------------------------------------------------------------
It is generally possible to specify multiple header values for the same header
name by separating them with a comma.
===== `proxy_url`
The URL of the proxy to use when connecting to the Elasticsearch servers. The
value may be either a complete URL or a "host[:port]", in which case the "http"
scheme is assumed. If a value is not specified through the configuration file
then proxy environment variables are used. See the
https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation]
for more information about the environment variables.
[[index-option-es]]
===== `index`
The index name to write events to. The default is
+"{beatname_lc}-%\{[beat.version]\}-%\{+yyyy.MM.dd\}"+ (for example,
+"{beatname_lc}-{version}-{localdate}"+). If you change this setting, you also
need to configure the `setup.template.name` and `setup.template.pattern` options
(see <<configuration-template>>). If you are using the pre-built Kibana
dashboards, you also need to set the `setup.dashboards.index` option (see
<<configuration-dashboards>>).
You can set the index dynamically by using a format string to access any event
field. For example, this configuration uses a custom field, `fields.log_type`,
to set the index:
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
index: "%\{[fields.log_type]\}-%\{[beat.version]\}-%\{+yyyy.MM.dd}\" <1>
------------------------------------------------------------------------------
<1> We recommend including `beat.version` in the name to avoid mapping issues
when you upgrade.
With this configuration, all events with `log_type: normal` are sent to an
index named +normal-{version}-{localdate}+, and all events with
`log_type: critical` are sent to an index named
+critical-{version}-{localdate}+.
TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
See the <<indices-option-es,`indices`>> setting for other ways to set the index
dynamically.
ifdef::deprecate_dashboard_loading[]
deprecated[{deprecate_dashboard_loading}]
endif::[]
[[indices-option-es]]
===== `indices`
An array of index selector rules. Each rule specifies the index to use for
events that match the rule. During publishing, {beatname_uc} uses the first
matching rule in the array. Rules can contain conditionals, format string-based
fields, and name mappings. If the `indices` setting is missing or no rule
matches, the <<index-option-es,`index`>> setting is used.
Rule settings:
*`index`*:: The index format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule fails.
*`mappings`*:: A dictionary that takes the value returned by `index` and maps it
to a new name.
*`default`*:: The default string value to use if `mappings` does not find a
match.
*`when`*:: A condition that must succeed in order to execute the current rule.
All the <<conditions,conditions>> supported by processors are also supported
here.
The following example sets the index based on whether the `message` field
contains the specified string:
["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
indices:
- index: "warning-%{[beat.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "WARN"
- index: "error-%{[beat.version]}-%{+yyyy.MM.dd}"
when.contains:
message: "ERR"
------------------------------------------------------------------------------
This configuration results in indices named +warning-{version}-{localdate}+
and +error-{version}-{localdate}+ (plus the default index if no matches are
found).
The following example sets the index by taking the name returned by the `index`
format string and mapping it to a new name that's used for the index:
["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
indices:
- index: "%{[fields.log_type]}"
mappings:
critical: "sev1"
normal: "sev2"
default: "sev3"
------------------------------------------------------------------------------
This configuration results in indices named `sev1`, `sev2`, and `sev3`.
The `mappings` setting simplifies the configuration, but is limited to string
values. You cannot specify format strings within the mapping pairs.
ifndef::no-pipeline[]
[[pipeline-option-es]]
===== `pipeline`
A format string value that specifies the ingest node pipeline to write events to.
["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
pipeline: my_pipeline_id
------------------------------------------------------------------------------
For more information, see <<configuring-ingest-node>>.
You can set the ingest node pipeline dynamically by using a format string to
access any event field. For example, this configuration uses a custom field,
`fields.log_type`, to set the pipeline for each event:
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
pipeline: "%\{[fields.log_type]\}_pipeline"
------------------------------------------------------------------------------
With this configuration, all events with `log_type: normal` are sent to a pipeline
named `normal_pipeline`, and all events with `log_type: critical` are sent to a
pipeline named `critical_pipeline`.
TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
See the <<pipelines-option-es,`pipelines`>> setting for other ways to set the
ingest node pipeline dynamically.
[[pipelines-option-es]]
===== `pipelines`
An array of pipeline selector rules. Each rule specifies the ingest node
pipeline to use for events that match the rule. During publishing, {beatname_uc}
uses the first matching rule in the array. Rules can contain conditionals,
format string-based fields, and name mappings. If the `pipelines` setting is
missing or no rule matches, the <<pipeline-option-es,`pipeline`>> setting is
used.
Rule settings:
*`pipeline`*:: The pipeline format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule
fails.
*`mappings`*:: A dictionary that takes the value returned by `pipeline` and maps
it to a new name.
*`default`*:: The default string value to use if `mappings` does not find a
match.
*`when`*:: A condition that must succeed in order to execute the current rule.
All the <<conditions,conditions>> supported by processors are also supported
here.
The following example sends events to a specific pipeline based on whether the
`message` field contains the specified string:
["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
pipelines:
- pipeline: "warning_pipeline"
when.contains:
message: "WARN"
- pipeline: "error_pipeline"
when.contains:
message: "ERR"
------------------------------------------------------------------------------
The following example sets the pipeline by taking the name returned by the
`pipeline` format string and mapping it to a new name that's used for the
pipeline:
["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
pipelines:
- pipeline: "%{[fields.log_type]}"
mappings:
critical: "sev1_pipeline"
normal: "sev2_pipeline"
default: "sev3_pipeline"
------------------------------------------------------------------------------
With this configuration, all events with `log_type: critical` are sent to
`sev1_pipeline`, all events with `log_type: normal` are sent to a
`sev2_pipeline`, and all other events are sent to `sev3_pipeline`.
For more information about ingest node pipelines, see
<<configuring-ingest-node>>.
endif::[]
===== `max_retries`
ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="winlogbeat")]
{beatname_uc} ignores the `max_retries` setting and retries indefinitely.
endif::[]
ifeval::[("{beatname_lc}"!="filebeat") and ("{beatname_lc}"!="winlogbeat")]
The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.
Set `max_retries` to a value less than 0 to retry until all events are published.
The default is 3.
endif::[]
===== `bulk_max_size`
The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.
Events can be collected into batches. {beatname_uc} will split batches larger than `bulk_max_size`
into multiple batches.
Specifying a larger batch size can improve performance by lowering the overhead of sending events.
However big batch sizes can also increase processing times, which might result in
API errors, killed connections, timed-out publishing requests, and, ultimately, lower
throughput.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.
===== `backoff.init`
The number of seconds to wait before trying to reconnect to Elasticsearch after
a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to
reconnect. If the attempt fails, the backoff timer is increased exponentially up
to `backoff.max`. After a successful connection, the backoff timer is reset. The
default is 1s.
===== `backoff.max`
The maximum number of seconds to wait before attempting to connect to
Elasticsearch after a network error. The default is 60s.
===== `timeout`
The http request timeout in seconds for the Elasticsearch request. The default is 90.
===== `ssl`
Configuration options for SSL parameters like the certificate authority to use
for HTTPS-based connections. If the `ssl` section is missing, the host CAs are used for HTTPS connections to
Elasticsearch.
See <<configuration-ssl>> for more information.
ifndef::only-elasticsearch[]
[[logstash-output]]
=== Configure the Logstash output
++++
<titleabbrev>Logstash</titleabbrev>
++++
The Logstash output sends events directly to Logstash by using the lumberjack
protocol, which runs over TCP. Logstash allows for additional processing and routing of
generated events.
include::./shared-logstash-config.asciidoc[]
==== Accessing metadata fields
Every event sent to Logstash contains the following metadata fields that you can
use in Logstash for indexing and filtering:
["source","json",subs="attributes"]
------------------------------------------------------------------------------
{
...
"@metadata": { <1>
"beat": "{beatname_lc}", <2>
"version": "{stack-version}" <3>
"type": "doc" <4>
}
}
------------------------------------------------------------------------------
<1> {beatname_uc} uses the `@metadata` field to send metadata to Logstash. See the
{logstashdoc}/event-dependent-configuration.html#metadata[Logstash documentation]
for more about the `@metadata` field.
<2> The default is {beatname_lc}. To change this value, set the
<<logstash-index,`index`>> option in the {beatname_uc} config file.
<3> The beats current version.
<4> The value of `type` is currently hardcoded to `doc`. It was used by previous
Logstash configs to set the type of the document in Elasticsearch.
WARNING: The `@metadata.type` field, added by the Logstash output, is
deprecated, hardcoded to `doc`, and will be removed in {beatname_uc} 7.0.
You can access this metadata from within the Logstash config file to set values
dynamically based on the contents of the metadata.
For example, the following Logstash configuration file for versions 2.x and
5.x sets Logstash to use the index and document type reported by Beats for
indexing events into Elasticsearch:
[source,logstash]
------------------------------------------------------------------------------
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" <1>
}
}
------------------------------------------------------------------------------
<1> `%{[@metadata][beat]}` sets the first part of the index name to the value
of the `beat` metadata field, `%{[@metadata][version]}` sets the second part to
the Beat's version, and `%{+YYYY.MM.dd}` sets the third part of the
name to a date based on the Logstash `@timestamp` field. For example:
+{beatname_lc}-{version}-2017.03.29+.
Events indexed into Elasticsearch with the Logstash configuration shown here
will be similar to events directly indexed by Beats into Elasticsearch.
==== Compatibility
This output works with all compatible versions of Logstash. See the
https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support
Matrix].
==== Configuration options
You can specify the following options in the `logstash` section of the
+{beatname_lc}.yml+ config file:
===== `enabled`
The enabled config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true.
[[hosts]]
===== `hosts`
The list of known Logstash servers to connect to. If load balancing is disabled, but
multiple hosts are configured, one host is selected randomly (there is no precedence).
If one host becomes unreachable, another one is selected randomly.
All entries in this list can contain a port number. If no port number is given, the
value specified for <<port,`port`>> is used as the default port number.
===== `compression_level`
The gzip compression level. Setting this value to 0 disables compression.
The compression level must be in the range of 1 (best speed) to 9 (best compression).
Increasing the compression level will reduce the network usage but will increase the cpu usage.
The default value is 3.
===== `escape_html`
Configure escaping of HTML in strings. Set to `false` to disable escaping.
The default value is `true`.
===== `worker`
The number of workers per configured host publishing events to Logstash. This
is best used with load balancing mode enabled. Example: If you have 2 hosts and
3 workers, in total 6 workers are started (3 for each host).
[[loadbalance]]
===== `loadbalance`
If set to true and multiple Logstash hosts are configured, the output plugin
load balances published events onto all Logstash hosts. If set to false,
the output plugin sends all events to only one host (determined at random) and
will switch to another host if the selected one becomes unresponsive. The default value is false.
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.logstash:
hosts: ["localhost:5044", "localhost:5045"]
loadbalance: true
index: {beatname_lc}
------------------------------------------------------------------------------
===== `ttl`
Time to live for a connection to Logstash after which the connection will be re-established.
Useful when Logstash hosts represent load balancers. Since the connections to Logstash hosts
are sticky operating behind load balancers can lead to uneven load distribution between the instances.
Specifying a TTL on the connection allows to achieve equal connection distribution between the
instances. Specifying a TTL of 0 will disable this feature.
The default value is 0.
NOTE: The "ttl" option is not yet supported on an async Logstash client (one with the "pipelining" option set).
===== `pipelining`
Configures number of batches to be sent asynchronously to logstash while waiting
for ACK from logstash. Output only becomes blocking once number of `pipelining`
batches have been written. Pipelining is disabled if a values of 0 is
configured. The default value is 2.
[[port]]
===== `port`
deprecated[5.0.0]
The default port to use if the port number is not given in <<hosts,`hosts`>>. The default port number
is 5044.
===== `proxy_url`
The URL of the SOCKS5 proxy to use when connecting to the Logstash servers. The
value must be a URL with a scheme of `socks5://`. The protocol used to
communicate to Logstash is not based on HTTP so a web-proxy cannot be used.
If the SOCKS5 proxy server requires client authentication, then a username and
password can be embedded in the URL as shown in the example.
When using a proxy, hostnames are resolved on the proxy server instead of on the
client. You can change this behavior by setting the
<<logstash-proxy-use-local-resolver,`proxy_use_local_resolver`>> option.
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.logstash:
hosts: ["remote-host:5044"]
proxy_url: socks5://user:password@socks5-proxy:2233
------------------------------------------------------------------------------
[[logstash-proxy-use-local-resolver]]
===== `proxy_use_local_resolver`
The `proxy_use_local_resolver` option determines if Logstash hostnames are
resolved locally when using a proxy. The default value is false which means
that when a proxy is used the name resolution occurs on the proxy server.
[[logstash-index]]
===== `index`
The index root name to write events to. The default is the Beat name. For
example +"{beatname_lc}"+ generates +"[{beatname_lc}-]{version}-YYYY.MM.DD"+
indices (for example, +"{beatname_lc}-{version}-2017.04.26"+).
===== `ssl`
Configuration options for SSL parameters like the root CA for Logstash connections. See
<<configuration-ssl>> for more information. To use SSL, you must also configure the
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash] to use SSL/TLS.
===== `timeout`
The number of seconds to wait for responses from the Logstash server before timing out. The default is 30 (seconds).
===== `max_retries`
ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="winlogbeat")]
{beatname_uc} ignores the `max_retries` setting and retries indefinitely.
endif::[]
ifeval::[("{beatname_lc}"!="filebeat") and ("{beatname_lc}"!="winlogbeat")]
The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.
Set `max_retries` to a value less than 0 to retry until all events are published.
The default is 3.
endif::[]
===== `bulk_max_size`
The maximum number of events to bulk in a single Logstash request. The default is 2048.
If the Beat sends single events, the events are collected into batches. If the Beat publishes
a large batch of events (larger than the value specified by `bulk_max_size`), the batch is
split.
Specifying a larger batch size can improve performance by lowering the overhead of sending events.
However big batch sizes can also increase processing times, which might result in
API errors, killed connections, timed-out publishing requests, and, ultimately, lower
throughput.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.
===== `slow_start`
If enabled only a subset of events in a batch of events is transferred per transaction.
The number of events to be sent increases up to `bulk_max_size` if no error is encountered.
On error the number of events per transaction is reduced again.
The default is `false`.
===== `backoff.init`
The number of seconds to wait before trying to reconnect to Logstash after
a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to
reconnect. If the attempt fails, the backoff timer is increased exponentially up
to `backoff.max`. After a successful connection, the backoff timer is reset. The
default is 1s.
===== `backoff.max`
The maximum number of seconds to wait before attempting to connect to
Logstash after a network error. The default is 60s.
[[kafka-output]]
=== Configure the Kafka output
++++
<titleabbrev>Kafka</titleabbrev>
++++
The Kafka output sends the events to Apache Kafka.
Example configuration:
[source,yaml]
------------------------------------------------------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
# message topic selection + partitioning
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
------------------------------------------------------------------------------
NOTE: Events bigger than <<kafka-max_message_bytes,`max_message_bytes`>> will be dropped. To avoid this problem, make sure {beatname_uc} does not generate events bigger than <<kafka-max_message_bytes,`max_message_bytes`>>.
==== Compatibility
This output works with all Kafka versions in between 0.11 and 2.0.0. Older versions
might work as well, but are not supported.
==== Configuration options
You can specify the following options in the `kafka` section of the +{beatname_lc}.yml+ config file:
===== `enabled`
The `enabled` config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true.
===== `hosts`
The list of Kafka broker addresses from where to fetch the cluster metadata.
The cluster metadata contain the actual Kafka brokers events are published to.
===== `version`
Kafka version ${beatname_lc} is assumed to run against. Defaults to 1.0.0.
Event timestamps will be added, if version 0.10.0.0+ is enabled.
Valid values are all kafka releases in between `0.8.2.0` and `1.1.1`.
===== `username`
The username for connecting to Kafka. If username is configured, the password
must be configured as well. Only SASL/PLAIN is supported.
===== `password`
The password for connecting to Kafka.
[[topic-option-kafka]]
===== `topic`
The Kafka topic used for produced events.
You can set the topic dynamically by using a format string to access any
event field. For example, this configuration uses a custom field,
`fields.log_topic`, to set the topic for each event:
[source,yaml]
-----
topic: '%{[fields.log_topic]}'
-----
TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
See the <<topics-option-kafka,`topics`>> setting for other ways to set the
topic dynamically.
[[topics-option-kafka]]
===== `topics`
An array of topic selector rules. Each rule specifies the `topic` to use for
events that match the rule. During publishing, {beatname_uc} sets the `topic`
for each event based on the first matching rule in the array. Rules
can contain conditionals, format string-based fields, and name mappings. If the
`topics` setting is missing or no rule matches, the
<<topic-option-kafka,`topic`>> field is used.
Rule settings:
*`topic`*:: The topic format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule
fails.
*`mappings`*:: A dictionary that takes the value returned by `topic` and maps it
to a new name.
*`default`*:: The default string value to use if `mappings` does not find a
match.
*`when`*:: A condition that must succeed in order to execute the current rule.
All the <<conditions,conditions>> supported by processors are also supported
here.
===== `key`
Optional Kafka event key. If configured, the event key must be unique and can be extracted from the event using a format string.
===== `partition`
Kafka output broker event partitioning strategy. Must be one of `random`,
`round_robin`, or `hash`. By default the `hash` partitioner is used.
*`random.group_events`*: Sets the number of events to be published to the same
partition, before the partitioner selects a new partition by random. The
default value is 1 meaning after each event a new partition is picked randomly.
*`round_robin.group_events`*: Sets the number of events to be published to the
same partition, before the partitioner selects the next partition. The default
value is 1 meaning after each event the next partition will be selected.
*`hash.hash`*: List of fields used to compute the partitioning hash value from.
If no field is configured, the events `key` value will be used.
*`hash.random`*: Randomly distribute events if no hash or key value can be computed.
All partitioners will try to publish events to all partitions by default. If a
partition's leader becomes unreachable for the beat, the output might block. All
partitioners support setting `reachable_only` to overwrite this
behavior. If `reachable_only` is set to `true`, events will be published to
available partitions only.
NOTE: Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed.
===== `client_id`
The configurable ClientID used for logging, debugging, and auditing purposes. The default is "beats".
===== `worker`
The number of concurrent load-balanced Kafka output workers.
===== `codec`
Output codec configuration. If the `codec` section is missing, events will be json encoded.
See <<configuration-output-codec>> for more information.
===== `metadata`
Kafka metadata update settings. The metadata do contain information about
brokers, topics, partition, and active leaders to use for publishing.
*`refresh_frequency`*:: Metadata refresh interval. Defaults to 10 minutes.
*`retry.max`*:: Total number of metadata update retries when cluster is in middle of leader election. The default is 3.
*`retry.backoff`*:: Waiting time between retries during leader elections. Default is 250ms.
===== `max_retries`
ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="winlogbeat")]
{beatname_uc} ignores the `max_retries` setting and retries indefinitely.
endif::[]
ifeval::[("{beatname_lc}"!="filebeat") and ("{beatname_lc}"!="winlogbeat")]
The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.
Set `max_retries` to a value less than 0 to retry until all events are published.
The default is 3.
endif::[]
===== `bulk_max_size`
The maximum number of events to bulk in a single Kafka request. The default is 2048.
===== `timeout`
The number of seconds to wait for responses from the Kafka brokers before timing
out. The default is 30 (seconds).
===== `broker_timeout`
The maximum duration a broker will wait for number of required ACKs. The default is 10s.
===== `channel_buffer_size`
Per Kafka broker number of messages buffered in output pipeline. The default is 256.
===== `keep_alive`
The keep-alive period for an active network connection. If 0s, keep-alives are disabled. The default is 0 seconds.
===== `compression`
Sets the output compression codec. Must be one of `none`, `snappy`, `lz4` and `gzip`. The default is `gzip`.
===== `compression_level`
Sets the compression level used by gzip. Setting this value to 0 disables compression.
The compression level must be in the range of 1 (best speed) to 9 (best compression).
Increasing the compression level will reduce the network usage but will increase the cpu usage.
The default value is 4.
[[kafka-max_message_bytes]]
===== `max_message_bytes`
The maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker's `message.max.bytes`.
===== `required_acks`
The ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1.
Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error.
===== `ssl`
Configuration options for SSL parameters like the root CA for Kafka connections. See
<<configuration-ssl>> for more information.
ifndef::no-redis-output[]
[[redis-output]]
=== Configure the Redis output
++++
<titleabbrev>Redis</titleabbrev>
++++
The Redis output inserts the events into a Redis list or a Redis channel.
This output plugin is compatible with
the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html[Redis input plugin] for Logstash.
Example configuration:
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.redis:
hosts: ["localhost"]
password: "my_password"
key: "{beatname_lc}"
db: 0
timeout: 5
------------------------------------------------------------------------------
==== Compatibility
This output works with Redis 3.2.4.
==== Configuration options
You can specify the following options in the `redis` section of the +{beatname_lc}.yml+ config file:
===== `enabled`
The enabled config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true.
===== `hosts`
The list of Redis servers to connect to. If load balancing is enabled, the events are
distributed to the servers in the list. If one server becomes unreachable, the events are
distributed to the reachable servers only. You can define each Redis server by specifying
`HOST` or `HOST:PORT`. For example: `"192.15.3.2"` or `"test.redis.io:12345"`. If you
don't specify a port number, the value configured by `port` is used.
===== `port`
deprecated[5.0.0]
The Redis port to use if `hosts` does not contain a port number. The default is 6379.
===== `index`
The index name added to the events metadata for use by Logstash. The default is "{beatname_lc}".
[[key-option-redis]]
===== `key`
The name of the Redis list or channel the events are published to. If not
configured, the value of the `index` setting is used.
You can set the key dynamically by using a format string to access any event
field. For example, this configuration uses a custom field, `fields.list`, to
set the Redis list key. If `fields.list` is missing, `fallback` is used:
["source","yaml"]
------------------------------------------------------------------------------
output.redis:
hosts: ["localhost"]
key: "%{[fields.list]:fallback}"
------------------------------------------------------------------------------
TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
See the <<keys-option-redis,`keys`>> setting for other ways to set the key
dynamically.
[[keys-option-redis]]
===== `keys`
An array of key selector rules. Each rule specifies the `key` to use for events
that match the rule. During publishing, {beatname_uc} uses the first matching
rule in the array. Rules can contain conditionals, format string-based fields,
and name mappings. If the `keys` setting is missing or no rule matches, the
<<key-option-redis,`key`>> setting is used.
Rule settings:
*`index`*:: The key format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule
fails.
*`mappings`*:: A dictionary that takes the value returned by `key` and maps it to
a new name.
*`default`*:: The default string value to use if `mappings` does not find a match.
*`when`*:: A condition that must succeed in order to execute the current rule.
All the <<conditions,conditions>> supported by processors are also supported
here.
Example `keys` settings:
["source","yaml"]
------------------------------------------------------------------------------
output.redis:
hosts: ["localhost"]
key: "default_list"
keys:
- key: "info_list" # send to info_list if `message` field contains INFO
when.contains:
message: "INFO"
- key: "debug_list" # send to debug_list if `message` field contains DEBUG
when.contains:
message: "DEBUG"
- key: "%{[fields.list]}"
mappings:
http: "frontend_list"
nginx: "frontend_list"
mysql: "backend_list"
------------------------------------------------------------------------------
===== `password`
The password to authenticate with. The default is no authentication.
===== `db`
The Redis database number where the events are published. The default is 0.
===== `datatype`
The Redis data type to use for publishing events.If the data type is `list`, the
Redis RPUSH command is used and all events are added to the list with the key defined under `key`.
If the data type `channel` is used, the Redis `PUBLISH` command is used and means that all events
are pushed to the pub/sub mechanism of Redis. The name of the channel is the one defined under `key`.
The default value is `list`.
===== `codec`
Output codec configuration. If the `codec` section is missing, events will be json encoded.
See <<configuration-output-codec>> for more information.
===== `host_topology`
deprecated[5.0.0]
The Redis host to connect to when using topology map support. Topology map support is disabled if this option is not set.
===== `password_topology`
deprecated[5.0.0]
The password to use for authenticating with the Redis topology server. The default is no authentication.
===== `db_topology`
deprecated[5.0.0]
The Redis database number where the topology information is stored. The default is 1.
===== `worker`
The number of workers to use for each host configured to publish events to Redis. Use this setting along with the
`loadbalance` option. For example, if you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).
===== `loadbalance`
If set to true and multiple hosts or workers are configured, the output plugin load balances published events onto all
Redis hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch
to another host if the currently selected one becomes unreachable. The default value is true.
===== `timeout`
The Redis connection timeout in seconds. The default is 5 seconds.
===== `backoff.init`
The number of seconds to wait before trying to reconnect to Redis after
a network error. After waiting `backoff.init` seconds, {beatname_uc} tries to
reconnect. If the attempt fails, the backoff timer is increased exponentially up
to `backoff.max`. After a successful connection, the backoff timer is reset. The
default is 1s.
===== `backoff.max`
The maximum number of seconds to wait before attempting to connect to
Redis after a network error. The default is 60s.
===== `max_retries`
ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="winlogbeat")]
{beatname_uc} ignores the `max_retries` setting and retries indefinitely.
endif::[]
ifeval::["{beatname_lc}"!="filebeat" and "{beatname_lc}"!="winlogbeat"]
The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.
Set `max_retries` to a value less than 0 to retry until all events are published.
The default is 3.
endif::[]
===== `bulk_max_size`
The maximum number of events to bulk in a single Redis request or pipeline. The default is 2048.
If the Beat sends single events, the events are collected into batches. If the
Beat publishes a large batch of events (larger than the value specified by
`bulk_max_size`), the batch is split.
Specifying a larger batch size can improve performance by lowering the overhead
of sending events. However big batch sizes can also increase processing times,
which might result in API errors, killed connections, timed-out publishing
requests, and, ultimately, lower throughput.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.
===== `ssl`
Configuration options for SSL parameters like the root CA for Redis connections
guarded by SSL proxies (for example https://www.stunnel.org[stunnel]). See
<<configuration-ssl>> for more information.
===== `proxy_url`
The URL of the SOCKS5 proxy to use when connecting to the Redis servers. The
value must be a URL with a scheme of `socks5://`. You cannot use a web proxy
because the protocol used to communicate with Redis is not based on HTTP.
If the SOCKS5 proxy server requires client authentication, you can embed
a username and password in the URL.
When using a proxy, hostnames are resolved on the proxy server instead of on the
client. You can change this behavior by setting the
<<redis-proxy-use-local-resolver,`proxy_use_local_resolver`>> option.
[[redis-proxy-use-local-resolver]]
===== `proxy_use_local_resolver`
This option determines whether Redis hostnames are resolved locally when using a proxy.
The default value is false, which means that name resolution occurs on the proxy server.
endif::[]
[[file-output]]
=== Configure the File output
++++
<titleabbrev>File</titleabbrev>
++++
The File output dumps the transactions into a file where each transaction is in a JSON format.
Currently, this output is used for testing, but it can be used as input for
Logstash.
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
output.file:
path: "/tmp/{beatname_lc}"
filename: {beatname_lc}
#rotate_every_kb: 10000
#number_of_files: 7
#permissions: 0600
------------------------------------------------------------------------------
==== Configuration options
You can specify the following options in the `file` section of the +{beatname_lc}.yml+ config file:
===== `enabled`
The enabled config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true.
[[path]]
===== `path`
The path to the directory where the generated files will be saved. This option is
mandatory.
===== `filename`
The name of the generated files. The default is set to the Beat name. For example, the files
generated by default for {beatname_uc} would be "{beatname_lc}", "{beatname_lc}.1", "{beatname_lc}.2", and so on.
===== `rotate_every_kb`
The maximum size in kilobytes of each file. When this size is reached, the files are
rotated. The default value is 10240 KB.
===== `number_of_files`
The maximum number of files to save under <<path,`path`>>. When this number of files is reached, the
oldest file is deleted, and the rest of the files are shifted from last to first.
The number of files must be between 2 and 1024. The default is 7.
===== `permissions`
Permissions to use for file creation. The default is 0600.
===== `codec`
Output codec configuration. If the `codec` section is missing, events will be json encoded.
See <<configuration-output-codec>> for more information.
[[console-output]]
=== Configure the Console output
++++
<titleabbrev>Console</titleabbrev>
++++
The Console output writes events in JSON format to stdout.
[source,yaml]
------------------------------------------------------------------------------
output.console:
pretty: true
------------------------------------------------------------------------------
==== Configuration options
You can specify the following options in the `console` section of the +{beatname_lc}.yml+ config file:
===== `pretty`
If `pretty` is set to true, events written to stdout will be nicely formatted. The default is false.
===== `codec`
Output codec configuration. If the `codec` section is missing, events will be json encoded using the `pretty` option.
See <<configuration-output-codec>> for more information.
===== `enabled`
The enabled config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true.
===== `bulk_max_size`
The maximum number of events to buffer internally during publishing. The default is 2048.
Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this
setting does not affect how events are published.
Setting `bulk_max_size` to values less than or equal to 0 disables the
splitting of batches. When splitting is disabled, the queue decides on the
number of events to be contained in a batch.
[[configure-cloud-id]]
=== Configure the output for the Elastic Cloud
++++
<titleabbrev>Cloud</titleabbrev>
++++
{beatname_uc} comes with two settings that simplify the output configuration
when used together with https://cloud.elastic.co/[Elastic Cloud]. When defined,
these setting overwrite settings from other parts in the configuration.
Example:
["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw=="
cloud.auth: "elastic:{pwd}"
------------------------------------------------------------------------------
These settings can be also specified at the command line, like this:
["source","sh",subs="attributes"]
------------------------------------------------------------------------------
{beatname_lc} -e -E cloud.id="<cloud-id>" -E cloud.auth="<cloud.auth>"
------------------------------------------------------------------------------
==== `cloud.id`
The Cloud ID, which can be found in the Elastic Cloud web console, is used by
{beatname_uc} to resolve the Elasticsearch and Kibana URLs. This setting
overwrites the `output.elasticsearch.hosts` and `setup.kibana.host` settings.
==== `cloud.auth`
When specified, the `cloud.auth` overwrites the `output.elasticsearch.username` and
`output.elasticsearch.password` settings. Because the Kibana settings inherit
the username and password from the Elasticsearch output, this can also be used
to set the `setup.kibana.username` and `setup.kibana.password` options.
[[configuration-output-codec]]
=== Change the output codec
For outputs that do not require a specific encoding, you can change the encoding
by using the codec configuration. You can specify either the `json` or `format`
codec. By default the `json` codec is used.
*`json.pretty`*: If `pretty` is set to true, events will be nicely formatted. The default is false.
*`json.escape_html`*: If `escape_html` is set to false, html symbols will not be escaped in strings. The default is true.
Example configuration that uses the `json` codec with pretty printing enabled to write events to the console:
[source,yaml]
------------------------------------------------------------------------------
output.console:
codec.json:
pretty: true
escape_html: false
------------------------------------------------------------------------------
*`format.string`*: Configurable format string used to create a custom formatted message.
Example configurable that uses the `format` codec to print the events timestamp and message field to console:
[source,yaml]
------------------------------------------------------------------------------
output.console:
codec.format:
string: '%{[@timestamp]} %{[message]}'
------------------------------------------------------------------------------
endif::[]
Reference in a new issue View git blame Copy permalink