youtubebeat/vendor/github.com/elastic/beats/filebeat/docs/autodiscover-hints.asciidoc

151 lines
6.1 KiB
Text
Raw Normal View History

2018-11-18 11:08:38 +01:00
{beatname_uc} supports autodiscover based on hints from the provider. The hints system looks for
hints in Kubernetes Pod annotations or Docker labels that have the prefix `co.elastic.logs`. As soon as
the container starts, {beatname_uc} will check if it contains any hints and launch the proper config for
it. Hints tell {beatname_uc} how to get logs for the given container. By default logs will be retrieved
from the container using the `docker` input. You can use hints to modify this behavior. This is the full
list of supported hints:
[float]
===== `co.elastic.logs/disable`
Filebeat gets logs from all containers by default, you can set this hint to `true` to ignore
the output of the container. Filebeat won't read or send logs from it.
[float]
===== `co.elastic.logs/multiline.*`
Multiline settings. See <<multiline-examples>> for a full list of all supported options.
[float]
===== `co.elastic.logs/include_lines`
A list of regular expressions to match the lines that you want {beatname_uc} to include.
See <<configuration-filebeat-options>> for more info.
[float]
===== `co.elastic.logs/exclude_lines`
A list of regular expressions to match the lines that you want {beatname_uc} to exclude.
See <<configuration-filebeat-options>> for more info.
[float]
===== `co.elastic.logs/module`
Instead of using raw `docker` input, specifies the module to use to parse logs from the container. See
<<filebeat-modules>> for the list of supported modules.
[float]
===== `co.elastic.logs/fileset`
When module is configured, map container logs to module filesets. You can either configure
a single fileset like this:
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
co.elastic.logs/fileset: access
-------------------------------------------------------------------------------------
Or configure a fileset per stream in the container (stdout and stderr):
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
co.elastic.logs/fileset.stdout: access
co.elastic.logs/fileset.stderr: error
-------------------------------------------------------------------------------------
[float]
===== `co.elastic.logs/raw`
When an entire input/module configuration needs to be completely set the `raw` hint can be used. You can provide a
stringified JSON of the input configuration. `raw` overrides every other hint and can be used to create bot a single or
a list of configurations.
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
co.elastic.logs/raw: "[{\"containers\":{\"ids\":[\"${data.container.id}\"]},\"multiline\":{\"negate\":\"true\",\"pattern\":\"^test\"},\"type\":\"docker\"}]"
-------------------------------------------------------------------------------------
[float]
===== `co.elastic.logs/processors`
Define a processor to be added to the {beatname_uc} input/module configuration. See <<filtering-and-enhancing-data>> for the list
of supported processors.
In order to provide ordering of the processor definition, numbers can be provided. If not, the hints builder will do
arbitrary ordering:
["source","yaml"]
-------------------------------------------------------------------------------------
co.elastic.logs/processors.1.dissect.tokenizer: "%{key1} %{key2}"
co.elastic.logs/processors.dissect.tokenizer: "%{key2} %{key1}"
-------------------------------------------------------------------------------------
In the above sample the processor definition tagged with `1` would be executed first.
[float]
==== Kubernetes
Kubernetes autodiscover provider supports hints in Pod annotations. To enable it just set `hints.enabled`:
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
-------------------------------------------------------------------------------------
You can annotate Kubernetes Pods with useful info to spin up {beatname_uc} inputs or modules:
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
annotations:
co.elastic.logs/multiline.pattern: '^\['
co.elastic.logs/multiline.negate: true
co.elastic.logs/multiline.match: after
-------------------------------------------------------------------------------------
[float]
===== Multiple containers
When a pod has multiple containers, the settings are shared unless you put the container name in the
hint. For example, these hints configure multiline settings for all containers in the pod, but set a
specific `exclude_lines` hint for the container called `sidecar`.
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
annotations:
co.elastic.logs/multiline.pattern: '^\['
co.elastic.logs/multiline.negate: true
co.elastic.logs/multiline.match: after
co.elastic.logs.sidecar/exclude_lines: '^DBG'
-------------------------------------------------------------------------------------
[float]
==== Docker
Docker autodiscover provider supports hints in labels. To enable it just set `hints.enabled`:
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
-------------------------------------------------------------------------------------
You can label Docker containers with useful info to spin up {beatname_uc} inputs, for example:
["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
co.elastic.logs/module: nginx
co.elastic.logs/fileset.stdout: access
co.elastic.logs/fileset.stderr: error
-------------------------------------------------------------------------------------
The above labels configure {beatname_uc} to use the Nginx module to harvest logs for this container.
Access logs will be retrieved from stdout stream, and error logs from stderr.