[[config-file-format]]
== Config file format

Beats config files are based on http://www.yaml.org[YAML], a file format that is
easier to read and write than other common data formats like XML or JSON.

In beats all YAML files start with a dictionary, an unordered collection of
name/value pairs. In addition to dictionaries, YAML also supports lists, numbers,
strings, and many other data types. All members of the same list or dictionary must
have the same indentation level.

Dictionaries are represented by simple `key: value` pairs all having the same
indentation level. The colon after `key` must be followed by a space.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
name: John Doe
age: 34
country: Canada
------------------------------------------------------------------------------

Lists are introduced by dashes `- `. All list members will be lines beginning
with `- ` at the same indentation level.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
- Red
- Green
- Blue
------------------------------------------------------------------------------

Lists and dictionaries are used in beats to build structured configurations.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
filebeat:
  inputs:
    - type: log
      paths:
        - /var/log/*.log
      multiline:
        pattern: '^['
        match: after
------------------------------------------------------------------------------

Lists and dictionaries can also be represented in abbreviated form. Abbreviated
form is somewhat similar to JSON using `{}` for dictionaries and `[]` for lists:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
person: \{name: "John Doe", age: 34, country: "Canada"}
colors: ["Red", "Green", "Blue"]
------------------------------------------------------------------------------

The following topics provide more detail to help you understand and work with config files in YAML:

* <<config-file-format-namespacing>>
* <<config-file-format-type>>
* <<config-file-format-env-vars>>
* <<config-gile-format-refs>>
* <<config-file-permissions>>
* <<config-file-format-cli>>
* <<config-file-format-tips>>

[[config-file-format-namespacing]]
=== Namespacing

All settings are structured using dictionaries and lists. Those are collapsed
into "namespaced" settings, by creating a setting using the full path of the
settings name and it's parent structures names, when reading the configuration
file.

For example this setting:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------

output:
  elasticsearch:
    index: 'beat-%{[beat.version]}-%{+yyyy.MM.dd}'

------------------------------------------------------------------------------

gets collapsed into `output.elasticsearch.index: 'beat-%{[beat.version]}-%{+yyyy.MM.dd}'`. The
full name of a setting is based on all parent structures involved.

Lists create numeric names starting with 0.

For example this filebeat setting:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------

filebeat:
  inputs:
    - type: log

------------------------------------------------------------------------------

Gets collapsed into `filebeat.inputs.0.type: log`.

Alternatively to using indentation, setting names can be used in collapsed form too.

Note: having two settings with same fully collapsed path is invalid.

Simple filebeat example with partially collapsed setting names and use of compact form:


["source","yaml",subs="attributes"]
------------------------------------------------------------------------------

filebeat.inputs:
- type: log
  paths: ["/var/log/*.log"]
  multiline.pattern: '^['
  multiline.match: after

output.elasticsearch.hosts: ["http://localhost:9200"]

------------------------------------------------------------------------------

[[config-file-format-type]]
=== Config file data types

Values of configuration settings are interpreted as required by beats.
If a value can not be correctly interpreted as the required type - for example a
string is given when a number is required - the beat will fail to start up.

==== Boolean

Boolean values can be either `true` or `false`. Alternative names for `true` are
`yes` and `on`. Instead of `false` the values `no` and `off` can be used.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
enabled: true
disabled: false
------------------------------------------------------------------------------

==== Number

Number values require you to enter the number to use without using single or
double quotes. Some settings only support a restricted number range though.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
integer: 123
negative: -1
float: 5.4
------------------------------------------------------------------------------

==== String

In YAML[http://www.yaml.org], multiple styles of string definitions are supported:
 double-quoted, single-quoted, unquoted.

The double-quoted style is specified by surrounding the string with `"`. This
style provides support for escaping unprintable characters using `\`, but comes
at the cost of having to escape `\` and `"` characters.

The single-quoted style is specified by surrounding the string with `'`. This
style supports no escaping (use `''` to quote a single quote). Only printable
characters can be used when using this form.

Unquoted style requires no quotes, but does not support any escaping plus care
needs to be taken to not use any symbol that has a special meaning in YAML.

Note: Single-quoted style is recommended when defining regular expressions,
event format strings, windows file paths, or non-alphabetical symbolic characters.

==== Duration

Durations require a numeric value with optional fraction and required unit.
Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based
on durations can be disabled by using zero or negative durations.

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
duration1: 2.5s
duration2: 6h
duration_disabled: -1s
------------------------------------------------------------------------------

==== Regular expression

Regular expressions are special strings getting compiled into regular
expressions at load time.

As regular expressions and YAML use `\` for escaping
characters in strings, it's highly recommended to use single quoted strings when
defining regular expressions. When single quoted strings are used, `\` character
is not interpreted by YAML parser as escape symbol.

==== Format String (sprintf)

Format strings enable you to refer to event field values creating a string based
on the current event being processed. Variable expansions are enclosed in
expansion braces `%{<accessor>:default value}`. Event fields are accessed using
field references `[fieldname]`. Optional default values can be specified in case the
field name is missing from the event.

You can also format time stored in the
`@timestamp` field using the `+FORMAT` syntax where FORMAT is a valid https://godoc.org/github.com/elastic/beats/libbeat/common/dtfmt[time
format].

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
constant-format-string: 'constant string'
field-format-string: '%{[fieldname]} string'
format-string-with-date: '%{[fieldname]}-%{+yyyy.MM.dd}'
------------------------------------------------------------------------------


[[config-file-format-env-vars]]
=== Environment variables

include::shared-env-vars.asciidoc[]

[[config-gile-format-refs]]
=== Reference variables

Beats settings can reference other settings splicing multiple optionally custom
named settings into new values. References use the same syntax as
<<config-file-format-env-vars>> do. Only fully collapsed setting names can be
referenced to.

For example the filebeat registry file defaults to:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
filebeat.registry: $\{path.data}/registry
------------------------------------------------------------------------------

With `path.data` being an implicit config setting, that is overridable from
command line, as well as in the configuration file.

Example referencing `es.host` in `output.elasticsearch.hosts`:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
es.host: '$\{ES_HOST:localhost}'

output.elasticsearch:
  hosts: ['http://$\{es.host}:9200']
------------------------------------------------------------------------------

Introducing `es.host`, the host can be overwritten from command line using
`-E es.host=another-host`.

Plain references, having no default value and are not spliced with other
references or strings can reference complete namespaces.

These setting with duplicate content:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
namespace1:
  subnamespace:
    host: localhost
    sleep: 1s

namespace2:
  subnamespace:
    host: localhost
    sleep: 1s
------------------------------------------------------------------------------

can be rewritten to

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
namespace1: $\{shared}
namespace2: $\{shared}

shared:
  subnamespace:
    host: localhost
    sleep: 1s
------------------------------------------------------------------------------

when using plain references.


[[config-file-permissions]]
=== Config file ownership and permissions

NOTE: This section does not apply to Windows or other non-POSIX operating systems.

On systems with POSIX file permissions, all Beats configuration files are
subject to ownership and file permission checks. The purpose of these checks is
to prevent unauthorized users from providing or modifying configurations that
are run by the Beat. The owner of the configuration files must be either `root`
or the user who is executing the Beat process. The permissions on each file must
disallow writes by anyone other than the owner.

When installed via an RPM or DEB package, the config file at
`/etc/{beatname}/{beatname}.yml` will have the proper owner and permissions. The
file is owned by `root` and has file permissions of `0644` (`-rw-r--r--`).

You may encounter the following errors if your config file fails these checks:

["source","sh"]
--------------------------------------------------------------------------------
Exiting: error loading config file: config file ("{beatname}.yml") must be
owned by the beat user (uid=501) or root
--------------------------------------------------------------------------------

To correct this problem you can use either `chown root {beatname}.yml` or
`chown 501 {beatname}.yml` to change the owner of the configuration file.

["source","sh"]
--------------------------------------------------------------------------------
Exiting: error loading config file: config file ("{beatname}.yml") can only be
writable by the owner but the permissions are "-rw-rw-r--" (to fix the
permissions use: 'chmod go-w /etc/{beatname}/{beatname}.yml')
--------------------------------------------------------------------------------

To correct this problem, use `chmod go-w /etc/{beatname}/{beatname}.yml` to
remove write privileges from anyone other than the owner.

Other config files, such as the files in the `modules.d` directory, are subject
to the same ownership and file permission checks. 

==== Disabling strict permission checks

You can disable strict permission checks from the command line by using
`--strict.perms=false`, but we strongly encourage you to leave the checks enabled.

[[config-file-format-cli]]
=== Command line arguments

Config files to load are set using the `-c` flag on command line. If no flag is
given, a beat and OS-specific default file path will be assumed.

You can specify multiple configuration files by repeating the `-c` flag. You can
use this, for example, for setting defaults in a base configuration file, and
overwrite settings via local configuration files.

In addition to overwriting settings using multiple configuration files,
individual settings can be overwritten using `-E <setting>=<value>`. The
`<value>` can be either a single value or a complex object, such as a list or
dictionary.

For example, given the following configuration:

["source","yaml"]
--------------------------------------------------------------------------------
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  username: username
  password: password
--------------------------------------------------------------------------------

You can disable the Elasticsearch output and write all events to the console by
setting:

["source","sh"]
--------------------------------------------------------------------------------
-E output='{elasticsearch.enabled: false, console.pretty: true}'
--------------------------------------------------------------------------------

Any complex objects that you specify at the command line are merged with the
original configuration, and the following configuration is passed to the Beat:

["source","yaml"]
--------------------------------------------------------------------------------
output.elasticsearch:
  enabled: false
  hosts: ["http://localhost:9200"]
  username: username
  password: password

output.console:
  pretty: true
--------------------------------------------------------------------------------


[[config-file-format-tips]]
=== YAML tips and gotchas

:allplatforms:
include::yaml.asciidoc[]
