217 lines
11 KiB
Text
217 lines
11 KiB
Text
|
[[multiline-examples]]
|
||
|
|
== Manage multiline messages
|
||
|
|
|
||
|
|
The files harvested by {beatname_uc} may contain messages that span multiple
|
||
|
|
lines of text. For example, multiline messages are common in files that contain
|
||
|
|
Java stack traces. In order to correctly handle these multiline events, you need
|
||
|
|
to configure `multiline` settings in the +{beatname_lc}.yml+ file to specify
|
||
|
|
which lines are part of a single event.
|
||
|
|
|
||
|
|
IMPORTANT: If you are sending multiline events to Logstash, use the options described here to handle multiline events
|
||
|
|
before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by
|
||
|
|
using the Logstash multiline codec) may result in the mixing of streams and corrupted data.
|
||
|
|
|
||
|
|
Also read <<yaml-tips>> and <<regexp-support>> to avoid common mistakes.
|
||
|
|
|
||
|
|
[float]
|
||
|
|
[[multiline]]
|
||
|
|
=== Configuration options
|
||
|
|
|
||
|
|
You can specify the following options in the +{beatname_lc}.inputs+ section of
|
||
|
|
the +{beatname_lc}.yml+ config file to control how {beatname_uc} deals with messages
|
||
|
|
that span multiple lines.
|
||
|
|
|
||
|
|
The following example shows how to configure {beatname_uc} to handle a multiline message where the first line of the message begins with a bracket (`[`).
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
multiline.pattern: '^\['
|
||
|
|
multiline.negate: true
|
||
|
|
multiline.match: after
|
||
|
|
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
{beatname_uc} takes all the lines that do not start with `[` and combines them with the previous line that does. For example, you could use this configuration to join the following lines of a multiline message into a single event:
|
||
|
|
|
||
|
|
["source","sh",subs="attributes,callouts"]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index]
|
||
|
|
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
|
||
|
|
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)
|
||
|
|
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)
|
||
|
|
at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
|
||
|
|
*`multiline.pattern`*:: Specifies the regular expression pattern to match. Note that the regexp patterns supported by {beatname_uc}
|
||
|
|
differ somewhat from the patterns supported by Logstash. See <<regexp-support>> for a list of supported regexp patterns.
|
||
|
|
Depending on how you configure other multiline options, lines that match the specified regular expression are considered
|
||
|
|
either continuations of a previous line or the start of a new multiline event. You can set the `negate` option to negate
|
||
|
|
the pattern.
|
||
|
|
|
||
|
|
*`multiline.negate`*:: Defines whether the pattern is negated. The default is `false`.
|
||
|
|
|
||
|
|
*`multiline.match`*:: Specifies how {beatname_uc} combines matching lines into an event. The settings are `after` or `before`. The behavior of these settings depends on what you specify for `negate`:
|
||
|
|
+
|
||
|
|
[options="header"]
|
||
|
|
|=======================
|
||
|
|
|Setting for `negate` | Setting for `match` | Result | Example `pattern: ^b`
|
||
|
|
|`false` | `after` | Consecutive lines that match the pattern are appended to the previous line that doesn't match. | image:./images/false-after-multi.png[Lines a b b c b b become "abb" and "cbb"]
|
||
|
|
|`false` | `before` | Consecutive lines that match the pattern are prepended to the next line that doesn't match. | image:./images/false-before-multi.png[Lines b b a b b c become "bba" and "bbc"]
|
||
|
|
|`true` | `after` | Consecutive lines that don't match the pattern are appended to the previous line that does match. | image:./images/true-after-multi.png[Lines b a c b d e become "bac" and "bde"]
|
||
|
|
|`true` | `before` | Consecutive lines that don't match the pattern are prepended to the next line that does match. | image:./images/true-before-multi.png[Lines a c b d e b become "acb" and "deb"]
|
||
|
|
|=======================
|
||
|
|
+
|
||
|
|
NOTE: The `after` setting is equivalent to `previous` in https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html[Logstash], and `before` is equivalent to `next`.
|
||
|
|
|
||
|
|
*`multiline.flush_pattern`*:: Specifies a regular expression, in which the current multiline will be flushed from memory, ending the multiline-message.
|
||
|
|
|
||
|
|
*`multiline.max_lines`*:: The maximum number of lines that can be combined into one event. If
|
||
|
|
the multiline message contains more than `max_lines`, any additional
|
||
|
|
lines are discarded. The default is 500.
|
||
|
|
|
||
|
|
*`multiline.timeout`*:: After the specified timeout, {beatname_uc} sends the multiline event even if no new pattern is found to start a new event. The default is 5s.
|
||
|
|
|
||
|
|
|
||
|
|
=== Examples of multiline configuration
|
||
|
|
|
||
|
|
The examples in this section cover the following use cases:
|
||
|
|
|
||
|
|
* Combining a Java stack trace into a single event
|
||
|
|
* Combining C-style line continuations into a single event
|
||
|
|
* Combining multiple lines from time-stamped events
|
||
|
|
|
||
|
|
[float]
|
||
|
|
==== Java stack traces
|
||
|
|
|
||
|
|
Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in
|
||
|
|
this example:
|
||
|
|
|
||
|
|
[source,java]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
Exception in thread "main" java.lang.NullPointerException
|
||
|
|
at com.example.myproject.Book.getTitle(Book.java:16)
|
||
|
|
at com.example.myproject.Author.getBookTitles(Author.java:25)
|
||
|
|
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
To consolidate these lines into a single event in {beatname_uc}, use the following multiline configuration:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
multiline.pattern: '^[[:space:]]'
|
||
|
|
multiline.negate: false
|
||
|
|
multiline.match: after
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
This configuration merges any line that begins with whitespace up to the previous line.
|
||
|
|
|
||
|
|
Here's a Java stack trace that presents a slightly more complex example:
|
||
|
|
|
||
|
|
["source","sh",subs="attributes,callouts"]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
Exception in thread "main" java.lang.IllegalStateException: A book has a null property
|
||
|
|
at com.example.myproject.Author.getBookIds(Author.java:38)
|
||
|
|
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
|
||
|
|
Caused by: java.lang.NullPointerException
|
||
|
|
at com.example.myproject.Book.getId(Book.java:22)
|
||
|
|
at com.example.myproject.Author.getBookIds(Author.java:35)
|
||
|
|
... 1 more
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
To consolidate these lines into a single event in {beatname_uc}, use the following multiline configuration:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
|
||
|
|
multiline.negate: false
|
||
|
|
multiline.match: after
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
In this example, the pattern matches the following lines:
|
||
|
|
|
||
|
|
* a line that begins with spaces followed by the word `at` or `...`
|
||
|
|
* a line that begins with the words `Caused by:`
|
||
|
|
|
||
|
|
[float]
|
||
|
|
==== Line continuations
|
||
|
|
|
||
|
|
Several programming languages use the backslash (`\`) character at the end of a line to denote that the line continues,
|
||
|
|
as in this example:
|
||
|
|
|
||
|
|
[source,c]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
printf ("%10.10ld \t %10.10ld \t %s\
|
||
|
|
%f", w, x, y, z );
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
To consolidate these lines into a single event in {beatname_uc}, use the following multiline configuration:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
multiline.pattern: '\\$'
|
||
|
|
multiline.negate: false
|
||
|
|
multiline.match: before
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
This configuration merges any line that ends with the `\` character with the line that follows.
|
||
|
|
|
||
|
|
[float]
|
||
|
|
==== Timestamps
|
||
|
|
|
||
|
|
Activity logs from services such as Elasticsearch typically begin with a timestamp, followed by information on the
|
||
|
|
specific activity, as in this example:
|
||
|
|
|
||
|
|
[source,shell]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
[2015-08-24 11:49:14,389][INFO ][env ] [Letha] using [1] data paths, mounts [[/
|
||
|
|
(/dev/disk1)]], net usable_space [34.5gb], net total_space [118.9gb], types [hfs]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
To consolidate these lines into a single event in {beatname_uc}, use the following multiline configuration:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
|
||
|
|
multiline.negate: true
|
||
|
|
multiline.match: after
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
This configuration uses the `negate: true` and `match: after` settings to specify that any line that does not match the
|
||
|
|
specified pattern belongs to the previous line.
|
||
|
|
|
||
|
|
[float]
|
||
|
|
==== Application events
|
||
|
|
|
||
|
|
Sometimes your application logs contain events, that begin and end with custom markers, such as the following example:
|
||
|
|
|
||
|
|
[source,shell]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
[2015-08-24 11:49:14,389] Start new event
|
||
|
|
[2015-08-24 11:49:14,395] Content of processing something
|
||
|
|
[2015-08-24 11:49:14,399] End event
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
To consolidate this as a single event in {beatname_uc}, use the following multiline configuration:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
multiline.pattern: 'Start new event'
|
||
|
|
multiline.negate: true
|
||
|
|
multiline.match: after
|
||
|
|
multiline.flush_pattern: 'End event'
|
||
|
|
-------------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
The `flush_pattern` option, specifies a regex at which the current multiline will be flushed. If you think of the `pattern` option specifying the beginning of an event, the `flush_pattern` option will specify the end or last line of the event.
|
||
|
|
|
||
|
|
=== Test your regexp pattern for multiline
|
||
|
|
|
||
|
|
To make it easier for you to test the regexp patterns in your multiline config, we've created a
|
||
|
|
https://play.golang.org/p/uAd5XHxscu[Go Playground]. You can simply plug in the regexp pattern along with
|
||
|
|
the `multiline.negate` setting that you plan to use, and paste a sample message between the content backticks (` `).
|
||
|
|
Then click Run, and you'll see which lines in the message match your specified configuration. For example:
|
||
|
|
|
||
|
|
image:images/go-playground.png[]
|
||
|
|
|
||
|
|
|