72 lines
3 KiB
Text
72 lines
3 KiB
Text
|
//////////////////////////////////////////////////////////////////////////
|
||
|
|
//// This content is shared by all Elastic Beats. Make sure you keep the
|
||
|
|
//// descriptions here generic enough to work for all Beats that include
|
||
|
|
//// this file. When using cross references, make sure that the cross
|
||
|
|
//// references resolve correctly for any files that include this one.
|
||
|
|
//// Use the appropriate variables defined in the index.asciidoc file to
|
||
|
|
//// resolve Beat names: beatname_uc and beatname_lc.
|
||
|
|
//// Use the following include to pull this content into a doc file:
|
||
|
|
//// include::../../libbeat/docs/shared-config-ingest.asciidoc[]
|
||
|
|
//////////////////////////////////////////////////////////////////////////
|
||
|
|
|
||
|
|
[[configuring-ingest-node]]
|
||
|
|
== Parse data by using ingest node
|
||
|
|
|
||
|
|
When you use Elasticsearch for output, you can configure {beatname_uc} to use
|
||
|
|
{elasticsearch}/ingest.html[ingest node] to pre-process documents before the
|
||
|
|
actual indexing takes place in Elasticsearch. Ingest node is a convenient
|
||
|
|
processing option when you want to do some extra processing on your data, but
|
||
|
|
you do not require the full power of Logstash. For example, you can create an
|
||
|
|
ingest node pipeline in Elasticsearch that consists of one processor that
|
||
|
|
removes a field in a document followed by another processor that renames a
|
||
|
|
field.
|
||
|
|
|
||
|
|
After defining the pipeline in Elasticsearch, you simply configure {beatname_uc}
|
||
|
|
to use the pipeline. To configure {beatname_uc}, you specify the pipeline ID in
|
||
|
|
the `parameters` option under `elasticsearch` in the +{beatname_lc}.yml+ file:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
output.elasticsearch:
|
||
|
|
hosts: ["localhost:9200"]
|
||
|
|
pipeline: my_pipeline_id
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
For example, let's say that you've defined the following pipeline in a file
|
||
|
|
named `pipeline.json`:
|
||
|
|
|
||
|
|
[source,json]
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
{
|
||
|
|
"description": "Test pipeline",
|
||
|
|
"processors": [
|
||
|
|
{
|
||
|
|
"lowercase": {
|
||
|
|
"field": "beat.name"
|
||
|
|
}
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
To add the pipeline in Elasticsearch, you would run:
|
||
|
|
|
||
|
|
[source,shell]
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' -d@pipeline.json
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
Then in the +{beatname_lc}.yml+ file, you would specify:
|
||
|
|
|
||
|
|
[source,yaml]
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
output.elasticsearch:
|
||
|
|
hosts: ["localhost:9200"]
|
||
|
|
pipeline: "test-pipeline"
|
||
|
|
------------------------------------------------------------------------------
|
||
|
|
|
||
|
|
When you run {beatname_uc}, the value of `beat.name` is converted to lowercase before indexing.
|
||
|
|
|
||
|
|
For more information about defining a pre-processing pipeline, see the
|
||
|
|
{elasticsearch}/ingest.html[Ingest Node] documentation.
|