gaugendre/youtubebeat
1
0
Fork 0
Code Issues Pull requests Releases Activity
youtubebeat/vendor/github.com/elastic/beats/packetbeat/docs/packetbeat-options.asciidoc

1364 lines
47 KiB
Text
Raw Normal View History

Add generated youtubebeat files
2018-11-18 11:08:38 +01:00
[[configuration-interfaces]]
== Set traffic capturing options
There are two main ways of deploying Packetbeat:
* On dedicated servers, getting the traffic from mirror ports or tap devices.
* On your existing application servers.
The first option has the big advantage that there is no overhead of any kind on
your application servers. But it requires dedicated networking gear, which is
generally not available on cloud setups.
In both cases, the sniffing performance (reading packets passively from the network)
is very important. In the case of a dedicated server, better
sniffing performance means that less hardware is required. When Packetbeat is installed
on an existing application server, better sniffing performance means less overhead.
Currently Packetbeat has several options for traffic capturing:
* `pcap`, which uses the libpcap library and works on most platforms, but
it's not the fastest option.
* `af_packet`, which uses memory mapped sniffing. This option is faster than libpcap
and doesn't require a kernel module, but it's Linux-specific.
The `af_packet` option, also known as "memory-mapped sniffing," makes use of a
Linux-specific
http://lxr.free-electrons.com/source/Documentation/networking/packet_mmap.txt[feature].
This could be the optimal sniffing mode for both the dedicated server and
when Packetbeat is deployed on an existing application server.
The way it works is that both the kernel and the user space program map the
same memory zone, and a simple circular buffer is organized in this memory zone.
The kernel writes packets into the circular buffer, and the user space program
reads from it. The poll system call is used for getting a notification for the
first packet available, but the remaining available packets can be simply read
via memory access.
The `af_packet` sniffer can be further tuned to use more memory in exchange for
better performance. The larger the size of the circular buffer, the fewer
system calls are needed, which means that fewer CPU cycles are consumed. The default size
of the buffer is 30 MB, but you can increase it like this:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
------------------------------------------------------------------------------
[float]
=== Sniffing configuration options
You can specify the following options in the `packetbeat.interfaces` section
of the +{beatname_lc}.yml+ config file. Here is an example configuration:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: any
packetbeat.interfaces.snaplen: 1514
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
------------------------------------------------------------------------------
[float]
==== `device`
The network device to capture traffic from. The specified device is set automatically to promiscuous mode,
meaning that Packetbeat can capture traffic from other hosts on the same LAN.
Example:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: eth0
------------------------------------------------------------------------------
On Linux, you can specify `any` for the device, and Packetbeat captures all
messages sent or received by the server where Packetbeat is installed.
NOTE: When you specify `any` for the device, the interfaces are not set
to promiscuous mode.
The `device` option also accepts specifying the device by its index in the list of
devices available for sniffing. To obtain the list of available devices,
run Packetbeat with the following command:
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
packetbeat devices
----------------------------------------------------------------------
This command returns a list that looks something like the following:
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
0: en0 (No description available)
1: awdl0 (No description available)
2: bridge0 (No description available)
3: fw0 (No description available)
4: en1 (No description available)
5: en2 (No description available)
6: p2p0 (No description available)
7: en4 (No description available)
8: lo0 (No description available)
----------------------------------------------------------------------
The following example sets up sniffing on the
first interface in the list:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: 0
------------------------------------------------------------------------------
Specifying the index is especially useful on Windows where device names can be long.
[float]
==== `snaplen`
The maximum size of the packets to capture. The
default is 65535, which is large enough for almost all networks and interface
types. If you sniff on a physical network interface, the optimal setting is
the MTU size. On virtual interfaces, however, it's safer to accept the default value.
Example:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: eth0
packetbeat.interfaces.snaplen: 1514
------------------------------------------------------------------------------
[float]
==== `type`
Packetbeat supports three sniffer types:
* `pcap`, which uses the libpcap library and works on most platforms, but
it's not the fastest option.
* `af_packet`, which uses memory-mapped sniffing. This option is faster than libpcap
and doesn't require a kernel module, but it's Linux-specific.
The default sniffer type is `pcap`.
Here is an example configuration that specifies
the `af_packet` sniffing type:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
------------------------------------------------------------------------------
On Linux, if you are trying to optimize the CPU usage of Packetbeat, we
recommend trying the `af_packet` option.
If you use the `af_packet` sniffer, you can tune its behaviour by specifying the
following options:
[float]
==== `buffer_size_mb`
The maximum size of the shared memory buffer to use
between the kernel and user space. A bigger buffer usually results in lower CPU
usage, but consumes more memory. This setting is only available for the
`af_packet` sniffer type. The default is 30 MB.
Example:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: eth0
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
------------------------------------------------------------------------------
[float]
==== `with_vlans`
Packetbeat automatically generates a
https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[BPF] for capturing only
the traffic on ports where it expects to find known protocols.
For example, if you have configured port 80 for HTTP and port 3306 for MySQL,
Packetbeat generates the following BPF filter: `"port 80 or port 3306"`.
However, if the traffic contains https://en.wikipedia.org/wiki/IEEE_802.1Q[VLAN]
tags, the filter that Packetbeat generates is ineffective because the
offset is moved by four bytes. To fix this, you can enable the `with_vlans` option, which
generates a BPF filter that looks like this: `"port 80 or port 3306 or (vlan and (port 80 or port 3306))"`.
[float]
==== `bpf_filter`
Packetbeat automatically generates a
https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[BPF] for capturing only
the traffic on ports where it expects to find known protocols.
For example, if you have configured port 80 for HTTP and port 3306 for MySQL,
Packetbeat generates the following BPF filter: `"port 80 or port 3306"`.
You can use the `bpf_filter` setting to overwrite the generated BPF filter. For example:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.interfaces.device: eth0
packetbeat.interfaces.bpf_filter: "net 192.168.238.0/0 and port 80 or port 3306"
------------------------------------------------------------------------------
NOTE: This setting disables automatic generation of the BPF filter. If
you use this setting, it's your responsibility to keep the BPF filters in sync with the
ports defined in the `protocols` section.
[float]
==== `ignore_outgoing`
If the `ignore_outgoing` option is enabled, Packetbeat ignores all the
transactions initiated from the server running Packetbeat.
This is useful when two Packetbeat instances publish the same transactions. Because one Packetbeat
sees the transaction in its outgoing queue and the other sees it in its incoming
queue, you can end up with duplicate transactions. To remove the duplicates, you
can enable the `packetbeat.ignore_outgoing` option on one of the servers.
For example, in the following scenario, you see a 3-server architecture
where a Beat is installed on each server. t1 is the transaction exchanged between
Server1 and Server2, and t2 is the transaction between Server2 and Server3.
image:./images/option_ignore_outgoing.png[Beats Architecture]
By default, each transaction is indexed twice because Beat2
sees both transactions. So you would see the following published transactions
(when `ignore_outgoing` is false):
- Beat1: t1
- Beat2: t1 and t2
- Beat3: t2
To avoid duplicates, you can force your Beats to send only the incoming
transactions and ignore the transactions created by the local server. So you would
see the following published transactions (when `ignore_outgoing` is true):
- Beat1: none
- Beat2: t1
- Beat3: t2
[[configuration-flows]]
== Set up flows to monitor network traffic
You can configure Packetbeat to collect and report statistics on network flows.
A _flow_ is a group of packets sent over the same time period that share
common properties, such as the same source and destination address and protocol.
You can use this feature to analyze network traffic over specific protocols on
your network.
For each flow, Packetbeat reports the number of packets and the total number of
bytes sent from the source to the destination. Each flow event also contains
information about the source and destination hosts, such as their IP address.
For bi-directional flows, Packetbeat reports statistics for the reverse flow.
Packetbeat collects and reports statistics up to and including the transport
layer. See <<exported-fields-flows_event>> for more info about the exported
data.
Here's an example of flow events visualized in the Flows dashboard:
image:./images/flows.png[]
To configure flows, use the `packetbeat.flows` option in the +{beatname_lc}.yml+
config file. Flows are enabled by default. If this section is missing
from the configuration file, network flows are disabled.
[source,yaml]
--------------------------------------------------------------------------------
packetbeat.flows:
timeout: 30s
period: 10s
--------------------------------------------------------------------------------
Here’s an example of a flow information sent by Packetbeat. See
<<exported-fields-flows_event>> for a description of each field.
["source","json",subs="attributes"]
--------------------------------------------------------------------------------
{
"@timestamp": "2017-05-03T19:42:40.003Z",
"beat": {
"hostname": "host.example.com",
"name": "host.example.com",
"version": "{stack-version}"
},
"connection_id": "AQAAAAAAAAA=",
"dest": {
"ip": "192.0.2.0",
"mac": "fe:ff:20:00:01:00",
"port": 80,
"stats": {
"net_bytes_total": 19236,
"net_packets_total": 16
}
},
"final": false, <1>
"flow_id": "EQwA////DP//////FBgBAAEAAAEAAAD+/yAAAQCR/qDtQdDk3ywNUAABAAAAAAAAAA",
"last_time": "2017-05-03T19:42:24.151Z",
"source": {
"ip": "203.0.113.0",
"mac": "00:00:01:00:00:00",
"port": 3372,
"stats": {
"net_bytes_total": 1243,
"net_packets_total": 14
}
},
"start_time": "2017-05-03T19:42:24.151Z",
"transport": "tcp",
"type": "flow"
}
--------------------------------------------------------------------------------
<1> Packetbeat sets the `final` flag to `false` to indicate that the event
contains an intermediate report about a flow that it's tracking. When the flow
completes, Packetbeat sends one last event with `final` set to `true`. If you
want to aggregate sums of traffic, you need to filter on `final:true`, or use
some other technique, so that you get only the latest update from each flow.
You can disable intermediate reports by setting `period: -1s`.
[float]
=== Configuration options
You can specify the following options in the `packetbeat.flows` section of
the +{beatname_lc}.yml+ config file:
[float]
==== `enabled`
Enables flows support if set to true. Set to false to disable network flows
support without having to delete or comment out the flows section. The default
value is true.
[float]
==== `timeout`
Timeout configures the lifetime of a flow. If no packets have been received for
a flow within the timeout time window, the flow is killed and reported. The
default value is 30s.
[float]
==== `period`
Configure the reporting interval. All flows are reported at the very same point
in time. Periodical reporting can be disabled by setting the value to -1. If
disabled, flows are still reported once being timed out. The default value is
10s.
[float]
[[packetbeat-configuration-flows-fields]]
==== `fields`
Optional fields that you can specify to add additional information to the
output. For example, you might add fields that you can use for filtering log
data. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. By default, the fields that you specify here will be
grouped under a `fields` sub-dictionary in the output document. To store the
custom fields as top-level fields, set the `fields_under_root` option to true.
If a duplicate field is declared in the general configuration, then its value
will be overwritten by the value declared here.
[float]
==== `fields_under_root`
If this option is set to true, the custom <<packetbeat-configuration-flows-fields,fields>>
are stored as top-level fields in the output document instead of being grouped
under a `fields` sub-dictionary. If the custom field names conflict with other
field names added by Packetbeat, then the custom fields overwrite the other
fields.
[float]
==== `tags`
A list of tags that will be sent with the protocol event. This setting is optional.
[float]
==== `processors`
A list of processors to apply to the data generated by the protocol.
See <<filtering-and-enhancing-data>> for information about specifying
processors in your config.
[[configuration-protocols]]
== Specify which transaction protocols to monitor
The `packetbeat.protocols` section of the +{beatname_lc}.yml+ config file
contains configuration options for each supported protocol, including common
options like `enabled`, `ports`, `send_request`, `send_response`, and options
that are protocol-specific.
Currently, Packetbeat supports the following protocols:
include::./shared-protocol-list.asciidoc[]
Example configuration:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: icmp
enabled: true
- type: dhcpv4
ports: [67, 68]
- type: dns
ports: [53]
- type: http
ports: [80, 8080, 8000, 5000, 8002]
- type: amqp
ports: [5672]
- type: cassandra
ports: [9042]
- type: memcache
ports: [11211]
- type: mysql
ports: [3306]
- type: redis
ports: [6379]
- type: pgsql
ports: [5432]
- type: thrift
ports: [9090]
- type: tls
ports: [443]
------------------------------------------------------------------------------
[[common-protocol-options]]
=== Common protocol options
The following options are available for all protocols:
[float]
==== `enabled`
The enabled setting is a boolean setting to enable or disable protocols without having to comment out configuration sections. If set to false, the protocol is disabled.
The default value is true.
[float]
==== `ports`
Exception: For ICMP the option `enabled` has to be used instead.
The ports where Packetbeat will look to capture traffic for specific
protocols. Packetbeat installs a https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[BPF]
filter based on the ports specified in this section.
If a packet doesn't match the filter, very little CPU is required to discard
the packet. Packetbeat also uses the ports specified here to determine which
parser to use for each packet.
[float]
[[send-request-option]]
==== `send_request`
If this option is enabled, the raw message of the request (`request` field) is
sent to Elasticsearch. The default is false. This option is useful when you want to
index the whole request. Note that for HTTP, the body is not included by
default, only the HTTP headers.
[float]
[[send-response-option]]
==== `send_response`
If this option is enabled, the raw message of the response (`response` field)
is sent to Elasticsearch. The default is false. This option is useful when you
want to index the whole response. Note that for HTTP, the body is not included
by default, only the HTTP headers.
[float]
[[transaction-timeout-option]]
==== `transaction_timeout`
The per protocol transaction timeout. Expired transactions will no longer be correlated to incoming responses, but sent to Elasticsearch immediately.
[float]
[[packetbeat-configuration-fields]]
==== `fields`
Optional fields that you can specify to add additional information to the
output. For example, you might add fields that you can use for filtering log
data. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. By default, the fields that you specify here will be
grouped under a `fields` sub-dictionary in the output document. To store the
custom fields as top-level fields, set the `fields_under_root` option to true.
If a duplicate field is declared in the general configuration, then its value
will be overwritten by the value declared here.
[source,yaml]
--------------------------------------------------------------------------------
packetbeat.protocols:
- type: http
ports: [80]
fields:
service_id: nginx
--------------------------------------------------------------------------------
[float]
[[packetbeat-fields-under-root]]
==== `fields_under_root`
If this option is set to true, the custom <<packetbeat-configuration-fields,fields>>
are stored as top-level fields in the output document instead of being grouped
under a `fields` sub-dictionary. If the custom field names conflict with other
field names added by Packetbeat, then the custom fields overwrite the other
fields.
[float]
==== `tags`
A list of tags that will be sent with the transaction event. This setting is optional.
[float]
==== `processors`
A list of processors to apply to the data generated by the protocol.
See <<filtering-and-enhancing-data>> for information about specifying
processors in your config.
[[packetbeat-icmp-options]]
=== Capture ICMP traffic
++++
<titleabbrev>ICMP</titleabbrev>
++++
The `icmp` section of the +{beatname_lc}.yml+ config file specifies options for
the ICMP protocol. Here is a sample configuration section for ICMP:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: icmp
enabled: true
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `enabled`
The ICMP protocol can be enabled/disabled via this option. The default is true.
If enabled Packetbeat will generate the following BPF filter: `"icmp or icmp6"`.
[[packetbeat-dns-options]]
=== Capture DNS traffic
++++
<titleabbrev>DNS</titleabbrev>
++++
The `dns` section of the +{beatname_lc}.yml+ config file specifies configuration options for the DNS protocol. The DNS protocol supports processing DNS messages on TCP and UDP. Here is a sample configuration section for DNS:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: dns
ports: [53]
include_authorities: true
include_additionals: true
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `include_authorities`
If this option is enabled, dns.authority fields (authority resource records) are added to DNS events.
The default is false.
===== `include_additionals`
If this option is enabled, dns.additionals fields (additional resource records) are added to DNS events.
The default is false.
[[packetbeat-http-options]]
=== Capture HTTP traffic
++++
<titleabbrev>HTTP</titleabbrev>
++++
The HTTP protocol has several specific configuration options. Here is a
sample configuration for the `http` section of the +{beatname_lc}.yml+ config file:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: http
ports: [80, 8080, 8000, 5000, 8002]
hide_keywords: ["pass", "password", "passwd"]
send_headers: ["User-Agent", "Cookie", "Set-Cookie"]
split_cookie: true
real_ip_header: "X-Forwarded-For"
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `hide_keywords`
A list of query parameters that Packetbeat will automatically censor in
the transactions that it saves. The values associated with these parameters are replaced
by `'xxxxx'`. By default, no changes are made to the HTTP messages.
Packetbeat has this option because, unlike SQL traffic, which typically only contains the
hashes of the passwords, HTTP traffic may contain sensitive data. To reduce security risks,
you can configure this option to avoid sending the contents of certain HTTP POST
parameters.
WARNING: This option replaces query parameters from GET requests and top-level
parameters from POST requests. If sensitive data is encoded inside a
parameter that you don't specify here, Packetbeat cannot censor it. Also, note that if
you configure Packetbeat to save the raw request and response fields (see the <<send-request-option>>
and the <<send-response-option>> options), sensitive data may be present in those
fields.
===== `redact_authorization`
When this option is enabled, Packetbeat obscures the value of
`Authorization` and `Proxy-Authorization` HTTP headers, and censors
those strings in the response.
You should set this option to true for transactions that use Basic Authentication because
they may contain the base64 unencrypted username and password.
===== `send_headers`
A list of header names to capture and send to Elasticsearch. These
headers are placed under the `headers` dictionary in the resulting JSON.
===== `send_all_headers`
Instead of sending a white list of headers to Elasticsearch, you can
send all headers by setting this option to true. The default is false.
===== `include_body_for`
The list of content types for which Packetbeat exports the full HTTP payload. The HTTP body is available under
`http.request.body` and `http.response.body` for these Content-Types.
In addition, if <<send-response-option>> option is enabled, then the HTTP body is exported together with the HTTP
headers under `response` and if
<<send-request-option>> enabled, then `request` contains the entire HTTP message including the body.
In the following example, the HTML attachments of the HTTP responses are exported under the `response` field and under
`http.request.body` or `http.response.body`:
[source,yml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: http
ports: [80, 8080]
send_response: true
include_body_for: ["text/html"]
------------------------------------------------------------------------------
===== `split_cookie`
If the `Cookie` or `Set-Cookie` headers are sent, this option controls whether
they are split into individual values. For example, with this option set, an
HTTP response might result in the following JSON:
[source,json]
------------------------------------------------------------------------------
"response": {
"code": 200,
"headers": {
"connection": "close",
"content-language": "en",
"content-type": "text/html; charset=utf-8",
"date": "Fri, 21 Nov 2014 17:07:34 GMT",
"server": "gunicorn/19.1.1",
"set-cookie": { <1>
"csrftoken": "S9ZuJF8mvIMT5CL4T1Xqn32wkA6ZSeyf",
"expires": "Fri, 20-Nov-2015 17:07:34 GMT",
"max-age": "31449600",
"path": "/"
},
"vary": "Cookie, Accept-Language"
},
"phrase": "OK"
}
------------------------------------------------------------------------------
<1> Note that `set-cookie` is a map containing the cookie names as keys.
The default is false.
===== `real_ip_header`
The header field to extract the real IP from. This setting is useful when
you want to capture traffic behind a reverse proxy, but you want to get the geo-location
information. If this header is present and contains a valid IP addresses, the
information is used for the `real_ip` field.
===== `max_message_size`
If an individual HTTP message is larger than this setting (in bytes), it will be trimmed
to this size. Unless this value is very small (<1.5K), Packetbeat is able to still correctly
follow the transaction and create an event for it. The default is 10485760 (10 MB).
[[packetbeat-amqp-options]]
=== Capture AMQP traffic
++++
<titleabbrev>AMQP</titleabbrev>
++++
The `amqp` section of the +{beatname_lc}.yml+ config file specifies configuration options for the AMQP 0.9.1
protocol. Here is a sample configuration:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: amqp
ports: [5672]
max_body_length: 1000
parse_headers: true
parse_arguments: false
hide_connection_information: true
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `max_body_length`
The maximum size in bytes of the message displayed in the request or response fields.
Messages that are bigger than the specified size are truncated. Use this option to
avoid publishing huge messages when <<send-request-option>> or <<send-request-option>>
is enabled. The default is 1000 bytes.
===== `parse_headers`
If set to true, Packetbeat parses the additional arguments specified in the headers field
of a message. Those arguments are key-value pairs that specify information such as the
content type of the message or the message priority. The default is true.
===== `parse_arguments`
If set to true, Packetbeat parses the additional arguments specified in AMQP methods.
Those arguments are key-value pairs specified by the user and can be of any length.
The default is true.
===== `hide_connection_information`
If set to false, the connection layer methods of the protocol are also
displayed, such as the opening and closing of connections and channels by clients,
or the quality of service negotiation. The default is true.
[[configuration-cassandra]]
=== Capture Cassandra traffic
++++
<titleabbrev>Cassandra</titleabbrev>
++++
The following settings are specific to the Cassandra protocol. Here is a sample
configuration for the `cassandra` section of the +{beatname_lc}.yml+ config file:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: cassandra
send_request_header: true
send_response_header: true
compressor: "snappy"
ignored_ops: ["SUPPORTED","OPTIONS"]
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `send_request_header`
If this option is enabled, the raw message of the response (`cassandra_request.request_headers` field)
is sent to Elasticsearch. The default is true. enable `send_request` first before enable this option.
===== `send_response_header`
If this option is enabled, the raw message of the response (`cassandra_response.response_headers` field)
is included in published events. The default is true. enable `send_response` first before enable this option.
===== `ignored_ops`
This option indicates which Operator/Operators captured will be ignored. currently support:
`ERROR` ,`STARTUP` ,`READY` ,`AUTHENTICATE` ,`OPTIONS` ,`SUPPORTED` ,
`QUERY` ,`RESULT` ,`PREPARE` ,`EXECUTE` ,`REGISTER` ,`EVENT` ,
`BATCH` ,`AUTH_CHALLENGE`,`AUTH_RESPONSE` ,`AUTH_SUCCESS` .
===== `compressor`
Configures the default compression algorithm being used to uncompress compressed frames by name. Currently only `snappy` is can be configured.
By default no compressor is configured.
[[packetbeat-memcache-options]]
=== Capture Memcache traffic
++++
<titleabbrev>Memcache</titleabbrev>
++++
The `memcache` section of the +{beatname_lc}.yml+ config file specifies configuration options for the memcache
protocol. Here is a sample configuration section for memcache:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: memcache
ports: [11211]
parseunknown: false
maxvalues: 0
maxbytespervalue: 100
transaction_timeout: 200
udptransactiontimeout: 200
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `parseunknown`
When this option is enabled, it forces the memcache text protocol parser to accept unknown commands.
NOTE: The unknown commands MUST NOT contain a data part.
===== `maxvalues`
The maximum number of values to store in the message (multi-get).
All values will be base64 encoded.
The possible settings for this option are:
* `maxvalue: -1`, which stores all values (text based protocol multi-get)
* `maxvalue: 0`, which stores no values (default)
* `maxvalue: N`, which stores up to N values
===== `maxbytespervalue`
The maximum number of bytes to be copied for each value element.
NOTE: Values will be base64 encoded, so the actual size in the JSON document will be 4 times the value that
you specify for `maxbytespervalue`.
===== `udptransactiontimeout`
The transaction timeout in milliseconds. The defaults is 10000 milliseconds.
NOTE: Quiet messages in UDP binary protocol get responses only if there is an error.
The memcache protocol analyzer will wait for the number of milliseconds specified by
`udptransactiontimeout` before publishing quiet messages. Non-quiet messages or
quiet requests with an error response are published immediately.
[[packetbeat-mysql-pgsql-options]]
=== Capture MySQL and PgSQL traffic
++++
<titleabbrev>MySQL and PgSQL</titleabbrev>
++++
The `mysql` and `pgsql` sections of the +{beatname_lc}.yml+ config file specify configuration options for the MySQL
and PgSQL protocols.
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: mysql
ports: [3306]
- type: pgsql
ports: [5432]
------------------------------------------------------------------------------
==== Configuration options
Also see <<common-protocol-options>>.
===== `max_rows`
The maximum number of rows from the SQL message to publish to Elasticsearch. The
default is 10 rows.
===== `max_row_length`
The maximum length in bytes of a row from the SQL message to publish to
Elasticsearch. The default is 1024 bytes.
[[configuration-thrift]]
=== Capture Thrift traffic
++++
<titleabbrev>Thrift</titleabbrev>
++++
https://thrift.apache.org/[Apache Thrift] is a communication protocol and RPC
framework initially created at Facebook. It is sometimes used in
http://martinfowler.com/articles/microservices.html[microservices]
architectures because it provides better performance when compared to the more
obvious HTTP/RESTful API choice, while still supporting a wide range of
programming languages and frameworks.
Packetbeat works based on a copy of the traffic, which means that you get
performance management features without having to modify your services in
any way and without any latency overhead. Packetbeat captures the transactions from the
network and indexes them in Elasticsearch so that they can be analyzed and
searched.
Packetbeat indexes the method, parameters, return value, and
exceptions of each Thrift-RPC call. You can search by and create statistics
based on any of these fields. Packetbeat automatically fills in the `status`
column with either `OK` or `Error`, so it's easy to find the problematic RPC calls.
A transaction is put into the `Error` state if it returned an exception.
Packetbeat also indexes the `responsetime` field so you can get performance
analytics and find the slow RPC calls.
Here is an example performance dashboard:
image:./images/thrift-dashboard.png[Thrift-RPC dashboard]
Thrift supports multiple http://en.wikipedia.org/wiki/Apache_Thrift[transport
and protocol types]. Currently Packetbeat supports the default `TSocket`
transport as well as the `TFramed` transport. From the protocol point of view,
Packetbeat currently supports only the default `TBinary` protocol.
Packetbeat also has several configuration options that allow you to get
the right balance between visibility, disk usage, and data protection. You can,
for example, choose to obfuscate all strings or to store the requests but not
the responses, while still capturing the response time for each of the RPC
calls. You can also choose to limit the size of strings and lists to a given
number of elements, so you can fine tune how much data you want to have stored in
Elasticsearch.
The Thrift protocol has several specific configuration options. Here is an
example configuration section for the Thrift protocol in the
+{beatname_lc}.yml+ config file:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: thrift
transport_type: socket
protocol_type: binary
idl_files: ["tutorial.thrift", "shared.thrift"]
string_max_size: 200
collection_max_size: 20
capture_reply: true
obfuscate_strings: true
drop_after_n_struct_fields: 100
------------------------------------------------------------------------------
Providing the Thrift IDL files to Packetbeat is optional. The binary
Thrift messages include the called method name and enough structural information
to decode the messages without needing the IDL files. However, if you
provide the IDL files, Packetbeat can also resolve the service name,
arguments, and exception names.
==== Configuration options
Also see <<common-protocol-options>>.
===== `transport_type`
The Thrift transport type. Currently this option accepts the values `socket`
for TSocket, which is the default Thrift transport, and `framed` for the TFramed Thrift
transport. The default is `socket`.
===== `protocol_type`
The Thrift protocol type. Currently the only accepted value is `binary` for
the TBinary protocol, which is the default Thrift protocol.
===== `idl_files`
The Thrift interface description language (IDL) files for the service that
Packetbeat is monitoring. Providing the IDL files is optional, because the Thrift
messages contain enough information to decode them without having the IDL
files. However, providing the IDL enables Packetbeat to include parameter and
exception names.
===== `string_max_size`
The maximum length for strings in parameters or return values. If a string is longer than
this value, the string is automatically truncated to this length. Packetbeat adds dots
at the end of the string to mark that it was truncated. The default is 200.
===== `collection_max_size`
The maximum number of elements in a Thrift list, set, map, or structure. If a collection
has more elements than this value, Packetbeat captures only the
specified number of elements. Packetbeat adds a fictive last element `...` to the end
of the collection to mark that it was truncated. The default is 15.
===== `capture_reply`
If this option is set to false, Packetbeat decodes the method name from
the reply and simply skips the rest of the response message. This setting can be useful
for performance, disk usage, or data retention reasons. The default is true.
===== `obfuscate_strings`
If this option is set to true, Packetbeat replaces all strings found in method parameters,
return codes, or exception structures with the `"*"` string.
===== `drop_after_n_struct_fields`
The maximum number of fields that a structure can have before Packetbeat
ignores the whole transaction. This is a memory protection mechanism (so that
Packetbeat's memory doesn't grow indefinitely), so you would typically set this
to a relatively high value. The default is 500.
[[configuration-mongodb]]
=== Capture MongoDB traffic
++++
<titleabbrev>MongoDB</titleabbrev>
++++
The following settings are specific to the MongoDB protocol. Here is a sample
configuration for the `mongodb` section of the +{beatname_lc}.yml+ config file:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: mongodb
send_request: true
send_response: true
max_docs: 0
max_doc_length: 0
------------------------------------------------------------------------------
==== Configuration options
The `max_docs` and `max_doc_length` settings are useful for limiting the amount
of data Packetbeat indexes in the `response` fields.
Also see <<common-protocol-options>>.
===== `max_docs`
The maximum number of documents from the response to index in the `response` field. The
default is 10. You can set this to 0 to index an unlimited number of documents.
Packetbeat adds a `[...]` line at the end to signify that there were additional documents
that weren't saved because of this setting.
===== `max_doc_length`
The maximum number of characters in a single document indexed in the `response`
field. The default is 5000. You can set this to 0 to index an unlimited number
of characters per document.
If the document is trimmed because of this setting, Packetbeat adds the string `...`
at the end of the document.
Note that limiting documents in this way means that they are no longer correctly
formatted JSON objects.
[[configuration-tls]]
=== Capture TLS traffic
++++
<titleabbrev>TLS</titleabbrev>
++++
TLS is a cryptographic protocol that provides secure communications on top of
an existing application protocol, like HTTP or MySQL.
Packetbeat intercepts the initial handshake in a TLS connection and extracts
useful information which helps an operator to diagnose problems as well as
strengthen the security of his or her network and systems. It does not
decrypt any information from the encapsulated protocol nor does it reveal any
sensitive information such as cryptographic keys. TLS versions 1.0 to 1.3 and
SSL 3.0 are supported.
It works by intercepting the client and server "hello" messages, which contain
the negotiated parameters for the connection such as cryptographic ciphers and
protocol versions. It can also intercept TLS alerts, which are sent by one
of the parties to signal a problem with the negotiation, such as an expired
certificate or a cryptographic error.
An example of indexed event:
[source,json]
------------------------------------------------------------------------------
"tls": {
"handshake_completed": true,
"server_certificate": {
"version": 3,
"issuer": {
"organization": "GlobalSign nv-sa",
"common_name": "GlobalSign CloudSSL CA - SHA256 - G3",
"country": "BE"
},
"subject": {
"organization": "Fastly, Inc.",
"locality": "San Francisco",
"province": "California",
"common_name": "r2.shared.global.fastly.net",
"country": "US"
},
"not_before": "2017-11-30T16:52:06.000Z",
"not_after": "2018-11-09T20:51:05.000Z",
"alternative_names": [
"elastic.co"
],
"serial_number": "19260053873395556258503998518",
"signature_algorithm": "SHA256-RSA",
"public_key_algorithm": "RSA"
},
"server_certificate_chain": [
{
"not_after": "2025-08-19T00:00:00.000Z",
"version": 3,
"serial_number": "1438827024893517455116777811697460",
"signature_algorithm": "SHA256-RSA",
"public_key_algorithm": "RSA",
"not_before": "2015-08-19T00:00:00.000Z",
"issuer": {
"organizational_unit": "Root CA",
"common_name": "GlobalSign Root CA",
"country": "BE",
"organization": "GlobalSign nv-sa"
},
"subject": {
"country": "BE",
"organization": "GlobalSign nv-sa",
"common_name": "GlobalSign CloudSSL CA - SHA256 - G3"
}
}
],
"resumed": false,
"client_hello": {
"extensions": {
"server_name_indication": [
"www.elastic.co"
],
"supported_groups": [
"secp256r1",
"secp384r1",
"secp521r1"
],
"ec_points_formats": [
"uncompressed"
],
"signature_algorithms": [
"rsa_pkcs1_sha256",
"rsa_pkcs1_sha1",
[...]
],
"application_layer_protocol_negotiation": [
"h2",
"h2-16",
[...]
],
},
"version": "3.3",
"supported_ciphers": [
"TLS_EMPTY_RENEGOTIATION_INFO_SCSV",
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
[...]
],
"supported_compression_methods": [
"NULL"
]
},
"server_hello": {
"extensions": {
"ec_points_formats": [
"uncompressed",
"ansiX962_compressed_prime",
"ansiX962_compressed_char2"
],
"application_layer_protocol_negotiation": [
"h2"
],
},
"version": "3.3",
"selected_cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"selected_compression_method": "NULL"
},
}
------------------------------------------------------------------------------
The `client_hello` contains the algorithms and extensions supported by the
client, as well as the maximum TLS version it supports (3.3).
The `server_hello` contains the final settings for the TLS session: The
selected cipher, compression method, TLS version to use and other extensions
such as application layer protocol negotiation (ALPN).
The `resumed` key indicates if the session has been resumed (`true`) or
a full handshake has been performed (`false`). When true, an additional field
is present: `resumption_method` which is `id` if the session has been
resumed using session IDs (stateful) or `ticket` if it has resumed using
session tickets (stateless).
See the <<exported-fields-tls>> section for more detailed information.
The following settings are specific to the TLS protocol. Here is a sample
configuration for the `tls` section of the +{beatname_lc}.yml+ config file:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.protocols:
- type: tls
send_certificates: true
include_raw_certificates: true
------------------------------------------------------------------------------
==== Configuration options
The `send_certificates` and `include_raw_certificates` settings are useful
for limiting the amount of data Packetbeat indexes, as multiple certificates
are usually exchanged in a single transaction, and those can take a considerable
amount of storage.
Also see <<common-protocol-options>>.
===== `send_certificates`
This setting causes the certificates presented by the client and server to be
included in the event. The server's certificate is indexed in the
`server_certificate` field and its certification chain in the
`server_certificate_chain` field. For the client, the
`client_certificate` and `client_certificate_chain` fields are used.
The default is true.
===== `include_raw_certificates`
When `send_certificates` is true, you can set `include_raw_certificates` to
include the raw certificate encoded in PEM format as a `raw` field.
If `send_certificates` is false, this setting is ignored. The default is false.
[[configuration-processes]]
== Specify which processes to monitor
This section of the +{beatname_lc}.yml+ config file is optional, but configuring
the processes enables Packetbeat to show you not only the servers that the
traffic is flowing between, but also the processes. Packetbeat can even show you
the traffic between two processes running on the same host, which is particularly
useful when you have many services running on the same server. By default,
process matching is disabled.
When Packetbeat starts, and then periodically afterwards, it scans the process table for
processes that match the configuration file. For each of these processes, it
monitors which file descriptors it has opened. When a new packet is captured,
it reads the list of active TCP and UDP connections and matches the corresponding one
with the list of file descriptors.
All this information is available via system interfaces: The `/proc` file system
in Linux and the IP Helper API (`iphlpapi.dll`) on Windows, so {beatname_uc}
doesn't need a kernel module.
NOTE: Process monitoring is currently only supported on
Linux and Windows systems. Packetbeat automatically disables
process monitoring when it detects other operating systems.
Example configuration:
[source,yaml]
------------------------------------------------------------------------------
packetbeat.procs:
enabled: true
monitored:
- process: mysqld
cmdline_grep: mysqld
- process: pgsql
cmdline_grep: postgres
- process: nginx
cmdline_grep: nginx
- process: app
cmdline_grep: gunicorn
------------------------------------------------------------------------------
When the process monitor is enabled, it will enrich all the events whose source
or destination is a local process. The `cmdline` and/or `client_cmdline` fields
will be added to an event, when the server side or client side of the connection
belong to a local process, respectively. Additionally, you can specify a pattern
using the `cmdline_grep` option, to also name those processes. This will cause
the `proc` and `client_proc` fields to be added to an event, with the name of
the matched process.
[float]
=== Configuration options
You can specify the following process monitoring options in the `monitored` section
of the +{beatname_lc}.yml+ config file:
[float]
==== `process`
The name of the process as it will appear in the published transactions. The name
doesn't have to match the name of the executable, so feel free to choose something
more descriptive (for example, "myapp" instead of "gunicorn").
[float]
==== `cmdline_grep`
The name used to identify the process at run time. When Packetbeat starts, and then
periodically afterwards, it scans the process table for
processes that match the values specified for this option. The match is done against the
process' command line as read from `/proc/<pid>/cmdline`.
[float]
[[shutdown-timeout]]
==== `shutdown_timeout`
How long Packetbeat waits on shutdown. By default, this option is disabled. Packetbeat
will wait for `shutdown_timeout` and then close. It will not track if all events were sent
previously.
Example configuration:
[source,yaml]
-------------------------------------------------------------------------------------
packetbeat.shutdown_timeout: 5s
-------------------------------------------------------------------------------------
Reference in a new issue Copy permalink