498 lines
16 KiB
Plaintext
498 lines
16 KiB
Plaintext
[[new-beat]]
|
|
== Creating a New Beat
|
|
|
|
This guide walks you through the steps for creating a new Elastic Beat. The
|
|
Beats are a collection of lightweight daemons that collect operational data from
|
|
your servers and ship it to Elasticsearch or Logstash. The common parts for
|
|
all Beats are placed in the libbeat library, which contains packages for sending
|
|
data to Elasticsearch and Logstash, for configuration file handling, for signal
|
|
handling, for logging, and more. By using this common framework, you can ensure
|
|
that all Beats behave consistently and that they are easy to package and run
|
|
with common tools.
|
|
|
|
In this guide, you learn how to use the Beat generator to create source code for
|
|
an example Beat called Countbeat. The Beat generator creates all the files required
|
|
for a working Beat. To create your own Beat, you modify the generated
|
|
files and implement the custom logic needed to collect the data you want to ship.
|
|
|
|
The following topics describe how to build a new Beat:
|
|
|
|
* <<newbeat-getting-ready>>
|
|
* <<newbeat-overview>>
|
|
* <<newbeat-generate>>
|
|
* <<setting-up-beat>>
|
|
* <<compiling-and-running>>
|
|
* <<beater-interface>>
|
|
* <<newbeat-sharing>>
|
|
* <<event-conventions>>
|
|
|
|
[[newbeat-getting-ready]]
|
|
=== Getting Ready
|
|
|
|
All Beats are written in http://golang.org/[Go], so having Go installed and knowing
|
|
the basics are prerequisites for understanding this guide.
|
|
But don't worry if you aren't a Go expert. Go is a relatively new
|
|
language, and very few people are experts in it. In fact, several
|
|
people learned Go by contributing to Packetbeat and libbeat, including the
|
|
original Packetbeat authors.
|
|
|
|
For general information about contributing to Beats, see <<beats-contributing>>.
|
|
|
|
After you have https://golang.org/doc/install[installed Go] and set up the
|
|
https://golang.org/doc/code.html#GOPATH[GOPATH] environment variable to point to
|
|
your preferred workspace location, clone the Beats repository in the correct location
|
|
under `GOPATH`:
|
|
|
|
[source,shell]
|
|
----------------------------------------------------------------------
|
|
mkdir -p ${GOPATH}/src/github.com/elastic
|
|
git clone https://github.com/elastic/beats ${GOPATH}/src/github.com/elastic/beats
|
|
----------------------------------------------------------------------
|
|
|
|
To build your beat
|
|
on a specific version of libbeat, check out the specific branch ({doc-branch} in the example below):
|
|
|
|
["source","sh",subs="attributes"]
|
|
----
|
|
cd ${GOPATH}/src/github.com/elastic/beats
|
|
git checkout {doc-branch}
|
|
----
|
|
|
|
NOTE: If you have multiple go paths, use `${GOPATH%%:*}` instead of `${GOPATH}`.
|
|
|
|
[[newbeat-overview]]
|
|
=== Overview
|
|
|
|
At the high level, a simple Beat has two main components:
|
|
|
|
* a component that collects the actual data, and
|
|
* a publisher that sends the data to the specified output, such as Elasticsearch or
|
|
Logstash.
|
|
|
|
The publisher is already implemented in libbeat, so you typically only have to worry about the logic
|
|
specific to your Beat (the code that creates the event and sends it to the publisher).
|
|
Libbeat also offers common services like configuration management, logging,
|
|
daemonzing, and Windows service handling, and data processing modules.
|
|
|
|
image:./images/beat_overview.png[Beat overview architecture]
|
|
|
|
The event that you create is a JSON-like object (Go type `map[string]interface{}`) that
|
|
contains the collected data to send to the publisher. At a minimum, the event object
|
|
must contain a `@timestamp` field and a `type` field. Beyond
|
|
that, events can contain any additional fields, and they can be created as often
|
|
as necessary.
|
|
|
|
The following example shows an event object in Lsbeat:
|
|
|
|
[source,json]
|
|
----------------------------------------------------------------------
|
|
{
|
|
"@timestamp": "2016-07-13T21:33:58.355Z",
|
|
"beat": {
|
|
"hostname": "mar.local",
|
|
"name": "mar.local"
|
|
},
|
|
"directory": false,
|
|
"filename": "winlogbeat.yml",
|
|
"filesize": 2895,
|
|
"modtime": "2016-07-13T20:56:21.000Z",
|
|
"path": "./vendor/github.com/elastic/beats/winlogbeat/winlogbeat.yml",
|
|
"type": "lsbeat"
|
|
}
|
|
|
|
{
|
|
"@timestamp": "2016-07-13T21:33:58.354Z",
|
|
"beat": {
|
|
"hostname": "mar.local",
|
|
"name": "mar.local"
|
|
},
|
|
"directory": true,
|
|
"filename": "system",
|
|
"filesize": 238,
|
|
"modtime": "2016-07-13T20:56:21.000Z",
|
|
"path": "./vendor/github.com/elastic/beats/winlogbeat/tests/system",
|
|
"type": "lsbeat"
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
Now that you have the big picture, let's dig into the code.
|
|
|
|
[[newbeat-generate]]
|
|
=== Generating Your Beat
|
|
|
|
To generate your own Beat, you use the Beat generator available in the beats repo on GitHub. If you haven't
|
|
downloaded the Beats source code yet, follow the instructions in <<newbeat-getting-ready>>.
|
|
|
|
Before running beat generator, you must decide on a name for your beat. The name should be one word with
|
|
the first letter capitalized. In our example, we use `Countbeat`.
|
|
|
|
Now create a directory under $GOPATH for your repository and change to the new directory:
|
|
|
|
[source,shell]
|
|
--------------------
|
|
mkdir ${GOPATH}/src/github.com/{user}
|
|
cd ${GOPATH}/src/github.com/{user}
|
|
--------------------
|
|
|
|
Run python and specify the path to the Beat generator:
|
|
|
|
NOTE: Python 2 is required (Python 3 will not work).
|
|
|
|
[source,shell]
|
|
--------------------
|
|
python $GOPATH/src/github.com/elastic/beats/script/generate.py
|
|
--------------------
|
|
|
|
Python will prompt you to enter information about your Beat. For the `project_name`, enter `Countbeat`.
|
|
For the `github_name`, enter your github id. The `beat` and `beat_path` are set to the correct values automatically (just press Enter to accept each default). For the `full_name`, enter your firstname and lastname.
|
|
|
|
[source,shell]
|
|
---------
|
|
project_name [Examplebeat]: Countbeat
|
|
github_name [your-github-name]: {username}
|
|
beat_path [github.com/{github id}]:
|
|
full_name [Firstname Lastname]: {Full Name}
|
|
---------
|
|
|
|
The Beat generator creates a directory called `countbeat` inside of your project folder.
|
|
|
|
You now have a raw template of the Beat, but you still need to <<setting-up-beat,fetch dependencies and set up the Beat>>.
|
|
|
|
[[setting-up-beat]]
|
|
=== Fetching Dependencies and Setting up the Beat
|
|
|
|
First you need to install the following tools:
|
|
|
|
* https://www.python.org/downloads/[Python]
|
|
* https://virtualenv.pypa.io/en/stable/[virtualenv]
|
|
|
|
To fetch dependencies and set up the Beat, run:
|
|
|
|
[source,shell]
|
|
---------
|
|
cd ${GOPATH}/src/github.com/{user}/countbeat
|
|
make setup
|
|
---------
|
|
|
|
The Beat now contains the basic config file, `countbeat.yml`, and template files. The Beat is "complete" in the sense
|
|
that you can compile and run it. However, to make it functionally complete, you need to add your custom logic (see <<beater-interface>>), along with any additional configuration parameters that your Beat requires.
|
|
|
|
[[compiling-and-running]]
|
|
=== Building and Running the Beat
|
|
|
|
To compile the Beat, make sure you are in the Beat directory (`$GOPATH/src/github.com/{user}/countbeat`) and run:
|
|
|
|
[source,shell]
|
|
---------
|
|
make
|
|
---------
|
|
|
|
NOTE: we don't support the `-j` option for make at the moment.
|
|
|
|
Running this command creates the binary called `countbeat` in `$GOPATH/src/github.com/{user}/countbeat`.
|
|
|
|
Now run the Beat:
|
|
|
|
[source,shell]
|
|
---------
|
|
./countbeat -e -d "*"
|
|
---------
|
|
|
|
The command automatically loads the default config file, `countbeat.yml`, and sends debug output to the console.
|
|
|
|
You can stop the Beat by pressing `Ctrl+C`.
|
|
|
|
[[beater-interface]]
|
|
=== The Beater Interface
|
|
|
|
Each Beat needs to implement the Beater interface defined in libbeat.
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
// Beater is the interface that must be implemented by every Beat. A Beater
|
|
// provides the main Run-loop and a Stop method to break the Run-loop.
|
|
// Instantiation and Configuration is normally provided by a Beat-`Creator`.
|
|
//
|
|
// Once the beat is fully configured, the Run() method is invoked. The
|
|
// Run()-method implements the beat its run-loop. Once the Run()-method returns,
|
|
// the beat shuts down.
|
|
//
|
|
// The Stop() method is invoked the first time (and only the first time) a
|
|
// shutdown signal is received. The Stop()-method normally will stop the Run()-loop,
|
|
// such that the beat can gracefully shutdown.
|
|
type Beater interface {
|
|
// The main event loop. This method should block until signalled to stop by an
|
|
// invocation of the Stop() method.
|
|
Run(b *Beat) error
|
|
|
|
// Stop is invoked to signal that the Run method should finish its execution.
|
|
// It will be invoked at most once.
|
|
Stop()
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
To implement the Beater interface, you need to define a Beat object that
|
|
implements two methods: <<run-method,`Run()`>> and <<stop-method,`Stop()`>>.
|
|
|
|
[source,go]
|
|
--------------
|
|
type Countbeat struct {
|
|
done chan struct{} <1>
|
|
config config.Config <2>
|
|
client publisher.Client <3>
|
|
|
|
...
|
|
}
|
|
|
|
func (bt *Countbeat) Run(b *beat.Beat) error {
|
|
...
|
|
}
|
|
|
|
|
|
func (bt *Countbeat) Stop() {
|
|
...
|
|
}
|
|
|
|
--------------
|
|
|
|
By default, the Beat object contains the following:
|
|
|
|
<1> `done`: Channel used by the `Run()` method to stop when the `Stop()` method is called.
|
|
<2> `config`: Configuration options for the Beat
|
|
<3> `client`: Publisher that takes care of sending the events to the
|
|
defined output.
|
|
|
|
The `Beat` parameter received by the `Run` method contains data about the
|
|
Beat, such as the name, version, and common configuration options.
|
|
|
|
Each Beat also needs to implement the <<new-function,`New()`>> function to create the Beat object. This means your
|
|
Beat should implement the following functions:
|
|
|
|
[horizontal]
|
|
<<new-function, New>>:: Creates the Beat object
|
|
<<run-method, Run>>:: Contains the main application loop that captures data
|
|
and sends it to the defined output using the publisher
|
|
<<stop-method, Stop>>:: Contains logic that is called when the Beat is signaled to stop
|
|
|
|
When you run the Beat generator, it adds implementations for all these functions to the source code (see
|
|
`beater/countbeat.go`). You can modify these implementations, as required, for your Beat.
|
|
|
|
We strongly recommend that you create a main package that contains only the main
|
|
method (see `main.go`). All your Beat-specific code should go in a separate folder and package.
|
|
This will allow other Beats to use the other parts of your Beat as a library, if
|
|
needed.
|
|
|
|
NOTE: To be consistent with other Beats, you should append `beat` to your Beat name.
|
|
|
|
Let's go through each of the methods in the `Beater` interface and look at a
|
|
sample implementation.
|
|
|
|
[[new-function]]
|
|
==== New function
|
|
|
|
The `New()` function receives the configuration options defined for the Beat and
|
|
creates a Beat object based on them. Here's the `New()` function that's generated in
|
|
`beater/countbeat.go` when you run the Beat generator:
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
func New(b *beat.Beat, cfg *common.Config) (beat.Beater, error) {
|
|
config := config.DefaultConfig
|
|
if err := cfg.Unpack(&config); err != nil {
|
|
return nil, fmt.Errorf("Error reading config file: %v", err)
|
|
}
|
|
|
|
bt := &Countbeat{
|
|
done: make(chan struct{}),
|
|
config: config,
|
|
}
|
|
return bt, nil
|
|
}
|
|
|
|
----------------------------------------------------------------------
|
|
|
|
Pointers are used to distinguish between when the setting is completely
|
|
missing from the configuration file and when it has a value that matches the
|
|
type's default value.
|
|
|
|
The recommended way of handling the configuration (as shown in the code example)
|
|
is to create a `Config` structure with the configuration options and a `DefaultConfig` with
|
|
the default configuration options.
|
|
|
|
When you use the Beat generator, the Go structures for a basic config are added to `config/config.go`:
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
package config
|
|
|
|
import "time"
|
|
|
|
type Config struct {
|
|
Period time.Duration `config:"period"`
|
|
}
|
|
|
|
var DefaultConfig = Config{
|
|
Period: 1 * time.Second,
|
|
}
|
|
|
|
----------------------------------------------------------------------
|
|
|
|
|
|
This mirrors the config options that are defined in the config file, `countbeat.yml`.
|
|
|
|
[source,yaml]
|
|
------------
|
|
countbeat:
|
|
# Defines how often an event is sent to the output
|
|
period: 10s
|
|
------------
|
|
|
|
- `period`: Defines how often to send out events
|
|
|
|
The config file is generated when you run `make setup` to <<setting-up-beat,set up the beat>>. The file contains
|
|
basic configuration information. To add configuration options to your Beat, you need to update the Go structures in
|
|
`config/config.go` and add the corresponding config options to `_meta/beat.yml`.
|
|
|
|
For example, if you add a config option called `path` to the Go structures:
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
type Config struct {
|
|
Period time.Duration `config:"period"`
|
|
Path string `config:"path"`
|
|
}
|
|
|
|
var DefaultConfig = Config{
|
|
Period: 1 * time.Second,
|
|
Path: ".",
|
|
}
|
|
|
|
----------------------------------------------------------------------
|
|
|
|
You also need to add `path` to `_meta/beat.yml`:
|
|
|
|
[source,yml]
|
|
----------------------------------------------------------------------
|
|
countbeat:
|
|
period: 10s
|
|
path: "."
|
|
----------------------------------------------------------------------
|
|
|
|
After modifying `beat.yml`, run the following command to apply your updates:
|
|
|
|
[source,shell]
|
|
----------------------------------------------------------------------
|
|
make update
|
|
----------------------------------------------------------------------
|
|
|
|
[[run-method]]
|
|
==== Run Method
|
|
|
|
The `Run` method contains your main application loop.
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
func (bt *Countbeat) Run(b *beat.Beat) error {
|
|
logp.Info("countbeat is running! Hit CTRL-C to stop it.")
|
|
|
|
bt.client = b.Publisher.Connect()
|
|
ticker := time.NewTicker(bt.config.Period)
|
|
counter := 1
|
|
for {
|
|
select {
|
|
case <-bt.done:
|
|
return nil
|
|
case <-ticker.C:
|
|
}
|
|
|
|
event := common.MapStr{ <1>
|
|
"@timestamp": common.Time(time.Now()), <2>
|
|
"type": b.Name,
|
|
"counter": counter,
|
|
}
|
|
bt.client.PublishEvent(event) <3>
|
|
logp.Info("Event sent")
|
|
counter++
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
<1> Create the event object.
|
|
<2> Specify a `@timestamp` field of time `common.Time`.
|
|
<3> Use the publisher to send the event out to the defined output
|
|
|
|
Inside the loop, the Beat sleeps for a configurable period of time and then
|
|
captures the required data and sends it to the publisher. The publisher client is available as part of the Beat object
|
|
through the `client` variable.
|
|
|
|
The `event := common.MapStr{}` stores the event in a json format, and `bt.client.PublishEvent(event)` publishes data to Elasticsearch.
|
|
In the generated Beat, there are three fields in the event: @timestamp, type, and counter.
|
|
|
|
When you add fields to the event object, you also need to add them to the `_meta/fields.yml` file:
|
|
|
|
[source,yaml]
|
|
----------------------------------------------------------------------
|
|
- key: countbeat
|
|
title: countbeat
|
|
description:
|
|
fields:
|
|
- name: counter
|
|
type: long
|
|
required: true
|
|
description: >
|
|
PLEASE UPDATE DOCUMENTATION
|
|
----------------------------------------------------------------------
|
|
|
|
Remember to run `make update` to apply your updates.
|
|
|
|
For more detail about naming the fields in an event, see <<event-conventions>>.
|
|
|
|
[[stop-method]]
|
|
==== Stop Method
|
|
|
|
The `Stop` method is called when the Beat is signaled to stop, for
|
|
example through the SIGTERM signal on Unix systems or the service control
|
|
interface on Windows. This method simply closes the channel
|
|
which breaks the main loop.
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
func (bt *Countbeat) Stop() {
|
|
bt.client.Close()
|
|
close(bt.done)
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
==== The main Function
|
|
|
|
If you follow the `Countbeat` model and put your Beat-specific code in its own type
|
|
that implements the `Beater` interface, the code from your main package is
|
|
very simple:
|
|
|
|
[source,go]
|
|
----------------------------------------------------------------------
|
|
package main
|
|
|
|
import (
|
|
"os"
|
|
|
|
"github.com/elastic/beats/libbeat/beat"
|
|
"github.com/elastic/beats/libbeat/cmd"
|
|
|
|
"github.com/kimjmin/countbeat/beater"
|
|
)
|
|
|
|
var RootCmd = cmd.GenRootCmd("countbeat", "", beater.New)
|
|
|
|
func main() {
|
|
if err := RootCmd.Execute(); err != nil {
|
|
os.Exit(1)
|
|
}
|
|
}
|
|
----------------------------------------------------------------------
|
|
|
|
[[newbeat-sharing]]
|
|
=== Sharing Your Beat with the Community
|
|
|
|
When you're done with your new Beat, how about letting everyone know? Open
|
|
a pull request to add your link to the {libbeat}/community-beats.html[Community Beats] list.
|