Skip to content

Telegraf

telegraf

If you use InfluxDB you'll be familiar with Telegraf, the open source server agent for collecting metrics.

You can send metrics directly from Telgraf in the Telegraf json format. To configure Telegraf to send metrics to Anomify you can enable the [[outputs.http]] plugin in Telegraf as shown below.

Configure Telegraf to send to Anomify

To use the Telegraf Graphite Template Pattern the following options in the Telegraf [agent] configuration section are required for Telegraf to add the host to the tags:

## Maximum number of unwritten metrics per output.
## Ideally change this from 10000 to 1000 to minimise thundering herd issues
## metric_buffer_limit = 10000
metric_buffer_limit = 1000
## Override default hostname, if empty use os.Hostname()
hostname = ""
## If set to true, do no set the "host" tag in the telegraf agent.
omit_hostname = false

Also add a [[outputs.http]] section to the Telegraf config as below: 

[[outputs.http]]
## URL is the Anomify address to send metrics to
url = "https://<YOUR_ANOMIFY_HOST>/flux/metric_data_post"
## Set a long timeout because if a network partition occurs between telegraf
## and anomify, telegraf will keep and batch the metrics to send through AND
## 10s of 1000s of metrics can then be sent when the network partition is
## resolved, these can take a while to process as often many other telegraf
## instances may have been partitioned at the same time, so a thundering herd
## is sent to anomify.
timeout = "120s"
method = "POST"
data_format = "json"
use_batch_format = true
json_timestamp_units = "1s"
content_encoding = "gzip"
## A list of statuscodes (<200 or >300) upon which requests should not be retried
# Should be available in March 2022 not available as of telegraf-1.21.4
#non_retryable_statuscodes = [400, 403, 409, 413, 499, 500, 502, 503]
[outputs.http.headers]
Content-Type = "application/json"
key = "<ANOMIFY_API_KEY>"
telegraf = "true"
## Optionally you can pass a prefix e.g.
# prefix = "telegraf"

Clarifications on the config above

  1. Anomify applies the same logic that is used by the Telegraf Graphite Template Pattern e.g. template = "host.tags.measurement.field"
  2. Anomify replaces . for _ and / for - in any tags, measurement or fields in which these characters are found.
  3. Anomify silently drops any metrics that have a string or boolean value e.g. non numeric values.
  4. The additional [[outputs.http.headers]] must be specified.
  5. content_encoding must be set to gzip.
  6. We prevent Telegraf resubmit data on certain status codes. This is to ensure Telegraf does not attempt to send the same bad data, or too send too much data again and again (available later in March 2022 in Telegraf).
  7. If there is a long network partition between Telegraf agents and Anomify, sometimes some data may be dropped, but this can often be preferable to the thundering herd swamping I/O.

Testing your ingestion configuration

To test your configuration you can add x-test-only as a header on HTTP requests to Anomify. Metrics sent this way won't be analysed by Anomify but will appear as a list in the Metric Ingestion Test Page in settings. Once you're happy that you are sending the correct metrics, remove the x-test-only header and analysis will start after 100 datapoints have been sent.