Setting up Fluentd Logs Ingestion

Fluentd is an open-source data collector which provides a unifying layer between different types of log inputs and outputs. If you already use Fluentd to collect application and system logs, you can forward the logs to LogicMonitor using the LM Logs Fluentd plugin.

The Fluentd plugin for LM Logs can be found at the following link: https://github.com/logicmonitor/lm-logs-fluentd/

Requirements

• LogicMonitor API tokens to authenticate all requests to the log ingestion API.
• Logs sent to the log ingestion API must include a “message” field. Requests sent without a “message” will not be accepted.

Install the plugin

You have a few options for installing the plugin:

• With gem (if td-agent/fluentd is installed with native Ruby): gem install fluent-plugin-lm-logs
• For native td-agent/fluentd plugin handling: td-agent-gem install fluent-plugin-lm-logs

Alternatively, you can add out_lm.rb to your Fluentd plugins directory.

Configure the plugin

Create a custom fluent.conf file or edit the existing one to specify which logs should forward to LogicMonitor. See Configuration properties for more details.

# Match events tagged with "lm.**" and
# send them to LogicMonitor
<match lm.**>
    @type lm
    resource_mapping {"<event_key>": "<lm_property>"}
    company_name <lm_company_name>
    access_id <lm_access_id>
    access_key <lm_access_key>
      <buffer>
        @type memory
        flush_interval 1s
        chunk_limit_size 5m
      </buffer> 
    debug false
    compression gzip
</match>

Request example

Sending the following request:

curl -X POST -d 'json={"message":"hello LogicMonitor from fluentd", "event_key":"lm_property_value"}' http://localhost:8888/lm.test

Returns the event:

{
    "message": "hello LogicMonitor from fluentd"
}

Configuration properties

Resource Mapping

When defining the resource mapping for the fluent event, the <event_key> in the incoming event is mapped to the LogicMonitor resource, which is the value of <lm_property>.

For example, you may map a hostname field in the log event to the LogicMonitor property system.hostname using:

resource_mapping {"hostname": 'system.hostname"}

If the LogicMonitor resource mapping is known, the event_key property can be overridden by specifying _lm.resourceId in each record. See Resource mapping examples below.

Example 1

In this example, all incoming records that match lm.** will go through the filter and the specified _lm.resourceId mapping is added before it is sent to LogicMonitor.

<filter lm.**>
    @type record_transformer
    <record>
_lm.resourceId { "system.aws.arn": "arn:aws:ec2:us-west-1:xxx:instance/i-xxx"}
   	tag ${tag}
    </record>
</filter>

Example 2

For Kubernetes logs in Fluentd, the resource mapping can always be defined with the statement:

 resource_mapping {"kubernetes.pod_name": "auto.name"}

Fluentd Examples

Because Fluentd provides a unified logging layer, you’re able to use it collect many types of logs which you can then forward to LogicMonitor for analysis.

We provide configuration examples for using the Fluentd plugin to send Windows Event Logs, Apache access logs, and more at: https://github.com/logicmonitor/lm-logs-fluentd/tree/master/Examples

Performance Tuning

You can optimize Fluentd performance by editing the buffer configuration block.

For example, if the log input speed is faster than the forwarding of logs, the batches will accumulate. To fix this, you can increase the flush_thread_count to parallelize the output to LogicMonitor.

<buffer>
@type memory
flush_interval 1s
chunk_limit_size 8m
flush_thread_count 8
<buffer>

Troubleshooting

Enable debug logging by setting the debug property to “true” in fluent.conf to see additional information in the Fluentd console.

• Log ingestion is delayed for multi-line event. For multi-line events it may appear that log ingestion is delayed until the next log entry is created. This delay occurs because Fluentd will only parse the last line when a line break is appended at the end of the line. To fix this, add or increase the configuration property multiline_flush_interval (in seconds) in fluent.conf.

See the Troubleshooting guide for more information.