Sending Fluent Bit Logs

Fluent Bit is an open-source, lightweight, and high-performance log processor and forwarder designed to collect, filter, and deliver log data. Fluent Bit supports multiple input sources (for example, files, system logs, Kubernetes logs) and a variety of output destinations, including cloud services, databases, and monitoring tools.

If you are already using Fluentbit to collect application and system logs, you can forward the logs to LogicMonitor using the LM Logs Fluentbit plugin. To download the plugin, go to the LM Logs Fluentbit plugin page from the LogicMonitor Github repository. This provides a LogicMonitor-developed plugin that contains the specific instructions for sending logs to LogicMonitor.

Supported Operating Systems for Sending Fluent Bit Logs

• Linux-AMD
• Linux-ARM
• MS-Windows

Configuration Properties for Fluent Bit Plugin

You can use the following properties to configure Fluent Bit plugin with LogicMonitor:

Requirements for Sending Fluent Bit Logs

To send the Fluent Bit logs, you need the following:

• A LogicMonitor account.
• A LogicMonitor API token to authenticate all requests to the log ingestion API. To know more, see API Tokens from the product documentation.
• LM Logs Fluent Bit plugin—To download the plugin, go to the LM Logs Fluentbit plugin page from the LogicMonitor Github repository. This provides a LogicMonitor-developed plugin that contains the specific instructions for sending logs to LogicMonitor.
• A Fluent Bit configuration file (such as fluent-bit.conf or your custom configuration file).
• The out_lm.so plugin file.

Downloading Fluent Bit Plugin

To download Fluent Bit plugin, use the following commands depending on your operating system architecture.

• For Linux-AMD:
  curl -L -o out_lm-linux.so https://github.com/logicmonitor/lm-logs-fluentbit/raw/master/plugins/out_lm-linux.so

• For Linux-ARM:
  curl -L -o out_lm-linux-arm64.so https://github.com/logicmonitor/lm-logs-fluentbit/raw/master/plugins/out_lm-linux-arm64.so

• For MS-Windows:
  curl -L -o out_lm-windows.so https://github.com/logicmonitor/lm-logs-fluentbit/blob/master/plugins/out_lm-windows.so

• For Other OS:
  curl -L -o out_lm.so https://github.com/logicmonitor/lm-logs-fluentbit/raw/master/plugins/out_lm.so

Installing Fluent Bit Plugin

To install Fluent Bit plugin on your Linux system, do the following:

1. Run the following command to get fluent bit installation script:
   curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
2. Run below command to install fluent bit on your machine:
   sudo apt-get install fluent-bit
3. Check fluentbit status with below command:
   sudo systemctl status fluent-bit
4. Copy the appropriate plugin file for your architecture from the LM Logs Plugin repository on LogicMonitor’s GitHub.

To install Fluent Bit plugin on your Windows system, do the following:

• Download the installation package from the Installation Packages section of the Fluent Bit official website, then run the .exe file to begin installation.

Configuring Fluent Bit Plugin

1. Open your existing fluent-bit.conf file or create a new one if it does not exist.

2. Add the required plugin parameters to the configuration file:
   [SERVICE]
     Flush        5
[INPUT]
Name        tail
Path        <file_path>/tail.txt
[OUTPUT]
Name logicmonitor
lmCompanyName  <company_name_with_domain>
Match *
Workers 1
accessKey <accessKey>
accessID <accessID>
bearerToken Bearer <bearerToken>
resourceMapping {"<event_key>": "<lm_property>"}
includeMetadata <boolean_value>
lmDebug <boolean_value>
3. Map the sources generating log data using the appropriate format to ensure logs are parsed correctly by LogicMonitor Logs.
4. Define the resource mapping for Fluent Bit events. Use the event_key from the incoming event and map it to a LogicMonitor resource property using the resource_mapping directive.For example, to map a "hostname" field in a log event to the LogicMonitor property system.hostname, add the following:
   resource_mapping {"hostname": 'system.hostname"}
5. This ensures the incoming log event is associated with the correct resource in your LogicMonitor portal.

Running out_lm Fluent Bit Plugin

You can run the out_lm Fluent Bit plugin for log ingestion in the following two ways:

• Running Fluent Bit with a Direct Command
• Running Fluent Bit to Use a Custom Configuration File”

Running Fluent Bit with a Direct Command

• To run Fluent Bit using the out_lm plugin and a configuration file, execute the following command:

<path_to_installed_fluentbit>/fluent-bit -e <path_to_out_lm_plugin>/out_lm.so -c <path_to_file>/fluent-bit.conf

In this command:

• <path_to_installed_fluentbit> is the path to your Fluent Bit executable (for example, /opt/fluent-bit/bin/fluent-bit).
• <path_to_out_lm_plugin> is the path to the out_lm.so plugin file.
• <path_to_file> is the path to your Fluent Bit configuration file.

Running Fluent Bit to Use a Custom Configuration File

Alternatively, to run Fluent Bit using a custom configuration file through the systemd service:

1. Open the Fluent Bit systemd service file:sudo nano /etc/systemd/system/fluent-bit.service
2. Update the ExecStart line to reference your custom configuration file:[Service] ExecStart=/opt/fluent-bit/bin/fluent-bit -c /path/to/your/custom.conf
3. Reload the systemd manager configuration and restart Fluent Bit:sudo systemctl daemon-reload sudo systemctl restart fluent-bit

Configuration Examples for the Fluent Bit Plugin

The following are examples of Fluent Bit plugin configuration.

Mapping Host Information to a Resource Attribute

This example maps the host field from the log source to the system.ips attribute in LogicMonitor.

[SERVICE]
    Flush        5
 
[INPUT]
    Name         tail
    Path         /var/log/sample_logs/*.log
    Tag          application_logs
 
[OUTPUT]
    Name         logicmonitor
    Match        application_logs
    Host         logs.logicmonitor.com
    Port         443
    URI          /rest/log/ingest
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    Format       json
    tls          on
    resourceMapping {"host":"system.ips"}

In this configuration:

• The resourceMapping field specifies that the host field from logs should map to the system.ips attribute in LogicMonitor.

Mapping Application Name to a LogicMonitor Attribute

This example maps the app_name field from the log metadata to the system.application attribute in LogicMonitor.

[SERVICE]
    Flush        10
 
[INPUT]
    Name         tail
    Path         /var/log/app_logs/*.log
    Tag          app_logs
 
[FILTER]
    Name         modify
    Match        app_logs
    Add          app_name my_application
 
[OUTPUT]
    Name         logicmonitor
    Match        app_logs
    Host         logs.logicmonitor.com
    Port         443
    URI          /rest/log/ingest
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    Format       json
    tls          on
    resourceMapping {"app_name":"system.application"}

Here:

• The FILTER section adds the app_name field with the value my_application.
• The resourceMapping associates app_name with the system.application attribute in LogicMonitor.

Including Metadata and Debugging

This example demonstrates including additional metadata and enabling debug logging.

[SERVICE]
    Flush        5
    Log_Level    debug
 
[INPUT]
    Name         tail
    Path         /var/log/system_logs/*.log
    Tag          system_logs
 
[OUTPUT]
    Name         logicmonitor
    Match        system_logs
    Host         logs.logicmonitor.com
    Port         443
    URI          /rest/log/ingest
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    Format       json
    tls          on
    resourceMapping {"source":"system.type", "host":"system.hostname"}
    includeMetadata true

In this configuration:

• includeMetadata true ensures log metadata such as timestamps and hostname are forwarded.
• resourceMapping maps the source field to system.type and the host field to system.hostname.

HTTP conf file

This Fluent Bit configuration example sets up an HTTP input to receive log data and forwards it to a specified output, a LogicMonitor endpoint.

[INPUT]
    name http
    listen 0.0.0.0
    port 9880
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match test
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"host":"system.ips"}
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

Multiple conf file

This Fluent Bit configuration file is structured to collect logs from two different files, process them using a specified parser, and forward the logs to an output destination with LogicMonitor-specific parameters.

[SERVICE]
    Flush        1
    Log_Level    info
    Parsers_File parsers-multiple-config.conf
 
[INPUT]
    Name         tail
    Path         <file_path>/tail1.log
    Parser       json
    Tag          app1.logs
 
[INPUT]
    Name         tail
    Path         <file_path>/tail2.log
    Parser       json
    Tag          app2.logs
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

This will also need a parser configuration parsers.conf which defines how Fluent Bit processes log data formatted as JSON:

[PARSER]
    Name         json
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S

TCP conf file

This Fluent Bit configuration file is designed to collect log data over TCP, process it, and forward it to an output destination with specific attributes like resource mapping and authentication.

[SERVICE]
    Flush        1
    Log_Level    info
    Parsers_File parsers.conf
 
[INPUT]
    Name        tcp
    Tag         tcp_log
    Listen      0.0.0.0
    Port        <port>
 
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"host":"system.ips"}
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

In this configuration, the parsers.conf should be similar to parsers.conf defined above.

Multiline conf file

This Fluent Bit configuration is designed to handle logs that span multiple lines (e.g., stack traces or long messages), parse them using a multiline parser, and forward them to a specified output.

[SERVICE]
    Flush        5
    parsers_file parsers_multiline.conf
[INPUT]
    Name        tail
    Path        tail.txt
    multiline.parser multiline-regex-test
[OUTPUT]
    Name logicmonitor
    lmCompanyName  <company_name_with_domain>
    Match *
    Workers 1
    accessKey <accessKey>
    accessID <accessID>
    bearerToken Bearer <bearerToken>
    resourceMapping {"host":"system.ips"}
    includeMetadata <boolean_value>
    id <output_id>
    lmDebug <boolean_value>

Below configuration file defines a multiline parser named multiline-regex-test for processing logs that span multiple lines, using regular expressions to group related log lines.

[MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    flush_timeout 1000
    #
    # Regex rules for multiline parsing
    # ---------------------------------
    #
    # configuration hints:
    #
    #  - first state always has the name: start_state
    #  - every field in the rule must be inside double quotes
    #
    # rules |   state name  | regex pattern                  | next state
    # ------|---------------|--------------------------------------------
    rule      "start_state"   "/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule      "cont"          "/^\s+at.*/"                     "cont"

Troubleshooting

Enable debug logging by setting the debug property to true in fluent-bit.conf to see additional information in the Fluentd console. For more information on logs troubleshooting, see Troubleshooting.