Custom Fluentd Filters in the lm-logs Helm Chart

When you deploy LM Logs in a Kubernetes environment using the Helm chart, you can extend log processing by configuring custom Fluentd filters. This capability enables you to control which logs are collected, enrich log data, and transform log content without modifying the Helm chart itself.

For more information on deploying LM Logs Using Helm Chart, see Sending Kubernetes Logs and Events.

You can configure custom filters using the fluent.extraFilters parameter in the lm-logs Helm chart when deploying LM Logs in Helm. This approach provides flexibility while maintaining upgrade compatibility.

The following example shows the basic syntax for configuring custom Fluentd filters:

lm-logs:
  enabled: true
  kubernetes:
  fluent:
   extraFilters: |
    <filter kubernetes.**>
      @type grep
      ...
    </filter>

For more information about Fluentd filter plugins and syntax, see the Fluentd documentation.

Fluentd Filter Use Cases

Fluentd filters enable you to control how logs are processed before they are sent to LM Logs. Using fluent.extraFilters, you can include or exclude logs based on Kubernetes metadata, filter logs by message content, and apply logical conditions to refine log collection. The following examples demonstrate common use cases, including how to apply AND and OR logic to match specific logging requirements.

Excluding Logs from Specific Namespaces

Use this configuration to drop logs from one or more Kubernetes namespaces.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <exclude>
        key $.kubernetes.namespace_name
        pattern /^(sidecar-ns1|sidecar-ns2)$/
      </exclude>
    </filter>

This configuration drops logs from the namespaces defined in the pattern. The | operator in the regex applies OR logic, so logs from either sidecar-ns1 or sidecar-ns2 are excluded.

Keeping Logs from Selected Namespaces

Use this configuration to collect logs only from specific namespaces.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <regexp>
        key $.kubernetes.namespace_name
        pattern /^(ns1|ns2)$/
      </regexp>
    </filter>

This configuration keeps logs from ns1 or ns2, using OR logic within the regex pattern.

Filtering Logs by Namespace and Message Content

Use this configuration when logs must meet multiple conditions.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <regexp>
        key $.kubernetes.namespace_name
        pattern /^default$/
      </regexp>
      <regexp>
        key message
        pattern /ERROR|WARN/
      </regexp>
    </filter>

This configuration applies AND logic:

• The namespace must be default
• AND the message must contain ERROR or WARN

Fluentd applies AND logic automatically when multiple <regexp> entries are defined in the same filter.

Filtering Logs by Multiple Message Patterns

Use this configuration to collect logs matching any of several keywords.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <regexp>
        key message
        pattern /ERROR|WARN|FATAL/
      </regexp>
    </filter>

This configuration uses OR logic to keep logs that contain ERROR, WARN, or FATAL.

Using Explicit OR Logic for Readability

For more complex scenarios, you can explicitly define OR conditions.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <or>
        <regexp>
          key $.kubernetes.namespace_name
          pattern /^ns1$/
        </regexp>
        <regexp>
          key $.kubernetes.namespace_name
          pattern /^ns2$/
        </regexp>
      </or>
    </filter>

This configuration keeps logs from ns1 or ns2. Use this approach when readability is more important than compact syntax.

Using Explicit AND Logic for Clarity

You can explicitly define AND conditions.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <and>
        <regexp>
          key $.kubernetes.namespace_name
          pattern /^default$/
        </regexp>
        <regexp>
          key message
          pattern /ERROR|WARN/
        </regexp>
      </and>
    </filter>

This configuration keeps logs only when both conditions match. This approach is equivalent to multiple <regexp> entries but can improve readability.

Excluding Logs from a Specific Pod

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <exclude>
        key $.kubernetes.pod_name
        pattern /^my-pod-name$/
      </exclude>
    </filter>

This configuration drops logs from the pod my-pod-name.

Excluding Logs from a Specific Container

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type grep
      <exclude>
        key $.kubernetes.container_name
        pattern /^sidecar-container$/
      </exclude>
    </filter>

This configuration drops logs from the container sidecar-container.

Adding a Custom Field to Logs

This configuration adds a new field to each record.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type record_transformer
      <record>
        filtername filtervalue
      </record>
    </filter>

Removing Fields from Logs

This configuration removes the stream field.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type record_transformer
      remove_keys stream
    </filter>

To remove multiple fields:

remove_keys stream,logtag

Parsing JSON from Log Messages

This configuration parses the message field as JSON and extracts structured fields.

fluent:
  extraFilters: |
    <filter kubernetes.**>
      @type parser
      key_name message
      reserve_data true
      remove_key_name_field false
      <parse>
        @type json
      </parse>
    </filter>