About LogSources

Last updated on 22 November, 2022

LogSources is a LogicModule that automatically lets you find logs for resources that you are monitoring. Use LogSources as a data collection template to configure data from monitored logs. LogSources contain details about what logs to get and where to get them, and which fields should be considered for parsing, for example timestamp, resource, and message. LogSources provides out-of-the-box setup and configuration for popular logsources like Windows Events and Kubernetes Events.

LogSources Data Collection

Use the LM Collector to collect logs for the different log source types, except for the Log Files type which uses the LM Collector.

LogSources have the same configuration properties as the LM Logs Collector configuration with system properties to map log ingestion, filters, and so on. However, instead of a single log collection configuration per collector, LogSources enables multi-log collection configurations on a single collector..

The following logsource types are supported:

  • Syslog
  • Windows Event Logging
  • Kubernetes Event Logging
  • Kubernetes Pods
  • Log Files

Processing Pipeline

The LogSources data collection processing pipeline consists of the following steps:

  1. Filtering to include or exclude data to reduce the log volume.
  2. Resource mapping and data enrichment (for some logsource types).

Logs are filtered on the collector side using critera based on standard comparison operators. Available operators vary depending on the logsource type.

Comparison operators:

  • Equal
  • NotEqual
  • GreaterThan
  • GreaterEqual
  • LessThan
  • LessEqual
  • Contain
  • NotContain
  • Exists
  • NotExist
  • RegexMatch
  • RegexNotMatch

The incoming log is parsed to populate and map the resource information. This can be for example a timestamp, device details, or some extra tags which can be used later for searching. Standard regular expression is used to get this information.

LogSource Types and Configuration Options

Similar to collector attributes, LogSources have additional attributes that you can configure to enrich the collected data and add tags for the logs. Examples of configurable attributes are applies to, log file path, timestamp, mapping from log, the LM property to match to, and other configuration items added through agent.conf for the collector.

You can also add the following attributes:

  • Static attributes, for example “cust.name = DataCenter”.
  • Token-based attributes, for example “cust.name = ##some.properties##”.

LogSources configuration components in the LogicMonitor portal:

  • Applies To: The resources to which the logsource is applied. 
  • Type: Select a supported logsource type.
  • Group: The LogSource group (optional).
  • Log Attributes: Varies based on logsource type.
  • Filters: Options to include/exclude sources.
  • Log Fields/Tags: Include custom metadata to be sent with the logs, either dynamic value parsed from the log, or a static value.
  • Resource Mapping: The resource the logs should map to and mapping method.
  • Other Options: For example alternative timestamp to use.

Syslog

For a Syslog logsource type, you can define:

  • Include filters
  • Log fields/tags
  • Resource mapping
  • Use received time stamp instead of log timestamp.

Windows Event Logging

For a Windows Event logsource type, you can define:

  • Exclude/include filters
  • Log fields/tags
  • Resource mapping

Kubernetes Event Logging

For a Kubernetes Event logsource type, you can define:

  • Include filters
  • Log fields/tags

Kubernetes Pods

For a Kubernetes Pods logsource type, you can define:

  • Include filters
  • Log fields/tags

Log Files

For a LM Logs logsource type, you can define:

  • Include filters
  • Log fields/tags

Note: This is a scrape logsource type which requires that you use the LM OTEL Collector.

In This Article