LogSource Overview

LogSource is an LM LogicModule that provide templates to help you enable logs and configure the sending of log data. LogSource contains details about what logs to get and where to get them, and which fields should be considered for parsing, for example timestamp, resource, and message. LogSource is available for Syslog, Windows Events, Kubernetes Events, and other common sources of log data.

The following explains the concept of a LogSource and how it works. For information on how to add a LogSource, see Configuring a LogSource.

LogSource Data Collection

LogSource for Syslog, Windows Events, and Kubernetes sources use the LM Collector to collect and forward log data. For information about collector versions, see Configuring a LogSource.

The Log Files LogSource type forwards traces from your instrumented applications to the LogicMonitor platform. This LogSource type uses the LM OTEL Collector to collect and forward log data.

For an overview of available and recommended methods for sending log data to LM Logs for ingestion, see About Log Ingestion.

LogSource Types

LogSource types are available for the following sources of log data:

• Kubernetes Event Logging
• Kubernetes Pods
• Log Files
• Script Logs
• Syslog Logs
• Windows Event Logging

Configuration Properties

A LogSource has the same set of configuration properties as in the LM Logs Collector configuration with system properties to map log ingestion, filters, and so on. 

However, instead of editing directly in the configuration files, the LogSource provides a user interface to walk you through the setup process. 

Additionally, instead of one log collection configuration per collector, a LogSource lets you configure multiple log collections on a single collector.

Processing Pipeline

The LogSource data collection processing pipeline consists of the following steps:

1. Filtering to include or exclude data to reduce the log volume.
2. Resource mapping and data enrichment (for some LogSource types).

Logs are filtered on the collector side using critera based on standard comparison operators like “Equal”, “Contain”, and “RegexMatch”. Available operators vary depending on the logsource type.

The incoming log is parsed to populate and map the resource information. This can be for example a timestamp, resource details, or some extra tags which can be used later for searching. Standard regular expression is used to get this information.

LogSource Types and Configuration Options

Similar to collector attributes, a LogSource has additional attributes that you can configure to enrich the collected log data. Examples of configurable attributes are “apply to”, “log file path”, “timestamp”, mapping from log, the LM property to match to, and other configuration items added through agent.conf for the collector.

You can also add the following attributes:

• Static attributes, for example “cust.name = DataCenter”.
• Token-based attributes, for example “cust.name = ##some.properties##”.

LogSources configuration components in the LogicMonitor portal:

• AppliesTo—The resources to which the LogSource is applied. 
• Type—Supported LogSource type.
• Group—The LogSource group (optional).
• Log Attributes—Varies based on logsource type.
• Filters—Options to exclude and include sources.
• Log Fields—Include custom metadata to be sent with the logs, either dynamic value parsed from the log, or a static value.
• Resource Mappings—The resource the logs should map to, and mapping method.

Enabling Preferred Log Collectors

Along with a preferred collector, you can identify a log collector group. The LM Collector can receive both metrics and logs from a resource. If logs are sent to a different collector than the monitoring collector, you need to define a logs collector group and preferred logs collector for each resource. For more information, see Configuring a LogSource.

If logs are sent to multiple collectors, the collector group will only ingest logs from the preferred or primary collector. This is done to achieve high availability, and is only available if logs are sent to multiple collectors.

A preferred log collector can also be applied to a resource group. When applying a preferred log collector to a group, the preferred collector will be applied to all current members of the group. Because resources can be members of multiple groups, preferred log collector configurations are not saved at the Resource Group level. Therefore you must reapply the preferred collector configurations if new resources are added to the resource group after the initial configuration.