Loading full navigation.

The LogFiles LogSource enables LM Logs to collect logs that are written to files on disk by a deployed and registered LogicMonitor OpenTelemetry (OTel) Collector. After the Collector is configured, you can create a LogFiles LogSource and apply it to the monitored resource that generates the log files.
For more information on configuring the OTel Collector, see OpenTelemetry Installation.

Note: Starting with OpenTelemetry Collector version 5.2.0.0, LogicMonitor uses improved log-to-resource mapping to associate logs with the correct cloud resource instead of defaulting them to the LM OTel Collector Name. If logs are not mapped to a resource, they are classified as “deviceless”. This occurs because either the host is not monitored in LogicMonitor or it is not a cloud based resource.

To manually associate logs with a resource, set the LM_DEVICE_ATTRIBUTES environment variable using key-value pairs (for example, LM_DEVICE_ATTRIBUTES="key1=value1".)

AppliesTo in Log Files LogSource

The AppliesTo Filter (ATF) determines which log data this LogSource applies to.

For Log Files collected through OpenTelemetry, the AppliesTo expression must reference the OpenTelemetry Collector identifier. If the AppliesTo expression does not match the OpenTelemetry Collector name, the LogSource does not process incoming logs.

AppliesTo Section in LogFiles LogSource Configuration

Important: Configure the AppliesTo expression to match the LM OpenTelemetry (OTel) Collector resource, not the monitored resource that generates the log files. If the AppliesTo expression matches the monitored resource instead of the LM OpenTelemetry Collector resource, the LogSource will not collect logs.

Example AppliesTo Expressions

The following examples demonstrate common AppliesTo expressions that you can copy and modify for your environment.

To match a single OpenTelemetry Collector, use the following expression with its resource display name:

system.displayname == "TEST-OTEL-COLLECTOR_OTEL_48439"

To match multiple OpenTelemetry Collector, use the following expression with their resource display names:

system.displayname == "TEST-OTEL-COLLECTOR_OTEL_48439" || system.displayname == "PROD-OTEL-COLLECTOR_OTEL_51234"

Add additional system.displayname expressions as needed by using the || (OR) operator.

To match an OpenTelemetry Collector Group, use the following expression:

isInGroup("OpenTelemetry Collectors")

Replace OpenTelemetry Collectors with the name of your LogicMonitor resource group.

For more information about creating AppliesTo expressions, see AppliesTo Function Configuration.

Filtering in Log Files LogSource

After the AppliesTo Filter matches, the system evaluates include and exclude filters to determine whether a log is ingested or dropped. You can toggle the AND or OR switch to control how this evaluation occurs. By default, these switches are disabled.

Include Filters

You can add include filters to specify which resources are eligible for log ingestion, such as a specific resource type or application. Logs that match the include filter criteria are forwarded to the log ingestion process.

Exclude Filters

You can add exclude filters to prevent logs from specific resources, resource types, or attributes from being ingested. Logs that match an exclude filter are dropped and are not forwarded to the log ingestion process.

The following table lists the available parameters when configuring LogFiles LogSource:

AttributesComparison operatorValue example
MessageRegexMatch, RegexNotMatch.Regular expression

Log Fields in Log Files LogSource

You can configure Log Fields (tags) to send additional metadata with the logs.

The following table lists the available parameters:

MethodKey exampleValue exampleDescription
Static“Customer”“Customer_XYZ”Sends a fixed key-value pair with every log.
Dynamic(REGEX)“Host”“host=*”Extracts the value from the log message using a regular expression.
LM Property(Token)“Device”“##system.deviceId##”Uses a LogicMonitor property token to populate the value at runtime.
Dynamic Group Regexuser,host,statususer=(\w+)\s+host=([\w.]+)\s+status=(\d+)Enables extraction of multiple metadata fields from a single log line using one regular expression with multiple capture groups

Note: You can use the optional resource_type property to define the resource type applied to all ingested logs.

The resource type value that you specify is used for all logs. If the value is not specified, the system assigns a resource type based on the value of the predef.externalResourceType field in the resource settings.

Requirements for Configuring Log Files LogSource

To configure Log Files LogSources, you must have the following:

Note: This LogSource requires a LogicMonitor OpenTelemetry (LM OTEL) Collector to be installed, registered in LogicMonitor, and in a Running state. Log collection does not occur unless an active LM OTEL Collector is available to collect and forward the log data.

Configuring Log Files LogSource

  1. In LogicMonitor, navigate to more options More > Modules > select Add Datasource Add.
  2. In the Add window, select LogSource. The Add New LogSource window displays.
  3. Configure the Basic Information for Info section with required information.
    For more information, see Configuring Basic Information for LogSource. 
  4. In the Summary for Info section, do the following:
    1. Select “LM Logs: Log Files“ in the Type dropdown menu.
    2. Provide the path name for the log file in the Log File Path field.
      You can enter one of the following:
      • A single log file path.
        For example:\\share\logfile.log
      • A log file path that uses wildcards to collect multiple log files or rollover log files across directories.
        For example:C:\inetpub\logs\LogFiles\W3SVC*\u_ex*.log
        Add New Logsource Info section
  5. Assign Access Groups to enable who can view, manage, and interact with the logs.
    For more information, see Access Groups.
  6. In the AppliesTo field, enter an AppliesTo expression that matches the OpenTelemetry Collector name deployed on the resource.
    For more information, see AppliesTo Function Scripting Overview.
    AppliesTo example for LogFiles LogSource
  7. Configure the fields in the Filters section.
    For more information for configuring Filters for LogSource, see Configuring Filters for LogSource.
  8. Configure Log Fields (tags) to send additional metadata with the logs.
    For more information for configuring Log Fields in LogSource, see Configuring Log Fields in LogSource .
  9. In the Sensitive Data Masking section, you can either select Load Default Data Masking Rules to use the predefined masking rules provided by LogicMonitor or create custom masking rules to define how sensitive data is detected and masked.
    To create a custom masking rule, do the following:
    1. Select Add icon blue Add Sensitive Data Masking Rules.
    2. Select an option from the Type dropdown menu. For example, Regex.
    3. In the Key field, enter the masking rule name. This identifies the type of sensitive data being masked.
    4. In the Masking Rule field, define the pattern used to detect sensitive data.
    5. In the Masked Value field, enter the replacement value. This value displays instead of the sensitive data.
    6. (Optional) Add additional details in the Description field.
      Add Masking Rule modal for LogFiles LogSource
    7. Select save Save. The masking rule is added.
      Note:
      • To edit an existing masking rule, select the masking rule and update the required fields.
      • To disable a sensitive data masking rule, delete the rule.
  10. Select save Save.
    The LogSource is configured and available for use.