Log Processing

Last updated on 07 December, 2022

Log events are received through the LogIngest API, before being mapped to resources in LogicMonitor and further processed and stored.

The high-level process includes the following:

  • Collection: Locally using agents, or sent by the data source.
  • Ingestion: Sending the collected data to an ingestion layer.
  • Storage: Storing logs as long as they are needed for compliance.
  • Analysis: Through search or a live analysis system, and alert creation.

The log processing is described in more detail in the following.

Log Processing Flow

The log processing flow is explained in more detail in the following.

  1. Incoming Traffic. Log events are received from various resources. These can be a host machine, LM Log collectors or log servers, cloud services, or other applications and technologies. See LM Logs Overview.
  2. Log Ingestion Endpoint. Forwards raw logs to the log ingestion services for further processing.
  3. Ingestion Services. Receives log events as JSON payloads. Performs validation, authentication, authorization and resource mapping.
  4. Processing Services. Consumes ingested logs, applies anomaly detection algorithm, and prepares logs for storage. Triggers alerts based on pipeline and alert condition configurations.
  5. LM Logs UI. Receives user input such as log queries, added pipelines, and log usage.
  6. Querying Services. Processes queries received from the UI and sends saved queries for storage.
  7. Storage. Stores events and anomalies from log processing, and queries from querying services.

Log Processing Components

See the following for more details about components involved in the log processing flow.

Logs Ingest Public API

The LogIngest API retrieves log events from different collection sources. You can also send logs directly to LM through the logs ingestion API. See Sending Logs to the LM Logs Ingestion API.

Log Ingestion

When setting up LM Logs, resources and services are configured to forward data to one of various log ingestion methods. See About Log Ingestion.

Log Anomaly Detection

Anomalies are changes in log data that fall outside of normal patterns. LM Logs detects anomalies based on parsed log event structures. See Log Anomaly Detection.

Log Processing Pipelines

Log events are channeled into pipelines analyzing structure patterns looking for anomalies. You can define filters and set alert conditions to track specific logs. See Log Processing Pipelines.

Log Alert Conditions

Log alerts are alert conditions based on log events and log processing pipelines. Alert conditions use regular expression patterns to match ingested logs and trigger LM alerts. See Log Alert Conditions.

Log Queries

The logs query language expands the standard search capabilities beyond keyword searches and filtering to narrow down information when troubleshooting. See Query Language Overview

Log Retention

Log retention refers to the archiving of event logs, concerning the duration for which you store these log entries. See Data Retention.   

Log Usage 

Logs volume usage and ingestion quotas are collected and enriched by the log process, and retrieved from metadata for input to the logs usage reporting. See LM Logs Usage Monitoring.

Collectors

Collectors retrieves data from monitored resources and provide the data to the log ingestion. There several types of collectors for log ingestion. See About the LM Collector.

Alerts

Alerts are generated based on filters and conditions configured for resources. The overall Alerts page displays alerts across all monitored resources. See What does LM Alert on?.

In This Article