Log Processing Pipelines
Last updated on 10 May, 2023Log pipelines raise alerts for specific log messages or categories of log messages. The pipelines themselves do not trigger any alerts. Instead they group logs into categories under which alert conditions (pipeline alerts) must be defined to trigger the desired alerts. Typically log pipelines are applied to resources or resource groups, and the pipeline alert conditions set the alert criteria, for example matching text in the log message.
How Pipelines Work
Pipelines and pipeline alerts are created using the LM Logs Query Language to identify which logs should trigger alerts. The querying can range from broad matches on entire resources or resource groups, to very specific matches on individual log messages. You can also apply regular expressions to match specific log patterns in cases where content differs between logs.
You can for example have certain types of log events or anomalies that you always want to track and take action on, such as errors or exceptions that should notify to be resolved immediately. Start by creating a log pipeline and defining filters for the logs you want to track, as described in the following. Then continue by creating alert conditions for the pipeline.
Recommendation: Pipelines and alert conditions are checked for every received log. For performance reasons, it is therefore recommended that your pipeline definitions are narrowed to for example specific device types and their associated alert conditions. Consider this especially in cases where you know that a device log will never match the alert, for example a Windows Event Log will never match Syslog facility and so on.
Viewing Pipelines
On the Logs page, select the Pipelines icon to open the Pipelines page. From here you can review and manage existing pipelines, and add new ones.
- Pipeline shows the name of the pipeline.
- Query lists the filtering conditions that define the log events in the pipeline. For example, the resources where the logs are received from.
- Alert Conditions lists the number of alert conditions defined for that pipeline. Select the icon or count to open the Alert Conditions page for the pipeline to configure alert conditions. For more information, see Log Alert Conditions.
- Description provides information about the pipeline.
- Select a pipeline in the list to review and edit pipeline settings, or to delete the pipeline.
Adding Pipelines
You can add pipelines from the Logs or Pipelines pages:
- From a log event or anomaly in the Logs page: Open the menu for Resource or Groups, and select Create Pipeline. This opens the Add Pipeline dialog with the Logs query field prefilled with matching events for the selected resource or groups.
- From the Pipelines page: Select the plus sign to open the Add Pipeline dialog and add a new pipeline.
In the Add Pipeline dialog, enter information as follows:
- Enter a Display Name (required) and Display Description. These will appear in the pipeline list.
- Under Logs query, define the events to match using the logs query language (unless prefilled). For more information, see Query Language Overview. Select the arrow to preview the results and refine your query before you save.
- Select the Save icon to add the pipeline.
When you return to the Pipelines page, you can review the pipeline you created in the table.
After adding a pipeline, continue by defining its alert conditions. For more information, see Log Alert Conditions.
Note: You can create log processing pipelines also for unmapped resources. Since there is no LM-monitored resource or resource group for these, LogicMonitor automatically associates the pipeline with a special resource and resource group. The resource name will be the same as the pipeline name. The resource group for unmapped resources is called “LogPipelineResources”. For more information, see Deviceless Logs.