Log Processing Pipelines

Log pipelines raise alerts for specific log messages or categories of log messages. The pipelines themselves do not trigger any alerts. Instead they group logs into categories under which alert conditions (pipeline alerts) must be defined to trigger the desired alerts. Typically log pipelines are applied to resources or resource groups, and the pipeline alert conditions set the alert criteria, for example matching text in the log message. 

How Pipelines Work

Pipelines and pipeline alerts are created using the LM Logs Query Language to identify which logs should trigger alerts. The querying can range from broad matches on entire resources or resource groups, to very specific matches on individual log messages. You can also apply regular expressions to match specific log patterns in cases where content differs between logs.

You can for example have certain types of log events or anomalies that you always want to track and take action on, such as errors or exceptions that should notify to be resolved immediately. Start by creating a log pipeline and defining filters for the logs you want to track, as described in the following. Then continue by creating alert conditions for the pipeline.               

Viewing Pipelines

On the Logs page, select the Pipelines icon to open the Pipelines page. From here you can review and manage existing pipelines, and add new ones. 

• Pipeline—Shows the name of the pipeline.
• Query—Lists the filtering conditions that define the log events in the pipeline. For example, the resources where the logs are received from.
• Alert Conditions—Lists the number of alert conditions defined for that pipeline. Select the icon or count to open the Alert Conditions page for the pipeline to configure alert conditions. For more information, see Log Alert Conditions.
• Description—Provides information about the pipeline.
• Select a pipeline in the list to review and edit pipeline settings, or to delete the pipeline.

Adding Pipelines

You can add pipelines from the Logs or Pipelines pages:

• From a log event or anomaly in the Logs page: Open the menu for Resource or Groups, and select Create Pipeline. This opens the Add Pipeline dialog with the Logs query field prefilled with matching events for the selected resource or groups.
• From the Pipelines page: Select the plus sign to open the Add Pipeline dialog and add a new pipeline.

In the Add Pipeline dialog, enter information as follows:

1. Enter a Display Name (required) and Display Description. These will appear in the pipeline list.
2. Under Logs query, define the events to match using the logs query language (unless prefilled). For more information, see Query Language Overview. Select the arrow to preview the results and refine your query before you save.
3. Select the Save icon to add the pipeline.

When you return to the Pipelines page, you can review the pipeline you created in the table.

After adding a pipeline, continue by defining its alert conditions. For more information, see Log Alert Conditions.