Log Processing Pipelines

There are certain types of log events or anomalies that you may want to always track and take action on, such as errors or exceptions that should notify to be resolved immediately.

Log pipelines are a series of processing steps executed on log events that match a certain set of filters. Once you have defined filters for the logs you want to track, you can define other processing steps, such as setting alert conditions on them.

Reviewing Pipelines

On the Logs page, click on the Pipelines icon to open the Pipelines page. From here you can review and manage existing pipelines, and add new ones. 

• Pipeline shows the name of the pipeline.
• Query lists the filtering conditions that define the log events in the pipeline. For example, the resources where the logs are received from.
• Alert Conditions lists the number of alert conditions defined for that pipeline. Click on the icon or count to open the Alert Conditions page for the pipeline to configure alert conditions. See Log Alert Conditions.
• Description provides information about the pipeline.
• Click on a pipeline in the list to review and edit pipeline settings, or to delete the pipeline.

Adding Pipelines

You can add pipelines from the Logs or Pipelines pages:

• From a log event or anomaly in the Logs page: Open the menu for Resource or Groups, and select Create Pipeline. This will open the Add Pipeline dialog with the Logs query field prefilled with matching events for the selected resource or groups.
• From the Pipelines page: Click the plus sign to open the Add Pipeline dialog and add a new pipeline.

In the Add Pipeline dialog, enter information as follows:

1. Enter a Display Name (required) and Display Description. These will appear in the pipeline list.
2. Under Logs query, define the events to match using the logs query language (unless prefilled). See Query Language Overview. Click the arrow to preview the results and refine your query before you save.
3. Click the Save icon to add the pipeline.

When you return to the Pipelines page, you can review the pipeline you created in the table.

After adding a pipeline, you may want to define alert conditions to notify when certain logs are received in the pipeline. See Log Alert Conditions.