Log Alert Conditions

Last updated on 09 June, 2023

Log alerts are alert conditions based on log events and log pipelines. Typically log pipelines are applied to resources or resource groups, and the associated log alert conditions set the alert criteria, for example matching text in the log message. 

Log alert conditions use regular expression patterns to match ingested logs and trigger LogicMonitor alerts to send you notifications when these log events or anomalies occur.

With log alerts, you can speed up your investigation by adding alert conditions to track the logs you always want to know about or creating alerts on a detected anomaly.

Viewing Alert Conditions

On the Logs page, select the Pipeline Alerts icon to open the Alert Conditions page where you can review and manage alert conditions.

  • Severity—Displays the level (Critical, Error, Warning) configured for the alert condition.
  • Display As—Shows the configured display name of the alert condition.
  • Logs query—Displays the regular expression pattern to match log events that will trigger the alert.
  • Pipeline—Displays the pipeline that the alert condition applies to.
  • Description—Provides a short explanation of the alert condition.
  • Active—Lets you toggle the alert condition on and off.
  • Filter alert conditions associated with a specific pipeline.

Adding Alert Conditions

You can add alert conditions directly from any log event or anomaly listed in the Logs page, or from the Pipelines and Pipeline Alerts pages.

  • From a log event or anomaly on the Logs page: Open the action menu at the beginning of the log line and select “Create alert condition” to open the Add Alert Condition dialog.
  • From the Pipelines page: Select the Alert Conditions icon or count to open the Alert Conditions page, and select the plus sign to add an alert condition.

Note: You need at least one pipeline to be able to add a log alert condition. For more information, see Log Processing Pipelines. Log queries for alert conditions cannot include aggregate functions.

In the Add Alert Condition dialog, enter information as follows:

  1. Select the type of Alert severity level (Critical, Error, Warning) to generate when conditions are met.
  2. Enter a Display name and a Description to be displayed in the list of alert conditions.
  3. Select a Pipeline to apply the alert condition (unless prefilled).
  4. Unless prefilled, enter a Logs query to filter the events in the pipeline. For more information see Writing a Filtering Query. Select the arrow to preview the results and refine your query before you save.
  5. From Trigger alert, to activate an alert condition, do the following:
    1. for each matching log – Select this option if you want to generate an alert for each log matching a condition.
    2. only after matching logs within – Select this option to generate an alert if a certain number of logs match a condition within a defined time period. To generate an alert, there should be at least 1 matching log. You can select the following time intervals: 10 minutes, 30 minutes, 1 hour, 8 hours, and 1 day.

Important: After an alert is generated, to generate the next alert, the time of the first log is considered as the starting point of the time interval and no alert is generated until the time interval is complete. For example, if you select only after 3 matching logs within 10 minutes, the alert is generated after the third log with matching condition is processed within 10 minutes. If there are 6 logs with matching condition within 10 minutes, then the alert for the first set of 3 matching logs is generated. After that, the time of the first log is considered as the starting point of the 10 minutes time interval and no alert is generated until the 10 minutes time interval is complete.

  1. To define Alert clear conditions, enter the time that the alert will persist. Default is 60 minutes.
  2. Toggle the Auto-clear after acknowledge switch on to clear the alert after it is acknowledged.
  3. Toggle the Active switch on to activate the alert condition.
  4. Select the Save icon to add the alert condition.

Note: The Clear after setting will clear alerts after the specified time, even if the Auto-clear after acknowledge toggle is off. Alerts will only be acknowledged if the Auto-clear after acknowledge toggle is on.

When you return to the Alert Conditions page, you can review the alert you created in the table.

Example

The pipeline Windows Servers (_resource.group.name="Windows") targets a subset of logs related to a “Windows” resource group. The alert triggers on specific keywords or values (eventcode 1100 and application log files). You do not need to include the resource group name in the query for the alert condition (however it would work).

Acknowledging Alerts

When log alerts conditions are matched, they trigger standard LogicMonitor alert notifications based on the alert settings (Critical, Warning, Error) and will route through the configured escalation chain.

You can also:

  • View the log alert in the Logs graph, where it will display as a line alongside the alerting log event.
  • Acknowledge or clear the alert in the Alerts list.

Limitations

Each pipeline can have no more than 90 alerts per minute. If the rate of alerts exceed this limit, they will be discarded and not processed.

Note: Log pipeline alert conditions are limited to 20 per portal. To increase the number of log pipeline alert conditions, contact your Customer Success Manager.

In This Article