Troubleshooting Alert Delivery

Overview

All alerts display within your LogicMonitor interface. You can additionally choose to have alerts routed (using alert rules and escalation chains) via a variety of delivery methods, including text, email, voice call, or integration with a third-party app such as a ticketing system.

If you think you aren’t receiving routed alert notifications that you should be receiving—or if you think you are receiving too many alert notifications, follow the troubleshooting tips listed in the following sections.

Troubleshooting Missing Alert Notifications

The generation of alerts and subsequent routing of alert notifications has many moving parts in LogicMonitor. In addition, there are features that seek to intelligently suppress alert notifications under targeted circumstances in order to reduce alert noise. Review the possible causes for missing alert notifications in the following sections to see if any apply to your situation.

Are the alerts being generated?

First, it is important to distinguish whether your problem is with alert generation or alert delivery. All alerts, whether routed or not, display on the Alerts page/tab in your LogicMonitor account.

If you cannot find the alerts for which you think you should be receiving notifications for within the interface (make sure to manually include cleared alerts in your filter criteria), then the alert probably isn’t being triggered in the first place. In this case, you’ll need to adjust the triggering criteria (e.g. datapoint thresholds, website alerting configurations, etc.) such that alerts are triggered as you expect.

Does the alert match an alert rule?

If you do see the alerts within your LogicMonitor account, but you aren’t receiving alert notifications, then you need to determine whether you have an alert rule configured to route notifications for that type of alert. Remember that in order for alert notifications to be routed, the particular website, EventSource, resource datapoint, etc. must match an alert rule, and this alert rule must reference an escalation chain that contains the recipients that you want to deliver notifications to.

In most cases, alert notifications do not reach their intended destinations because they are being matched to an unexpected alert rule. To troubleshoot this possibility, you can:

• Test alert routing, as discussed in Testing Alert Delivery.
• Display the Alert Rule column on the Alerts page to see what alert rule is matching the alert. For more information on customizing columns on the Alerts page, see Managing Alerts from the Alerts Page.

Was the alert triggered during SDT?

Keep in mind that alerts that occur during periods of scheduled downtime (SDT) display in the LogicMonitor interface, but are never routed for external delivery. A resource (or website, EventSource, etc.) that is in SDT is denoted with a unique clock icon throughout the LogicMonitor interface to help you quickly identify SDT status.

Are alert notifications being suppressed by one of LogicMonitor’s AIOps features?

It is possible that an alert could match an alert rule, but still not be routed beyond LogicMonitor’s interface. This scenario occurs if alert notification suppression is enabled via one of LogicMonitor’s AIOps features that serve to intelligently reduce alert noise. For more information on these features, see Enabling Dynamic Thresholds for Datapoints and Enabling Root Cause Analysis respectively.

Is the escalation chain rate limited?

If rate limiting is enabled for the escalation chain, the number of alert notifications that can be sent to the escalation chain in a specified time period is limited. For more information on rate limiting, see Escalation Chains.

Is the contact information for your user incorrect?

Escalation chain recipients are typically specified using user accounts. If the information for a user in an escalation chain is incorrect, alert notifications won’t be delivered correctly. Double check the contact settings (Settings | Users & Roles | Users) for the user account in question.

Is your receiving email or SMS gateway refusing messages or queuing messages for delivery?

Alert notification messages could be refused or queued because of spam control, gateway misconfiguration, DNS issues, etc.

Was the alert notification marked as spam by your email client?

Check your spam folder.

Is the missing alert notification for a Collector, website, EventSource, or external alert?

• Collector. If notifications for Collector down alerts are not being received, make sure there is a valid escalation chain specified for your Collector, as discussed in Monitoring Your Collectors.
• Websites. LogicMonitor uses checkpoints to determine if websites are accessible. Configured Web Checks and Ping Checks allow you to differentiate alert notification settings depending upon the failure of multiple or individual checkpoints. Make sure these settings are as you expect. For more information on alert settings for website, see Adding a Ping Check or Adding a Web Check.
• EventSources. LogicMonitor automatically suppresses some duplicate EventSource alert notifications. Review the duplicate suppression details provided in Creating EventSources to ensure behavior is as you expect.
• External alerting. Ensure that the referenced Collector is online.

Troubleshooting Too Many Alert Notifications

Receiving too many LogicMonitor alert notification emails can ultimately lead to alert fatigue and the ignoring of important alerts. Some tips for avoiding this undesirable situation include:

• Tuning your static datapoint thresholds to suit your environment, as discussed in Tuning Static Thresholds for Datapoints.
• Enabling AIOps features that serve to intelligently suppress alert notifications for targeted situations. For more information on these features, see Enabling Dynamic Thresholds for Datapoints and Enabling Root Cause Analysis respectively.
• Avoiding routing all alerts. Some alerts, such as alerts with a severity of warning (as compared to error or critical), are better viewed regularly in LogicMonitor reports, or being posted to a ticketing system using custom alert delivery methods.