Come join our live training webinar every other Wednesday at 11am PST and hear LogicMonitor experts explain best practices and answer common questions. We understand these are uncertain times, and we are here to help!
Once a triggered alert is matched to an alert rule, it is assigned an escalation interval and dispatched to an escalation chain. The escalation is made up of one or more stages, telling LogicMonitor what people (or third-party applications) should be notified of the alert, and in what order. Later stages of an escalation chain only come into play if the alert is still in effect and prior stage recipients have not acknowledged or suppressed the alert. You should create a chain for each functional group in your organization that will be receiving alert notifications (e.g. on-call engineers, network team, database team, etc). Once created, escalation chains are assigned from alert rules, as discussed in Alert Rules.
Note: It’s possible that an alert could match an alert rule, but still not be forwarded to the rule’s specified escalation chain. This scenario occurs if alert notification suppression is enabled via one of LogicMonitor’s AIOps features that serve to reduce targeted alert noise. For more information, see Enabling Dynamic Thresholds for Datapoints and Enabling Root Cause Analysis respectively.
You can create a new escalation chain from Settings | Escalation Chains | Add. As shown (and discussed) next, there are several settings to configure for escalation chains.
The name of the escalation chain.
The description for the escalation chain.
Check the Enable Rate Limit option if you would like to set the maximum number of alerts that can be sent to a stage within this escalation chain during a specified time period.
If the number of alerts delivered to the chain’s initial stage exceeds the rate limit, then a throttle message is sent to the individuals assigned to that stage. The message states that the number of alerts has exceeded the throttling level. From this point forward, alerts will be escalated to subsequent stages in accordance with your chain’s configuration. Throttle messages, however, will not be escalated and will continue to be sent to the first stage. Alert clear and acknowledgment notifications will still be sent to all parties involved, regardless of their escalation stage.
The time period (in minutes) during which the number of alert notifications specified in the following Rate Limit alerts field can be delivered.
The maximum number of alert notifications that can be delivered during the Rate Limit Period. Note that re-sent alert notifications count towards this number.
If the Create time-based chain option is checked, you can configure an escalation chain that varies depending upon the day and time the alert is triggered. As shown next, time-based escalation chains consist of one or more subchains; each subchain consists of a day/time combination and the corresponding stage(s) and stage recipient(s) for that effective time. This functionality lets you route alert notifications to different recipients depending on the day and time that the alert is triggered. New subchains are added by clicking the + icon that displays to the right of the Subchains heading.
Whenever an alert is routed to a time-based escalation chain, the subchains are processed in order until a subchain has an effective time that matches the current day and time. If there is no matching subchain, the alert will not be routed anywhere. Once the subchain is chosen, alerts escalate through the subchain’s specified stage(s) and stage recipient(s) the same as for normal chains.
For every escalation chain (or subchain), one or more stages can be configured. Stages consist of one or more recipients that alert notifications will be routed to. Stage one recipients will be notified first, and, if additional stages are present, the alert will continue escalating through subsequent stages if the alert is not acknowledged or cleared within the escalation interval, which is defined in the alert rule.
A new stage, and by definition its recipient(s), is added by clicking the + icon that displays to the right of the Stages heading. Additional stages can continue to be added in this way.
Note: Stages can be configured with no recipients. Most commonly used as a first stage, an empty stage is useful for delaying alert notifications for a particular DataSource, EventSource, etc. without impacting timely delivery of all alert notifications. An empty stage delays notification for the duration of the escalation interval (as defined in the corresponding alert rule), at which point the next stage is triggered.
Stage recipients can be any combination of individual users, alert integrations, recipient groups, or arbitrary emails. Each type of recipient is discussed next.
Note: Your level of permissions determine which, if any, users/recipient groups are available for selection when assigning recipients from the Add User field.
When designating a user account as a recipient, you can choose to deliver notifications to either the email address, SMS email address, or phone number stored in the user account. For more information on user accounts, see Users.
Recipient groups act as time-saving shortcuts when the same set of recipients needs to be notified of a variety of different types of alerts. Recipient groups can consist of user accounts, arbitrary email addresses, and even other recipient groups. For more information, see Recipient Groups.
You can also choose to deliver alert notifications to any alert integrations defined in your LogicMonitor account. This allows you to direct alert notification delivery to third-party ticketing or team collaboration applications. In order to select an alert integration, you must first indicate a user account and then, from the user account’s contact method, select the name of the integration.
Although it technically doesn’t matter which user account you select (the entire list of integrations always displays regardless of user chosen), LogicMonitor recommends that you create a dedicated user account to associate with integrations. For more information on alert integrations, see Alert Integrations Overview.
In the Arbitrary Emails field, you can enter one or more email addresses that are not associated with existing user accounts or recipient groups. Separate multiple email addresses with spaces.
Note: An arbitrary email address can represent a distribution list, but you must ensure that the distribution list itself is configured to allow external addresses (known to be relevant for Exchange email servers).
Recipients listed in this field will receive all notifications sent to every stage in the escalation chain. Multiple CC recipients should be separated by spaces.
The escalation chain depicted in the following screenshot will send alert notifications to a ticketing system, a chat room, and Bill’s email.
If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert escalates and email notifications are sent to Bill and Management. If the alert escalates again, a voice alert notification will be sent to Bill. Since there is an email specified in the CC field, [email protected] will be emailed for all three stages of the alert. If the alert remains active and unacknowledged for the escalation interval time period after the third stage, notifications will be repeatedly sent to the third stage recipients at the period specified by the escalation interval (e.g. if the escalation interval is 20 minutes, Bill will receive a voice alert every 20 minutes) until the alert clears or is acknowledged.
The escalation chain is configured to limit alert notifications to 20 alert notifications in 10 minutes. Note that resent alert notifications do increment the rate limit counter.
The escalation chain shown in the following screenshot depicts a time-based chain that consists of three subchains.
If an alert triggers Monday through Friday between the hours of 8am to 5pm, this time-based escalation chain will send an SMS alert notification to Bill. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, resend an SMS alert notification to Bill.
If an alert triggers Monday through Friday between the hours of 5pm to 8am, this time-based escalation chain will post a message to a chat room. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, repost the message to the same chat room.
If an alert triggers any time on Saturday or Sunday, this time-based escalation chain will post a message to a chat room. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, repost the message to the same chat room.
The escalation chain depicted in the following screenshot will send alert notifications to [email protected], but only after an escalation interval has passed.
The first stage of this escalation chain has no recipients assigned to it (i.e. it is an empty stage). Empty stages serve to delay alert notification for the duration of an escalation interval. In this case, assuming the escalation interval for the alert rule that routes to this chain is set at 15 minutes, [email protected] would not receive notification until 15 minutes have passed and only if the alert was not acknowledged or cleared within those 15 minutes. If the alert remains active and unacknowledged for the entirety of the second stage, it will escalate again, but since there is no third stage, notification will be resent to [email protected]
This escalation chain has no rate limits set so an unlimited number of alert notifications could be routed to [email protected] once the alert has escalated to the second stage. (Remember, no alerts will be delivered during the empty first stage.)
In This Article