Settings

Escalation Chains

Introduction to Escalation Chains

Once a triggered alert is matched to an alert rule, it is assigned an escalation interval and dispatched to an escalation chain. The escalation chain tells LogicMonitor what people or groups of people should be contacted about the alert, and in what order. You should create a chain for each functional group in your organization that will be receiving alert notifications (e.g. on-call engineers, network team, database team, etc).

Adding Escalation Chains

You can create a new escalation chain from Settings | Escalation Chains | Add. As shown (and discussed) next, there are several settings to configure for escalation chains.

Name

The name of the escalation chain.

Description

The description for the escalation chain.

Enable Rate Limit

Check the Enable Rate Limit option if you would like to set the maximum number of alerts that can be sent to a stage within this escalation chain during a specified time period.

If the number of alerts delivered to the chain’s initial stage exceeds the rate limit, then a throttle message is sent to the individuals assigned to that stage. The message states that the number of alerts has exceeded the throttling level. From this point forward, alerts will be escalated to subsequent stages in accordance with your chain’s configuration. Throttle messages, however, will not be escalated and will continue to be sent to the first stage. Alert clear and acknowledgment notifications will still be sent to all parties involved, regardless of their escalation stage.

Rate Limit period (min)

The time period (in minutes) during which the number of alert notifications specified in the following Rate Limit alerts field can be delivered.

Rate Limit alerts

The maximum number of alert notifications that can be delivered during the Rate Limit Period. Note that re-sent alert notifications count towards this number.

Create time-based chain

If the Create time-based chain option is checked, new options dynamically appear that allow you to configure an escalation chain that varies depending upon the day and time the alert is triggered. As shown next, time-based escalation chains consist of one or more subchains; each subchain consists of a day/time combination and the corresponding stage(s) and stage recipient(s) for that effective time. This functionality lets you route alert notifications to different recipients depending on the day and time that the alert is triggered. New subchains are added by clicking the + icon that displays to the right of the Subchains heading.

Whenever an alert is routed to a time-based escalation chain, the subchains are processed in order until a subchain has an effective time that matches the current day and time. If there is no matching subchain, the alert will not be routed anywhere. Once the subchain is chosen, alerts escalate through the subchain's specified stage(s) and stage recipient(s) the same as for normal chains.

Stages

For every escalation chain (or subchain), one or more stages can be configured. Stages consist of one or more recipients that alert notifications will be routed to. Stage one recipients will be notified first, and, if additional stages are present, the alert will continue escalating through subsequent stages if the alert is not acknowledged or cleared within the escalation interval, which is defined in the alert rule.

A new stage, and by definition its recipient(s), is added by clicking the + icon that displays to the right of the Stages heading. Additional stages can continue to be added in this way.

Note: Stages can be configured with no recipients. Most commonly used as a first stage, an empty stage is useful for delaying alert notifications for a particular DataSource, EventSource, etc. without impacting timely delivery of all alert notifications. An empty stage delays notification for the duration of the escalation interval (as defined in the corresponding alert rule), at which point the next stage is triggered.

Recipients

Each recipient consists of a user and a contact method, or a group of users. Contact methods come from two places: the phone number and SMS email address defined for the user; and the custom delivery methods defined in the account. You also have the option of adding arbitrary email addresses as recipients for a stage.

CC

Recipients listed in this field will receive all notifications sent to every stage in the escalation chain. Multiple CC recipients should be separated by spaces.

Escalation Chain Examples

Example 1

The escalation chain depicted in the following screenshot will send alert notifications to a ticketing system, a HipChat room and Bill's email.

If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert escalates and email notifications are sent to Bill and Management. If the alert escalates again, a voice alert notification will be sent to Bill. Since there is an email specified in the CC field, lmNetworking@logicmonitor.com will be emailed for all three stages of the alert. If the alert remains active and unacknowledged for the escalation interval time period after the third stage, notifications will be repeatedly sent to the third stage recipients at the period specified by the escalation interval (e.g. if the escalation interval is 20 minutes, Bill will receive a voice alert every 20 minutes) until the alert clears or is acknowledged.

The escalation chain is configured to limit alert notifications to 20 alert notifications in 10 minutes. Note that resent alert notifications do increment the rate limit counter.

Example 2

The escalation chain shown in the following screenshot depicts a time-based chain that consists of three subchains.

If an alert triggers Monday through Friday between the hours of 8am to 5pm, this time-based escalation chain will send an SMS alert notification to Bill. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, resend an SMS alert notification to Bill.

If an alert triggers Monday through Friday between the hours of 5pm to 8am, this time-based escalation chain will post a message to a HipChat room. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, repost the message to the same HipChat room.

If an alert triggers any time on Saturday or Sunday, this time-based escalation chain will post a message to a HipChat room. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, repost the message to the same HipChat room.

Example 3

The escalation chain depicted in the following screenshot will send alert notifications to IT@acme.com, but only after an escalation interval has passed.

The first stage of this escalation chain has no recipients assigned to it (i.e. it is an empty stage). Empty stages serve to delay alert notification for the duration of an escalation interval. In this case, assuming the escalation interval for the alert rule that routes to this chain is set at 15 minutes, IT@acme.com would not receive notification until 15 minutes have passed and only if the alert was not acknowledged or cleared within those 15 minutes. If the alert remains active and unacknowledged for the entirety of the second stage, it will escalate again, but since there is no third stage, notification will be resent to IT@acme.com.

This escalation chain has no rate limits set so an unlimited number of alert notifications could be routed to IT@acme.com once the alert has escalated to the second stage. (Remember, no alerts will be delivered during the empty first stage.)