Alerts

Escalation Chains

Introduction to Escalation Chains

Once a triggered alert is matched to an alert rule, it is assigned an escalation interval and dispatched to an escalation chain. The escalation is made up of one or more stages, telling LogicMonitor what people (or third-party applications) should be notified of the alert, and in what order. Later stages of an escalation chain only come into play if the alert is still in effect and prior stage recipients have not acknowledged or suppressed the alert. You should create a chain for each functional group in your organization that will be receiving alert notifications (e.g. on-call engineers, network team, database team, etc). Once created, escalation chains are assigned from alert rules, as discussed in Alert Rules.

Adding Escalation Chains

You can create a new escalation chain from Settings | Escalation Chains | Add. As shown (and discussed) next, there are several settings to configure for escalation chains.

Name

The name of the escalation chain.

Description

The description for the escalation chain.

Enable Rate Limit

Check the Enable Rate Limit option if you would like to set the maximum number of alerts that can be sent to a stage within this escalation chain during a specified time period.

If the number of alerts delivered to the chain’s initial stage exceeds the rate limit, then a throttle message is sent to the individuals assigned to that stage. The message states that the number of alerts has exceeded the throttling level. From this point forward, alerts will be escalated to subsequent stages in accordance with your chain’s configuration. Throttle messages, however, will not be escalated and will continue to be sent to the first stage. Alert clear and acknowledgment notifications will still be sent to all parties involved, regardless of their escalation stage.

Rate Limit period (min)

The time period (in minutes) during which the number of alert notifications specified in the following Rate Limit alerts field can be delivered.

Rate Limit alerts

The maximum number of alert notifications that can be delivered during the Rate Limit Period. Note that re-sent alert notifications count towards this number.

Create time-based chain

If the Create time-based chain option is checked, you can configure an escalation chain that varies depending upon the day and time the alert is triggered. As shown next, time-based escalation chains consist of one or more subchains; each subchain consists of a day/time combination and the corresponding stage(s) and stage recipient(s) for that effective time. This functionality lets you route alert notifications to different recipients depending on the day and time that the alert is triggered. New subchains are added by clicking the + icon that displays to the right of the Subchains heading.

Whenever an alert is routed to a time-based escalation chain, the subchains are processed in order until a subchain has an effective time that matches the current day and time. If there is no matching subchain, the alert will not be routed anywhere. Once the subchain is chosen, alerts escalate through the subchain's specified stage(s) and stage recipient(s) the same as for normal chains.

Stages

For every escalation chain (or subchain), one or more stages can be configured. Stages consist of one or more recipients that alert notifications will be routed to. Stage one recipients will be notified first, and, if additional stages are present, the alert will continue escalating through subsequent stages if the alert is not acknowledged or cleared within the escalation interval, which is defined in the alert rule.

A new stage, and by definition its recipient(s), is added by clicking the + icon that displays to the right of the Stages heading. Additional stages can continue to be added in this way.

Note: Stages can be configured with no recipients. Most commonly used as a first stage, an empty stage is useful for delaying alert notifications for a particular DataSource, EventSource, etc. without impacting timely delivery of all alert notifications. An empty stage delays notification for the duration of the escalation interval (as defined in the corresponding alert rule), at which point the next stage is triggered.

Recipients

Each recipient consists of a user and a contact method, or a previously-configured group of users called a recipient group. Contact methods come from two places: the phone number and SMS email address defined for the user; and the custom delivery methods defined in the account. You also have the option of adding arbitrary email addresses as recipients for a stage.

Note: Your level of permissions determine which, if any, users/recipient groups are available for selection when assigning recipients from the Add User field.

CC

Recipients listed in this field will receive all notifications sent to every stage in the escalation chain. Multiple CC recipients should be separated by spaces.

Escalation Chain Examples

Example 1

The escalation chain depicted in the following screenshot will send alert notifications to a ticketing system, a HipChat room and Bill's email.

If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert escalates and email notifications are sent to Bill and Management. If the alert escalates again, a voice alert notification will be sent to Bill. Since there is an email specified in the CC field, [email protected] will be emailed for all three stages of the alert. If the alert remains active and unacknowledged for the escalation interval time period after the third stage, notifications will be repeatedly sent to the third stage recipients at the period specified by the escalation interval (e.g. if the escalation interval is 20 minutes, Bill will receive a voice alert every 20 minutes) until the alert clears or is acknowledged.

The escalation chain is configured to limit alert notifications to 20 alert notifications in 10 minutes. Note that resent alert notifications do increment the rate limit counter.

Example 2

The escalation chain shown in the following screenshot depicts a time-based chain that consists of three subchains.

If an alert triggers Monday through Friday between the hours of 8am to 5pm, this time-based escalation chain will send an SMS alert notification to Bill. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, resend an SMS alert notification to Bill.

If an alert triggers Monday through Friday between the hours of 5pm to 8am, this time-based escalation chain will post a message to a HipChat room. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, repost the message to the same HipChat room.

If an alert triggers any time on Saturday or Sunday, this time-based escalation chain will post a message to a HipChat room. If the alert is not acknowledged or cleared within the escalation interval defined in the alert rule that routes the alert to this chain, then the alert will escalate and since there is only one stage, repost the message to the same HipChat room.

Example 3

The escalation chain depicted in the following screenshot will send alert notifications to [email protected], but only after an escalation interval has passed.

The first stage of this escalation chain has no recipients assigned to it (i.e. it is an empty stage). Empty stages serve to delay alert notification for the duration of an escalation interval. In this case, assuming the escalation interval for the alert rule that routes to this chain is set at 15 minutes, [email protected] would not receive notification until 15 minutes have passed and only if the alert was not acknowledged or cleared within those 15 minutes. If the alert remains active and unacknowledged for the entirety of the second stage, it will escalate again, but since there is no third stage, notification will be resent to [email protected]

This escalation chain has no rate limits set so an unlimited number of alert notifications could be routed to [email protected] once the alert has escalated to the second stage. (Remember, no alerts will be delivered during the empty first stage.)