Developing a monitoring and alert management strategy that reduces the frequency of alerts while increasing the effectiveness of the alerts you do receive is easier than you might think. Follow these 9 steps and not only will alert storms be a thing of the past, but you and your IT team will be able to keep your systems and applications up and available. For more information on how to prevent alert storms, develop a stronger, more effective alerting strategy and help keep your organization sailing smoothly:
Video Transcription
Have you ever been caught in an IT alert storm? Notifications coming at you from every direction, yeah, it’s not great. Or have you ever been perfectly in the zone at work, only to receive an angry phone call out of the blue, asking why the website’s online shopping functionality is down?
The world is counting on you to prevent alert storms, but you can’t do anything about them unless you know something’s wrong in the first place. Developing an effective alerting strategy is often much simpler than you might think.
Here are nine steps to follow to avoid IT alert storms, and to get out in front of system failures, saving you time and frustration down the road. Step one, adopt sensible escalation policies. You need to distinguish alert levels and route warnings and mission critical alerts appropriately.
No one enjoys being woken up in the middle of the night, especially for a meaningless alert, but if there’s a mission critical incident, well that’s a whole different story.
Step two, route the right alerts to the right people. The networking team probably isn’t going to react to an alert about a product shortage in the east coast warehouse.
Step three, tune your thresholds. Make sure every alert is valid and worthwhile, by tuning alerting thresholds to get rid of false positives or making sure alerts are turning off on test systems.
Step four, investigate those alerts that are triggered when things objectively seem okay on the surface. If it’s truly a false alarm, adjust your thresholds to be less sensitive, or disable the alert.
Step five, ensure that all alerts are acknowledged, resolved, and cleared. When there is a problem, keeping your alert queue clear of old or unacknowledged alerts makes it easier to quickly hone in on immediate issues.
Step six, sort alerts by duration periodically. Again, practice good alert queue hygiene by clearing alerts that have been sitting uncleared for more than a day.
Step seven, create weekly alert reports that cover the week, and deliver them to your department. By meeting with your team to review top alerts, you can investigate monitoring, system, or operational processes to maximize efficiencies and minimize time wasted, and the frequency of alerts.
Step eight, consider a weekly alert summary report. If your weekly alert report is too large to be useful, consider putting together a weekly report listing just the hosts with the most alerts. This can help you to resolve resource issues, and tune thresholds, to get the weekly alert report discussed in step seven to truly be useful.
Step nine, use trend reports. Keep tabs on the hosts and groups with the most alerts by visually tracking and monitoring them.
By enacting these nine simple steps, you can decrease the volume, and increase the effectiveness of the alerts you and your team receive. Doing this will ensure that, over time, your monitoring and alerting strategy gets stronger, making useless notifications and alert storms a thing of the past. To learn more about improving your alerting strategy, check out our best practices guide.