Operations | Monitoring | ITSM | DevOps | Cloud

This blog highlights the advantages of having an internal status page for your business.

Postmortem reports are essential in incident management, helping teams learn from past mistakes and prevent future issues. Traditionally, creating these reports was a slow, tedious process, requiring teams to gather data from multiple sources and piece together what happened. But with AI and Large Language Models (LLMs), this process can become faster, smarter, and much less of a headache.

Today, we’re excited to unveil the Anomaly Detection feature. It will enable users to create smarter alerts based on dynamic metrics, moving beyond traditional fixed-threshold alerts. It will soon be available to all our users and is currently undergoing beta testing with select users. By detecting deviations from expected patterns, Anomaly Detection will help you stay informed about critical issues without getting overwhelmed by irrelevant alerts. Let’s dig in deeper.

In an ideal world, every alert would signal a unique and critical issue. However, in reality, alerts often come in waves. Alert noise refers to the overwhelming volume of notifications that incident response teams receive, many of which may be redundant or irrelevant. This can lead to alert fatigue, where critical issues might be overlooked due to the sheer number of notifications. ‍

A recent question in an SRE forum triggered this train of thought. I've paraphrased the question to reflect its essence. There is plenty to unravel here. My first reaction to this question was that the SRE who posted this is in a difficult place with systemic issues.

Following the Icinga Notifications beta announcement, we already had a more general post on how to get started and one going into the details of schedules. This week’s blog post is a follow up in this series and will describe incidents, escalations, and event rules in Icinga Notifications in more detail. In case you haven’t seen the first two referenced blog posts, you might want to have a look at them first, otherwise, you could miss out on the big picture.

AlertOps announced its new integration with ServiceNow to enhance incident management and response capabilities for ServiceNow customers. This joint effort enables AlertOps to create better experiences and drive value for customers by providing real-time notifications, bi-directional data synchronization, and seamless integrations. ServiceNow’s expansive partner ecosystem and partner program is critical in supporting the Now Platform’s $275 billion forecasted market opportunity through 2026.

A year ago we rebuilt our alert rule state history, using Grafana Loki for storage and updating the UI to display a timeline of all state changes of an alert rule. As a result, users can now conduct better root cause analysis by going down to the level of an alert rule and seeing when certain alert instances started or stopped firing. But we aren’t stopping there. To ensure system stability and avert outages, you also need one place to see the state history for all the alerts in your system.