Operations | Monitoring | ITSM | DevOps | Cloud

When it comes to keeping your business’s lights on, you need to manage and orchestrate your operational activities, prioritize high-impact and urgent work, and maintain day-to-day precision. Trust is paramount during mission-critical, time-sensitive crisis response and the narrow margin for error means there is little room and low acceptance for generative AI hallucinations or false positives.

The first moments of incident response can be among the most crucial, which in turn can also make them among the most stressful. There are many ways to ensure incidents are kicked off smoothly, but a recent focus of ours was to ensure they could be kicked off quickly. After all, the faster you're able to start mitigating your incident, the more successful you'll be!

When our CEO and co-founder Tomer Levy delivered his “Observability is Broken” presentation at last year’s AWS re:Invent, he highlighted numerous challenges faced by today’s organizations as they seek to advance their observability practices. Of the six individual points that he noted, two specifically dealt with the current shortage of available engineering expertise, with another two focused on data overload.

Although the causes and solutions for incidents vary widely, most incidents follow a similar timeline from declaration to resolution. We call the period of time it takes to move from one phase or milestone of an incident to the next cycle time.

We understand that everyone values a simple and straightforward approach when it comes to setting up schedules. We at Squadcast are fully aware of the difficulties involved in creating an on-call schedule from scratch or migrating it to a new platform. Hence we have come up with a blog to assist you in seamlessly setting up your on-call schedule using Squadcast. Our goal is to provide guidance and support to make the process as effortless as possible for you.

Prometheus is a favored open-source monitoring system that collects, stores, and queries metrics from various sources. In Prometheus, an exporter is a component that collects and exposes metrics in a format Prometheus can scrape. The Prometheus Blackbox Exporter is designed to monitor “black box” systems with internal workings that are not accessible by Prometheus. It sends HTTP, TCP, and ICMP requests to the external systems and measures their response times and statuses.

Before we dive into the nitty-gritty of incident management, let’s look a bit closer at the actual meaning of ‘incident.’ In the world of IT service management, the official definition for ‘incident’ is an “unplanned interruption to an IT service or reduction in the quality of an IT service.” Whether that means a slowdown in response time or a total system crash, you’re looking at an incident.

This week I’ve been reading through the recent judgment from the Swedish FSA on the Swedbank outage. If you’re unfamiliar with this story, Swedbank had a major outage in April 2022 that was caused by an unapproved change to their IT systems. It temporarily left nearly a million customers with incorrect balances, many of whom were unable to meet payments.