Earlier this year, we upgraded from Confluent Platform 7.0.10 to 7.6.0. While the upgrade went smoothly, there was one thing that was different from previous upgrades: due to changes in the metadata format for Confluent’s Tiered Storage feature, all of our tiered storage metadata files had to be converted to a newer format.

Implementing a consistent and reliable alerting system across a sprawling organization is a significant challenge for just about any engineering team. For example, diverse infrastructures across different teams and numerous team-specific customizations may not translate well when investigating specific incidents. Inconsistent alerting practices can eventually lead to fatigue, leading to triggering of alerts that may not be relevant or actionable.

Observability is a broad topic that provides visibility into the key metrics powering customer-facing applications. These applications range from external facing applications ( e.g., Internet banking/online education/e-commerce/government records ) to internal facing applications ( e.g., Trading systems by brokers, Logistics controllers, Traffic Management, and Hotel Reservations). Observability also incorporates backend systems powering industries that ensure smooth operations of tools and processes.

Over the last few decades, software systems have grown complex due to the emergence of cloud-native architectures and multi-cloud environments. On the one hand, this makes it difficult to detect issues faster in the deployed application. It also requires intricate coordination between development, DevOps, and SRE teams, as they are also expected to speed up the whole software delivery process.

Imagine one night you receive a notification from your team member that a critical production problem has caused chaos in your application. There is a sudden drop in sales as customers are unable to access the application and reporting issues relating to the same. Now, when you reach the office to fix the issue, you demand the team to run through all the files.

Modern digital organizations have rapidly adopted microservices-based architecture for their applications. Microservices-based apps have components designed around business capabilities serving a specific purpose. It enables smaller engineering teams to own specific services that lead to increased productivity. But componentization also leads to complexity. Today’s modern internet-scale businesses have hundreds or thousands of microservices.