Beep, beep, beeeeeeeep. Read path SLO page, again. And I’ve almost found the noisy neighbor! That was me. And will probably be me again at some point in the future. As we continue to scale up the team that builds and runs Grafana Loki at Grafana Labs, I’ve decided to record how I find and diagnose problems in Loki.

With more than 1.5M room nights booked per day, Booking.com requires a solid infrastructure that’s constantly monitored. And indeed, Booking.com now has a footprint of 50,000+ physical servers running across four data centers and six additional points of presence. The sheer size of this server fleet makes it viable for Booking.com to have dedicated teams specializing into looking only at the reliability of those servers.

Content Delivery Network produces numerous log files called CDN logs to deliver video across the internet to our homes and mobile devices. These logs contain crucial information about the CDN servers' performance and video streaming quality. Also, it contains terabytes of data, which has its own set of hurdles in terms of handling it in real-time and performing analytics to understand user experience and network concerns.

It’s virtually impossible to manage today’s complex IT environments at scale without a comprehensive system monitoring solution that allows you to check the health of all your applications and services from a single pane of glass. When your end users are experiencing difficulties, you must have such a tool in place that lets you quickly ascertain and remediate the root cause of the slowdown or error.

You’ve probably seen the term AIOps appear as the subject of an article or talk recently, and there’s a reason. AIOps is merging DevOps principles with Artificial Intelligence, Big Data, and Machine Learning. It provides visibility into performance and system data on a massive scale, automating IT operations through multi-layered platforms while delivering real-time analytics.

In the last few years, many organizations I worked with have significantly increased their cloud footprint. I’ve also seen a large percentage of newly launched companies go with cloud services almost exclusively, limiting their on-premises infrastructure to what cannot be done in the cloud — things like WiFi access points in offices or point of sale (POS) hardware for physical stores.

Thomas Stringer has a couple of great blog posts on how to understand your Azure monitoring costs and also on how to reduce your costs, see Azure Monitor Log Analytics too Expensive? Part 2 – Save Some Money | Thomas Stringer (trstringer.com). In the past I’ve blogged on How to calculate the Azure Monitor and Log Analytics costs associated with AVD (not an easy task!).