Welcome to Observability 101, the comprehensive guide from Observo.ai, unveiling the mysteries of observability in 38 insightful chapters. Have you ever wanted to know something specific about observability but were afraid to ask? Fear not, this series is designed to be your go-to resource, answering all of the questions you might have.

A large, global Data Management and AI software company with over 5,000 customers across more than 100 countries had seen unprecedented growth (more than 30% year over year) in telemetry data from their multi-cloud infrastructure being sent to the Elasticsearch Observability and SIEM Platform. The growth of this data contributed to a multi-million dollar price tag for Elasticsearch.

A Global 1000 E-commerce company struggled with the rapid growth in telemetry data that their security team analyzes with Splunk, Grafana, and other Observability tools in the cloud. Specifically, the increase in VPC Flow log and Firewall log volumes caused a spike in Splunk costs on certain data sets and triggered daily indexing limit overage fees. As this deluge of data began piling up in block storage within their Splunk index, the team saw corresponding spikes in storage costs.

Amazon Web Services (AWS) VPC Flow Logs is a feature designed to capture and provide information about the IP traffic that flows to and from network interfaces within your Virtual Private Cloud (VPC). This data can be published to various destinations, including AWS CloudWatch Logs, AWS S3, or AWS Kinesis Data Firehose. Flow logs serve several important purposes, such as diagnosing security group rule issues, monitoring incoming and outgoing traffic, and determining traffic directions.

A firewall is a network security device or software that is used to monitor and control incoming and outgoing network traffic based on predetermined security rules. Firewall Logs contain valuable information about network and security events. These logs are essential for security and infrastructure monitoring for enterprises. While this data is critical to securing enterprise networks, they are also one of the most voluminous data types security teams use to monitor and secure their networks.

In today's digital landscape, Observability and telemetry data play a crucial role in ensuring the performance, reliability, and security of modern applications and services. However, as data volumes explode due to the proliferation of micro-services, cloud-based applications, and connected devices, existing architectures are hitting a scalability wall.

Log management costs are growing, and it's a concern for companies, users, and developers trying to scale their organizations in today’s macro environment. Companies are making investments in systems that collect data from the cloud, applications, and infrastructure in order to monitor their performance and security. The amount of machine data generated every day is skyrocketing as businesses digitize and automate operations.

As cloud deployments scale, Amazon Web Services (AWS) VPC flow logs become an invaluable network visibility and security tool. They are also one of the most voluminous classes of data, making them an expensive choice to add to analytics platforms. With growing infrastructure and traffic, managing these logs presents significant challenges. ‍In part 1 of this series, we took a look at common use cases and problems associated with storing and processing VPC Flow Logs.