AWS S3 Server Access logs provide detailed records for requests made to S3 buckets. They’re useful for many applications. For example, access log information can be useful in security and access audits. It can also help generate customer insights and better understand your Amazon S3 bill. Coralogix makes it easy to integrate with your S3 server access logs via a Lambda function.

AWS VPC Flow Logs record details about the traffic passing through your application, including requests that were allowed or denied according to your ACL (access control list) rules. It also has information about the IP addresses, and ports for each request, the number of packets, bytes sent, and timestamps for each request. This information brings deep visibility and the ability to improve your security posture over time.

Regular expressions can be crucial for wrangling log data efficiently. You may want to extract specific data from your logs to make it easier to analyze and visualize. Sometimes you might want to capture an email when a particular message is logged. Other times, you may find yourself needing to hide sensitive data in logs before they are saved. And more often than not, you need to match using a RegEx pattern rather than an exact text search. This guide is split into three parts.

CloudTrail logs track actions taken by a user, role, or an AWS service, whether taken through the AWS console or API operations. In contrast to on-premise-infrastructure where something as important as network flow monitoring (Netflow logs) could take weeks or months to get off the ground, AWS has the ability to track flow logs with a few clicks at relatively low cost.

At Coralogix, we strive to ensure that our customers get a stable, real-time service at scale. As part of this commitment, we are constantly improving our data ingestion pipeline resiliency and performance. Coralogix ingests messages at extremely high rates — up to tens of billions of messages per day. Every one of these records needs to go through our entire pipeline at near real-time rates: validation, parsing, classification, and ingestion to Elasticsearch.

Throughout the past few months, I had the opportunity to work with and serve hundreds of Coralogix’s customers, the challenges in performing efficient Log Analytics are numerous, from collecting, searching, visualizing, and alerting. What I have come to learn is that at the heart of each and every one of these challenges laid the challenge of data parsing. JSON structured logs are easier to read, easier to search, alert, and visualize.

Kafka is an open source real-time streaming messaging system and protocol built around the publish-subscribe system. In this system, producers publish data to feeds for which consumers are subscribed to. With Kafka, clients within a system can exchange information with higher performance and lower risk of serious failure. Instead of establishing direct connections between subsystems, clients communicate via a server which brokers the information between producers and consumers.

Creating a topic in production is an operative task that requires awareness and preparation. In this tutorial, we’ll explain all the parameters to consider when creating a new topic in production. Setting the partition count and replication factor is required when creating a new Topic and the following choices affect the performance and reliability of your system.

AWS offers a plethora of log and metric data but in order to extract meaningful insights and react to production issues on a dime, a centralized logging solution is critical. Today we’re excited to announce the availability of Coralogix on the AWS Marketplace for the first time. You can now get our advanced machine learning log insights directly in your AWS cloud account for full-stack log aggregation, convenient billing, tighter security with your AWS infrastructure, and faster deployment.

As Elasticsearch is gradually becoming the standard for textual data indexing (specifically log data) more companies struggle to scale their ELK stack. We decided to pick up the glove and create a series of posts to help you tackle the most common Elasticsearch performance and functional issues. This post will help you in understanding and solving one of the most frustrating Elasticsearch issues – Mapping exceptions.