Coralogix can be configured to automatically and dynamically archive logs to an S3 bucket. This saves Coralogix customers money, but of course there are times when the data needs to be reindexed. This operation counts the reindexed logs against the daily quota. Many times customers would like to search and focus on the exact logs to be reindexed or even query the logs outside of Coralogix all together.

Elastic is “an index”, “a search engine”, “a big data solution”, an analytics platform with advanced data visualizations and incredibly fast search capabilities. In short, it’s a solution for many problems. The Elasticsearch platform provides a distributed search cluster that enables large amounts of data to be indexed and searched at scale.

Syslog takes its name from the System Logging Protocol. It is a standard for message logging and has been in use for decades to send system logs or event messages to a specific server, called a Syslog Server.

In a previous post, we went through a few input plugins like the file input plugin, the TCP/UDP input plugins, etc for collecting data using Logstash. In this post, we will see a few more useful input plugins like the HTTP, HTTP poller, dead letter queue, twitter input plugins, and see how these input plugins work.

The recent changes to the Elasticsearch license could have consequences on your intellectual property. On the 14th of January 2021, Elastic announced through their blog that Elasticsearch and Kibana will be moving over to a Server Side Public License (SSPL). This license change, effective from Elasticsearch version 7.11, has business owners that rely on the ELK stack rightly concerned.

Network security has changed a lot over the years, it had to. From wide open infrastructures to tightly controlled environments, the standard practices of network security have grown more and more sophisticated. This post will take us back in time to look at the journey that a typical network has been on over the past 15+ years. From a wide open, “chewy” network, all the way to zero trust networking. Let’s get started.

In a previous post, we explored the basic concepts behind using Grok patterns with Logstash to parse files. We saw how versatile this combo is and how it can be adapted to process almost anything we want to throw at it. But the first few times you use something, it can be hard to figure out how to configure for your specific use case.

AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you. The code that runs on the AWS Lambda service is called Lambda functions, and the events the functions respond to are called triggers. Lambda functions are very useful for log collection (think of log arrival as a trigger), and Coralogix makes extensive use of them in its AWS integrations.

Syslog is a popular standard for centralizing and formatting log data generated by network devices. It provides a standardized way of generating and collecting log information, such as program errors, notices, warnings, status messages, and so on. Almost all Unix-like operating systems, such as those based on Linux or BSD kernels, use a Syslog daemon that is responsible for collecting log information and storing it.

Platform thinking is a term that has spread throughout the business and technology ecosystem. But what is platform thinking, and how can a platform strategy force multiply the observability capabilities of your team? Platform thinking is an evolution from the traditional pipeline model. In this model, we have the provider/producer at one end and the consumer at the other, with value traveling in one direction.

In days gone by, highly regulated industries like pharmaceuticals and finance were the biggest targets for nefarious cyber actors, due to the financial resources at banks and drug companies’ disposal – their respective security standards were indicative of this. Verizon reports in 2020 that, whilst banks and pharma companies account for 25% of major data breaches, big tech, and supply chain are increasingly at risk.

One of the keys to CloudWatch’s success is its no bang, no buck billing system. The pricing structure has been designed from the outset to ensure that CloudWatch users only pay for what they actually use. In addition, the CloudWatch Free Tier allows first time users to test the waters without shelling out. The downside of this flexibility and adaptability is complexity.

Prometheus metrics are an essential part of your observability stack. Observability comes hand in hand with monitoring, and is covered extensively here in this Essential Observability Techniques article. A well-monitored application with flexible logging frameworks can pay enormous dividends over a long period of sustained growth, but Prometheus has a problem when it comes to scale.