We founded Logz.io four years ago with a vision to provide a platform for engineers like us to operate the next generation of applications and infrastructure. As engineers, we experienced firsthand the challenges of running large-scale, open-source tools and the frustration of dealing with slow, proprietary tools which were not designed to support the rapidly changing technology stack we operated.

Here are a few updates you can look forward to in LogDNA’s log management and analysis features.

Over the years, automation has become a key component in the management of the entire software release lifecycle. Automation helps teams get code from development into the hands of users faster and more reliably. While this principle is critical to your source code and continuous integration and delivery processes, it is equally essential to the underlying infrastructure you depend on.

Containers provide a nifty solution to package up applications along with their dependencies, and for the whole encapsulated process to be run on a host system. This technology is undeniably popular due to its ability to allow developers to create flexible, scalable, reliable solutions in a quicker amount of time. It has enabled more freedom in choosing the technology we use in our applications and has brought development and production environments closer to parity.

Sifting through all your logs to find what you need can be challenging—especially during an outage, when time is critical and you’re flooded with WARN and ERROR messages. To help you immediately surface useful information from large volumes of logs, we developed Log Patterns.

This recipe is similar to the previous rsyslog + Redis + Logstash one, except that we’ll use Kafka as a central buffer and connecting point instead of Redis. You’ll have more of the same advantages.

Starting from Elasticsearch 5.0, you’re able to define pipelines within it that process your data, in the same way you’d normally do it with something like Logstash. We decided to take it for a spin and see how this new functionality (called Ingest) compares with Logstash filters in both performance and functionality. Is it worth sending data directly to Elasticsearch or should we keep Logstash?

Since its first release in 2007, Splunk quickly became one of the leading log management solutions. Its focus on enterprise grade log analysis and security incident and event management (SIEM) made it the de facto choice for organizations generating large volumes of log files and machine data. But over the past decade, the log management landscape has changed drastically.

Most software and systems generate audit logs. They are a means to examine what activities have occurred on the system and are typically used for diagnostic performance and error correction. System Administrators, network engineers, developers, and help desk personnel all use this data to aid them in their jobs and maintain system stability. Audit logs have also taken on new importance for cybersecurity and are often the basis of forensic analysis, security analysis, and criminal prosecution.

Here at Sematext we use Java and rely on Logsene, our hosted ELK logging SaaS, a lot. We like them so much that we regularly share our logging experience with everyone and help others with logging, especially, ELK stack. Centralized logging plays nice with Java (and anything else that can write pretty logs). However, there is one tricky thing that can be hard to get right: properly capturing exception stack traces.