Ever since humankind developed the ability to write, much of our progress has been made thanks to recording and using data. In ages long past, notes were made on the production and gathering of resources, the exact number of available soldiers and other important personnel, and were compiled and stored by hand. Because of this documentation method, important information was also prone to being misplaced, lost, or even mishandled.

Servers take a lot of requests daily, we know that…We also know that the server responds instantly. But who makes the request? What do they want, and what exactly are they looking for? Where do these visitors come from? How often they are making a request: once a month, once a day, almost every minute? Well, answers to these, and potentially a lot more questions, can be found in a single place - the server log file.

Companies are generating massive amounts of data every minute. It’s impossible, unrealistic, and cost-prohibitive for analysts to spot every threat. That’s why even though breaches are in decline year over year, the first quarter of 2018 saw 686 breaches that exposed 1.4 billion records through hacking, skimming, inadvertent Internet disclosure, phishing, and malware.

Today we are releasing Graylog v2.4.6 to fix a few bugs.

Juraj Kosik, an Infrastructure Security Technical Lead at Deutsche Telekom Pan-Net, has written a detailed case study of how his organization implemented Graylog to centralize log data from multiple data centers exceeding 1 TB/day. His case study provides thorough insights into real-world issues you might run into when implementing and operating a log management platform in a large-scale cloud environment.

Centralized log management lets you decide who can access log data without actually having access to the servers. You can also correlate data from different sources, such as the operating system, your applications, and the firewall. Another benefit is that user do not need to log in to hundreds of devices to find out what is happening. You can also use data normalization and enhancement rules to create value for people who might not be familiar with a specific log type.

Getting the right information at the right time can be a difficult task in large corporate IT infrastructures. Whether you are dealing with a security issue or an operational outage, the right data is key to prevent further breakdowns. With central log management, security analysts or IT operators have a single place to access server log data. But what happens if the one log file that is urgently needed is not collected by the system?

Welcome to part two of a three-part series on trend analysis of log event data. Today, we will explore how to perform, using Graylog, a few of the types of trend analysis discussed previously.

Centralized log collection has become a necessity for many organizations. Much of the data we need to run our operations and secure our environments comes from the logs generated by our devices and applications. Centralizing these logs creates a large repository of data that we can query to enable various types of analysis. The most common types are conditional analysis and trend analysis. They both have their place, but trend analysis is perhaps the more often underutilized source of information.

Central storage is vitally important in log management. Just as storing and processing logs into lumber is done in one place, a sawmill, a central repository makes it cheaper and more efficient to process event logs in one location. Moving between multiple locations to process logs can decrease performance. To continue the analogy, once boards are cut at a sawmill, a tool such as a wood jointer smoothes out the rough edges of the boards and readies them for use in making beautiful things.