Today we are releasing the next public beta of Graylog v3.1. This release brings a whole new alerting and event system that provides more flexible alert conditions and event correlation based on the new search APIs that also power the views. In addition, some extended search capabilities introduced in Graylog Enterprise v3.0 are now available in the open source edition in preparation for unifying the various search features.
Apache Flume helps organizations stream large log files from various sources to distributed data storage like Hadoop HDFS. This article focuses on the features and capabilities of Apache Flume and how it can help applications efficiently process data for reporting and analytical purposes.
Apache Ambari is an open source management tool that helps organizations operate Hadoop clusters at scale. Ambari provides a web UI and REST API to help users configure, spin up, and monitor Hadoop clusters with one centralized platform. As your Hadoop deployment grows in size and complexity, you need deep visibility into your clusters as well as the Ambari servers that manage them. If issues arise in Ambari, it can lead to problems in your data pipelines and cripple your ability to manage clusters.
Monitoring is an essential aspect of any IT system. System metrics such as CPU, RAM, disk usage, and network throughput are the basic building blocks of a monitoring setup. Nowadays, they are often supplemented by higher-level metrics that measure the performance of the application (or microservice) itself as seen by its users (human beings on the internet or other microservices in the same or different clusters).
Five worthy reads is a regular column on five noteworthy items we’ve discovered while researching trending and timeless topics. This week, we define what data sprawl is and how organizations can cope with it effectively. Data sprawl—defined as the proliferation of data into endpoints, servers, applications, BYODs, operating systems, network environments, and even other geo-servers—can be a challenge to monitor and control.
Research firm IDC predicts that overall data volumes will grow from 33 zettabytes in 2018 to 175 zettabytes by 2025. 50% of the data will come from IoT devices and nearly 30% of the data will get consumed in real-time by 2025. The data economy has changed the game on how enterprises will need to work with digital technologies to become more effective, competitive, and innovative.
As many of you know, Gartner is recognized as one of the premier analyst firms by most enterprise IT organizations. Given the broad and diverse set of customers Gartner serves -- many risk averse and conservative towards new technologies, Gartner generally tends to recommend incremental and measured changes. Imagine our surprise when Gartner made this very bold statement "on-premises is the new legacy.
Only about four months to go before nearly 60,000 people descend on Las Vegas for the eighth annual AWS re:Invent user conference. However, that doesn’t mean that Amazon is going to embargo new service announcements until then. The recent AWS Summit in NYC looked more like a mini-re:Invent with nearly 12,000 people in attendance.
In a previous post, I explained the role Apache Kafka plays in production-grade ELK deployments, as a message broker and a transport layer deployed in front of Logstash. As I mentioned in that piece, Redis is another common option. I recently found out that it is even more popular than Kafka! Known for its flexibility, performance and wide language support, Redis is used both as a database and cache but also as a message broker.
