In this lesson, we’ll learn how we can use Elasticsearch Hadoop to process very large amounts of data. For our exercise, we’ll use a simple Apache access log to represent our “big data”. We’ll learn how to write a MapReduce job to ingest the file with Hadoop and index it into Elasticsearch.
Prometheus is the cornerstone of many monitoring solutions, and sooner or later, prometheus federation will appear on your radar. A well monitored application with flexible logging frameworks can pay enormous dividends over a long period of sustained growth. However, once you begin to scale your prometheus stack, it becomes difficult to keep up with your application’s demands. Prometheus is an extremely popular choice when it comes down to collecting and querying real-time metrics.
Modern day Security Information and Event Management (SIEM) tooling enterprise security technology combine systems together for a comprehensive view of IT security. This can be tricky, so we’ve put together a simple SIEM tutorial to help you understand what a great SIEM provider will do for you. A SIEM’s responsibility is to collect, store, analyze, investigate and report on log and other data for incident response, forensics and regulatory compliance purposes.
Surveys show that developers spend roughly 25% of their time troubleshooting issues, amounting to over one working day per week! Let’s examine the solutions that will guide your developers to the log insights they need to efficiently troubleshoot issues.
The latest Elasticsearch release version was made available on September 24, 2020 and contains several bug fixes and new features from the previous minor version released this past August. This article highlights some of the crucial bug fixes and enhancements made, discusses issues common to upgrading to this new minor version and introduces some of the new features released with 7.9 and its subsequent patches. A complete list of release notes can be found on the elastic website.
An Elastic Security Advisory (ESA) is a notice from Elastic to its users of a new Elasticsearch vulnerability. The vendor assigns both a CVE and an ESA identifier to each advisory along with a summary and remediation details. When Elastic receives an issue, they evaluate it and, if the vendor decides it is a vulnerability, work to fix it before releasing a remediation in a timeframe that matches the severity.
Your log analysis solution works through millions of lines of logs, which makes implementing a machine learning solution essential. Organizations are turning to machine learning log alerts as a replacement or enhancement of their traditional threshold alerts. As service uptime becomes a key differentiator, threshold alerts are only as good as your ability to foresee an issue.
The first time I created a cloud compute instance, then still called a “Cloud VM”, was an almost transcendent moment. It was like magic. I was at my first organization which had adopted the cloud, in my first DevOps position, and I immediately knew that the world had changed.
Before we dive into the gap in cloud network security, let’s take a step back. If you’ve been in Operations for a while, you might remember how it used to be. “Network” was a team. When you needed to open a port on the network, you had to provide an exhaustive definition of the change, explaining what port you needed, what external addresses should be able to reach it, and where it should be routed to internally.
