Spring has finally sprung! I realize that was a groaner, but new features are no joke – using them will simplify your troubleshooting process and open up time for all the cool tasks you want to do. Besides parsing logs, of course.
The ELK Stack (Elasticsearch, Logstash and Kibana) is the weapon of choice for many Kubernetes users looking for an easy and effective way to gain insight into their clusters, pods and containers. The “L” in “ELK” has gradually changed to an “F” reflecting the preference to use Fluentd instead of Logstash and making the “EFK Stack” a more accurate acronym for what has become the de-facto standard for Kubernetes-native logging.
Amazon EC2 is the cornerstone for any Amazon-based cloud deployment. Enabling you to provision and scale compute resources with different memory, CPU, networking and storage capacity in multiple regions all around the world, EC2 is by far Amazon’s most popular and widely used service.
In a previous post we looked at 6 key considerations to keep in mind when selecting a log management solution: data collection, search experience, scalability, security, advanced analytics and cost effectiveness. Hopefully, you’ve managed to use this list to finally select your solution. What now?
An important element of operating Kubernetes is monitoring. Hosted Kubernetes services simplify the deployment and management of clusters, but the task of setting up logging and monitoring is mostly up to us. Yes, Kubernetes offer built-in monitoring plumbing, making it easier to ship logs to either Stackdriver or the ELK Stack, but these two endpoints, as well as the data pipeline itself, still need to be set up and configured.
The threat landscape has become increasingly diverse and the systems used to attack are more sophisticated than ever before. In 2018, enterprises and organizations of all sizes and across all industries faced serious data breaches (information leaks were experienced by Aadhar —1.1B users; myFitnessPal —150M; Quora—100M; Facebook—29M and many more). One of the biggest victims was Marriot.
Logstash plays an extremely important role in any ELK-based data pipeline but is still considered as one of the main pain points in the stack. Like any piece of software, Logstash has a lot of nooks and crannies that need to be mastered to be able to log with confidence. One super-important nook and cranny is the Logstash configuration file (not the software’s configuration file (/etc/logstash/logstash.yml), but the .conf file responsible for your data pipeline).
Famed management thinker Peter Drucker is often quoted as saying, “You can’t manage what you can’t measure.” Tracking and analyzing data of a system provides metrics to measure, predict, and improve the underlining health of the system. Logging data is the simplest act of collecting data for measurement and plays an important role in modern enterprises, as it provides a way to measure the health of hardware devices and software applications alike.
If you could know information about your systems in advance, what would you choose to know? If there was a set of repeating behaviors that happened consistently before an outage, would you want to know what they were? This is the idea behind proactive monitoring – the switching of context from “reactive” monitoring to something that allows you to act before the problem arises. Here are some guidelines to help you get started with your customized solution.
Java programs running on JVM create objects on the heap. At some stage, these objects are no longer used and can pile up as “garbage” needlessly taking up memory. Replacing the manual process of explicitly allocating and freeing memory, the Java Garbage Collection process was designed to take care of this problem automatically.
