Operations | Monitoring | ITSM | DevOps | Cloud

November 2021

Amazon S3: Lake is the New Bus

This is a short blog post about a pattern that we’ve observed more frequently among some of the large enterprises: the use of AWS S3 as both an observability lake and a data bus. AWS S3’s simple API, ubiquitous language support, unmatched reliability and durability, retention options, and numerous pricing plans have made it the de facto standard for storing massive amounts of data.

Monitor Salesforce's Real-Time Events with Splunk

In 2019 Salesforce announced the general availability of Real-Time Event Monitoring (RTEM) which includes 19 different events that help monitor & secure your Salesforce data. Real-Time Event Monitoring stores events for 6 months as Salesforce Big Objects and streams events via Salesforce’s Streaming API in near real-time.

User experience is a focus of Sumo Logic Observability innovations

Technology environments are rapidly evolving as organizations look to remain competitive, accelerate innovation and make themselves more agile. But in the process, many of the observers, i.e., stakeholders who track infrastructure and application metrics, are falling behind, unable to monitor and manage modern, cloud-native apps and multi-cloud environments due to the complexity that comes with them.

The Stream Life Episode 25: SaaS Security Platforms

In this episode of the Stream Life Podcast, Nick Heudecker and Ed Bailey look at SaaS security platforms and managed security providers and how they’ve grown over time. They look at the benefits the model brings to organizations, how it is growing across the world, the challenges it can also bring, and the questions you should be asking your vendors.

GCP Integrations for Metrics with Logz.io

Logz.io has dedicated itself to encouraging and supporting cloud-native development. That has meant doubling down on support for AWS and Azure, but also increasing our tie-ins with Google Cloud Platform – GCP. Recently, our team added dozens of new integrations for metrics covering the gamut of products in the GCP ecosystem.

Creating a Free Data Lake with Coralogix

Like many cool tools out there, this project started from a request made by a customer of ours. Having recently migrated to our service, this customer had ~30TB of historical logging data. This is a considerable amount of operational data to leave behind when moving from one SaaS platform to another. Unfortunately, most observability solutions are built around the working assumption that data flows are future-facing.

5 Things Developers Need to Know About Kubernetes Management

Kubernetes management can be daunting for developers who don’t have specialized understanding of the orchestration technology. Learning Kubernetes takes practice and time, a precious commodity for devs who are under pressure to deliver new applications. This post provides direction on what you need to know and what you can skip to take advantage of Kubernetes. Let’s start with five things you need to know.

We're Thrilled To Share - Coralogix has Received AWS DevOps Competency

At Coralogix, we believe in giving companies the best of the best – that’s what we strive for with everything we do. With that, we are happy to share that Coralogix has received AWS DevOps Competency! Coralogix started working with AWS in 2017, and our partnership has grown immensely in the years since. So, what is our new AWS DevOps Competency status, and what does it mean for you?

LogStream Cloud How To: Sending Data to LogStream from Various Agents

Cribl released LogStream Cloud to the world in the Spring of 2021, making it easier than ever to stand up a functional o11y pipeline. The service is free for up to 1TB per day and can be upgraded to unlock all the features and support with paid plans starting at $0.17 per GB so you pay for only exactly what you use. In this blog post, we’ll go over how to quickly get data flowing into LogStream Cloud from a few common log sources.

Splunking Your *.conf Files: How to Audit Configuration Changes Like a Boss

For years customers have leveraged the power of Splunk configuration files to customize their environments with flexibility and precision. And for years, we’ve enabled admins to customize things like system settings, deployment configurations, knowledge objects and saved searches to their hearts’ content. Unfortunately a side effect of this was that multiple team members could change underlying.conf files and forget that those changes ever occurred.

DevOps State of Mind Podcast Episode 2: Giving People the Power to Participate

‍Sean Tierney is the DevOps lead at Athos, a company that's building better athletes through smart clothing and AI. Sean reinforces a DevOps state of mind across the organization by building empathy between hardware and software teams and putting the systems in place to allow them to move faster as a single unit.

How to Make Splunk Run 100x Faster With Cribl LogStream

Enterprises leveraging Splunk for data ingestion and analytics need an observability solution that scales well with their business requirements and provides a cost-effective way to retain data long-term. Cribl LogStream is an essential part of observability, providing a pipeline that works with all tooling, keeps costs down, and scales with any business – making it the perfect complement to Splunk.

Embracing invokedynamic to tame class loaders in Java agents

One of the nicest things about Byte Buddy is that it allows you to write a Java agent without manually having to deal with byte code. To instrument a method, agent authors can simply write the code they want to inject in pure Java. This makes writing Java agents much more accessible and avoids complicated on-boarding requirements.

How to streamline Windows monitoring for better security

If you’re responsible for a significant number of Windows servers, you already understand the importance of being aware of the health and security of your environment. Unfortunately, you’re probably also aware of the tremendous amount of effort and resources required to monitor your Windows environment. Let’s take a look into why and how you should be closely monitoring your Windows server environments from a security perspective.

Understanding Apache Logging: How to View, Locate and Analyze Access & Error Logs

Apache – the technology that powers the web. I’m not sure if that is correct, but I think that we wouldn’t see the world wide web in its current shape without it. Launched in 1995 and since April 1996, it has been the most popular web server around the world. Because of handling your users’ requests, Apache serves as the front application. It is crucial to understand what your server is doing, what file users are accessing, from where they came, and much, much more.

Simple, scalable deployment for Grafana Loki and Grafana Enterprise Logs

Loki 2.4 and GEL 1.2 introduced a hybrid deployment model that takes the simplicity of running the Loki log aggregation solution as a single binary and introduces an easy path to high availability and scalability. Particularly for organizations running on virtual machine and bare metal (non-Kubernetes) environments, this is a game-changer! Learn more in this tutorial from Grafana Labs Senior Software Engineer Trevor Whitney.

Istio Log Analysis Guide

Istio has quickly become a cornerstone of most Kubernetes clusters. As your container orchestration platform scales, Istio embeds functionality into the fabric of your cluster that makes monitoring, observability, and flexibility much more straightforward. However, it leaves us with our next question – how do we monitor Istio? This Istio log analysis guide will help you get to the bottom of what your Istio platform is doing.

Extend your DevOps analysis to CircleCI and GitLab data

Every company is a software company and every company wants to get better at it. That’s the reason we built Software Development Optimization or SDO. SDO helps you track siloed data across the DevOps toolchain. It normalizes and correlates data, provides you with DORA’s 4 key metrics and gives you deep insights into the velocity and quality of delivery across services and teams.

Managing the Mess of Modern IT: Log Analytics and Operations Engineering

IT is messy stuff. Enterprise applications and devices rely on a web of interdependent clouds, networks, and containers. IT operations (ITOps), development operations (DevOps), and cloud operations (CloudOps) engineers work hard to manage this mess. If they succeed, they create a stable, agile IT environment that makes their enterprise more productive. If they fail, their enterprise becomes less productive.

Log Management Guide: Why You Should Track Logs?

IT experts agree that log management and monitoring is one of the most effective ways to keep IT infrastructure performing optimally. Logs play a vital role in improving performance, enhancing security, and detecting issues. But at the same time, a lot of people don’t use logs to the best of their ability. This guide will not only introduce you to log management but also reveal which logs to track and what information they are giving to you.

Introducing Logz.io Event Management: Accelerating Collaborative Threat Response

In the domain of cyber threat response, there’s a critical resource that every organization is desperately seeking to maximize: time. It’s not like today’s DevOps teams aren’t already ruthlessly focused on optimizing their work to unlock the greater potential of their human talent. Ensuring your organization to identify and address production issues faster – and increase focus on innovation – is the primary reason why Logz.io and its observability platform exist.

Video: The new simple, scalable deployment for Grafana Loki and Grafana Enterprise Logs

With the recent release of Loki 2.4 and Grafana Enterprise Logs 1.2, we’re excited to introduce a new deployment architecture. Previously, if you wanted to scale a Loki installation, your options were: 1) run multiple instances of a single binary (not recommended!), or 2) run Loki as microservices. The first option was easy, but it led to brittle environments where a heavy query load could take down data ingestion and problems were often difficult to debug.

Logz.io Anomaly Detection: Shedding Light on "Unknown Unknowns"

Moving beyond traditional monitoring to embrace full stack observability offers a seemingly endless range of benefits. Beyond unifying logs, metrics, and traces in a single platform, the opportunity to enlist advanced analytics and engage a more predictive approach represents another huge step forward.

Papertrail and Heroku

SolarWinds® Papertrail™ has supported Heroku almost from the beginning, as an add-on in the Heroku Marketplace and as a compatible standalone log management tool. Heroku’s focus on empowering developers to build and deliver applications by providing an easy-to-use platform as a service fits perfectly with the vision of Papertrail. Both developer-focused technologies can be set up typically in minutes, are easy to use, and offer powerful functions.

Get Started with Connected Experiences

November, the season of post-conf, is upon us. Hopefully all you Splunk admins and sc_admins are craving the release of a ton of new.conf21 Splunk features. Well, good news, because Connected Experiences is here to help you get started with everything Splunk Mobile, Augmented Reality, TV and iPad with this one handy guide. Let’s get started!

Splunk Developer Fall 2021 Update

While it’s cooling down here in California as Fall arrives, we have some really hot and exciting updates from.conf21, including the announcement of Splunk Cloud Developer Edition, the new Splunkbase user experience, detailed guidelines to help you deliver cloud-ready apps for Splunk Cloud Platform, AppInspect updates with new checks, a helpful blog about storing app secrets, updated docs for Modular Inputs and External Lookups, a summary of SDK updates, and more.

Understanding business and security risk

Even if an organization has developed a governance team, aligning integration decisions with business needs must be incorporated into the zero trust architecture. The company’s business model drives the applications chosen. The senior leadership team needs someone who can translate technology risks and apply them to business risks. For example, security might be an organization’s differentiator.

Announcing Logz.io Unified Dashboards

In today’s cloud environments, a typical observability stack might include an Elasticsearch cluster for logging, a few Prometheus servers for metrics monitoring, and an AppDynamics deployment for APM. You may run something similar – most observability stacks consist of multiple siloed tools dedicated to collecting and analyzing specific types of monitoring data.

ScaleUP 2021: Taking the Logz.io Observability Platform to the Next Level

Today was a very exciting day for Logz.io, as we held ScaleUP 2021 – our second annual user conference – dedicated to elevating our customers’ success, discussing best practices for modern observability, and unveiling Logz.io’s latest product updates. These product advancements were presented by our Co-Founder and VP of Product Asaf Yigal, and members of the Logz.io software engineering team.

DevOps State of Mind Podcast Episode 1: Trust, tooling, and a no-blame culture with LogDNA

Tucker Callaway is the CEO of LogDNA. He has more than 20 years of experience in enterprise software with an emphasis on developer and DevOps tools. Tucker fosters a DevOps culture at LogDNA by tying technical projects to business outcomes, practicing extreme transparency, and empowering every person in the company to contribute.

Logz.io Moves to Embrace OpenSearch at the Core of its Platform

As Logz.io prepares to hold its annual ScaleUP user conference tomorrow, celebrating another amazing year of customer success and continued advancement of our observability platform, we’ve got exciting news to share about our involvement with the OpenSearch project.

How to set up Stanza as the log agent for your GCP?

Stanza is a robust log agent. GCP users can use Stanza for ingesting large volumes of log data. Before we dive into the configuration steps, here’s a matrix detailing the functional differences between all the common log agents used by GCP users. Stanza was built as a modernized version of FluentD, Fluentbit, and Logstash. GCP users now have the ability to install Stanza to their VMs/ GKE clusters to ingest logs and route them to GCP log explorer.

Help your business, learn everything about the log and event correlator

Pandora FMs alerting system allows building alerts based not only on the value received by modules,but also in received events or in the information collected by the log collector. We will see how it is possible to build simple alerts or more complex ones, based on a set of logic rules about said events or logs. We present the correlation alert system.

Logs and tracing: not just for production, local development too

We're a small team of engineers right now, but each engineer has experience working at companies who invested heavily in observability. While we can't afford months of time dedicated to our tooling, we want to come as close as possible to what we know is good, while running as little as we can- ideally buying, not building. Even with these constraints, we've been surprised at just how good we've managed to get our setup.

Why and how to monitor AWS EKS

Amazon Elastic Kubernetes Service, or EKS, is a managed Kubernetes service. That means that Amazon Web Services (AWS) handles some of the deployment and management tasks for users. But the fact that EKS is a managed service doesn’t mean that AWS manages all administrative tasks. One key management task that isn’t fully covered as part of EKS is monitoring.

Stay Alert! Building the Coralogix-Nagios Connector

Ask any DevOps engineer, and they will tell you about all the alerts they enable so they can stay informed about their code. These alerts are the first line of defense in the fight for Perfect Uptime SLA. With every good solution out there, you can find plenty of methods for alerting and monitoring events in the code. Each method has its own reasons and logic for how it works and why it’s the best option. But what can you do when you need to connect two opposing methodologies? You innovate!

Loki 2.4 is easier to run with a new simplified deployment model

Loki 2.4 is here! It comes with a very long list of cool new features, but there are a couple things I really want to focus on here. Be sure to check out the full release notes and of course the upgrade guide to get all the latest info about upgrading Loki. Also check out our ObservabilityCON 2021 session Why Loki is easier to use and operate than ever before.

Enabling SRE best practices: new contextual traces in Cloud Logging

The need for relevant and contextual telemetry data to support online services has grown in the last decade as businesses undergo digital transformation. These data are typically the difference between proactively remediating application performance issues or costly service downtime. Distributed tracing is a key capability for improving application performance and reliability, as noted in SRE best practices.

Limit Coralogix usage per account using Azure Functions

At Payoneer, we use Coralogix to collect logs from all our environments from QA to PROD. Each environment has its own account in Coralogix and thus its own limit. Coralogix price modules are calculated per account. We as a company have our budget per account and we know how much we pay per each one. In case you exceed the number of logs assigned per account you will pay for the “extra” logs. You can see the exact calculation in this link.

What's Wrong With Observability Pricing?

There’s something wrong with the pricing of observability services. Not just because it costs a lot – it certainly does – but also because it’s almost impossible to discern, in many cases, exactly how the costs are calculated. The service itself, the number of users, the number of sources, the analytics, the retention period, and extended data retention, and the engineers on staff who maintain the whole system are all relevant factors that feed into the final expense.

How Sumo Logic monitors unit economics to improve cloud cost-efficiency

An often overlooked aspect of a company’s journey to the cloud is cost visibility. While the single number delivered by the cloud provider on a monthly invoice is straightforward, understanding where this number comes from is often more tricky. Fortunately, this task can be facilitated through the usage of various cost monitoring tools available on the market, coming from both third-party companies and the cloud providers themselves.

Introducing Log Data Restoration on LogDNA

If you’re reading this, I’m pretty sure I don’t need to do much to convince you of the importance of logs. They are the core atomic unit for understanding your environments and provide the insights required to troubleshoot, debug, and more. The fact of the matter is that everyone in your organization needs logs to perform critical functions of their job.

3 Ways Ops Teams Benefit From LM Logs

Sifting through logs in real-time or post-mortem to pinpoint the problem can take hours – and is often like trying to find the needle in the alert/log haystack. Further, keeping the troubleshooting process efficient can be a challenge due to context switching and relying on manual interpretation of events and technology-specific knowledge.

Updated ELK Stack Guide For 2022 (Installation, Tutorials & More)

The ELK Stack has millions of users globally due to its effectiveness for log management, SIEM, alerting, data analytics, e-commerce site search and data visualisation. In this extensive guide (updated for 2021) we cover all of the essential basics you need to know to get started with installing ELK, exploring its most popular use cases and the leading integrations you’ll want to start ingesting your logs and metrics data from.

NGINX Monitoring: Best Tools and Key Metrics You Should Know About

NGINX is a popular web server featuring a wide range of capabilities, including reverse proxy, mail proxy, HTTP cache, and load balancing. It offers TLS offloading and a health check of the backends and supports gRPC, WebSocket, and HTTP/2. In short, NGINX is a one-stop solution for most of your web server needs. When using NGINX, monitoring its metrics is crucial for tackling issues.

Who needs CMMC certification (Resource Guide for 2022)

If your company works with the US Department of Defense (DoD) as a contractor or subcontractor, you will need to prepare to meet CMMC requirements in order to successfully bid on and win contracts. This recent development has been a significant adjustment for small organisations who wish to work with or continue working with the DoD.

A Study in Graylog's Versatility

Recently, I explored the case for Graylog as an outstanding means of aggregating the specialized training data needed to build a successful, customized artificial intelligence (AI) project. Well, that’s true, of course. My larger point, though, was that Graylog is a powerful and flexible solution applicable to a very broad range of use cases (of which AI development is just one).

Best practices for Cloud Operations in the enterprise

How can you get the most value out of Cloud Operations, especially as your Cloud footprint grows? In this episode of Engineering for Reliability, we look at the enterprise best practices for setting up and using Cloud Operations. Watch to learn how to improve the security of your services, better manage capacity, and keep your users happy!

Introducing Log Observability for Microservices

Two popular deployment architectures exist in software: the out-of-favor monolithic architecture and the newly popular microservices architecture. Monolithic architectures were quite popular in the past, with almost all companies adopting them. As time went on, the drawbacks of these systems drove companies to rework entire systems to use microservices instead.

Perfect Your Cloud's Deployment with Logz.io & AWS CloudFormation Public Registry

AWS CloudFormation is a service that enables you to create and provision AWS infrastructure deployments predictably and repeatedly. This helps you leverage AWS products such as EC2 instances, Amazon Elastic Block Store, Amazon SNS, Elastic Load Balancing, and Auto Scaling to build highly reliable, highly scalable, cost-effective applications in the cloud – without worrying about creating and configuring the underlying AWS infrastructure.

New Integrations from Logz.io: November 2021

It’s been a busy couple of months at Logz.io. We’ve added new features, made critical updates, and added a slew of integrations. Those integrations run the gamut from observability and security services, to cloud tools and container orchestration. Let’s take a quick look at what’s new and what’s coming up at Logz.io.

Reducing the Carbon Footprint of the Manufacturing Industry with the Power of Data

As the world watches the UN Climate Conference unfold in Glasgow, the manufacturing industry is under increased pressure to reduce its CO2 emissions. Companies are stepping up to the challenge, but much more remains to be done to keep climate change under control. Digitalization is part of the solution. The Power of Data has much to offer such as environmentally friendly manufacturing, efficient logistics, improved building and energy management and optimized renewable energy production.