Operations | Monitoring | ITSM | DevOps | Cloud

May 2022

What is OpenTelemetry

You may have previously heard about OpenTelemetry (also known as OTel) if you have looked into improved ways of standardising different data types. In this article, we’ll delve into the key things you need to know about OpenTelemetry and how this unified standard may become the future of how logs, metrics, events and traces are all handled.

Looking for Needles In a Stack of Needles? Develop an Observability Mindset

When I talk with Splunk customers, their challenges sometimes sound like trying to find a needle in a stack of needles. Feel the same way? The answers you need are out there, hidden in your data. Our job is to help you find them. Securing your networks, keeping them up and running and maximizing efficiency are key priorities. You also face the challenges of speeding up development and driving innovation to stay competitive.

Cribl Search: Powering the Future of Observability

Cribl Search turns the traditional search process on its head, allowing users to search data in place. No longer must data be collected and moved to storage before being examined. With Cribl Search, administrators can search data at the edge, moving through an observability pipeline, stored in a data lake, or even stored in their existing solutions like TSDBs or log stores.

How to deploy Grafana Enterprise Logs on Red Hat OpenShift

Here at Grafana Labs, we’re always looking for ways to provide our customers with a choice of platforms where they can run Grafana Enterprise Logs (GEL). As part of that mission, we’re pleased to announce that we’ve added Red Hat OpenShift 4.x support to GEL. GEL, as you may know, is a leading enterprise logs solution.

Splunk in the Financial Services Industry Today

In the late 1960s, there was a rock band called Ten Years After and I liked the name the first I heard about them. I wanted to use "Splunk and the Financial Services Industry: Ten Years After" as the title of this blog entry, but it’s been more than ten years since I wrote the first Splunk Blogs entry on Splunk and the Financial Services Industry. As you can tell, a lot has changed since then and more than a decade is an internet lifetime in technology.

Auto-Instrumenting NestJS Apps with OpenTelemetry

In this tutorial, we will go through a working example of a NestJS application auto-instrumented with OpenTelemetry. In our example we will use a simple application that outputs “Hello World!” when we call it in the browser. We will instrument this application with OpenTelemetry’s Node.js client library to generate trace data and send it to an OpenTelemetry Collector. The Collector will then export the trace data to an external distributed tracing analytics tool of our choice.

Synthetic Monitoring With Sematext | Release and Features | The Best Website Monitoring Solution

Sematext’s Synthetic monitoring tool is a website monitoring solution that lets you track the availability and performance of your websites. Monitor your entire website or an individual HTTP request, including 3rd party APIs. Get the best website monitoring tools with Sematext’s synthetics and Real User monitoring tools.

Logz.io's New Integration with AWS Kinesis: Send Metric Data Without a Single Line of Code

After creating your Logz.io account, the first step for onboarding is to send you log, metric, and trace data. Logz.io makes this flexible – allowing for multiple ways to get data into your Logz.io account depending on your use case and technology stack. Today, we’re excited to announce another easy and fast way to get AWS metric data into Logz.io: by setting up a CloudWatch metrics stream and a Kinesis Firehose.

BIG things are Happening at Graylog!

Did you hear the news? Graylog is on a mission to help make your IT environment and data more efficient and secure by making it super easy to uncover the answers stored in your machine data. At Graylog, coming up with solutions to problems faced by IT and Security professionals is what drives us. Our teams are always working on ways to add meaningful functionality that increases productivity so you can focus your resources on the innovation and core competencies that you’re known for.

How to Monitor Active Directory with OpenTelemetry

We’re excited to announce that we’ve recently contributed Active Directory Domain Services (abbreviated Active Directory DS) monitoring support to the OpenTelemetry collector. You can check it out here! You can utilize this receiver in conjunction with any OTel collector: including the contrib collector, the observIQ’s distribution of the collector, as well as Google’s Ops Agent, as a few examples.

Best practices to collect, customize and centralize Node.js logs

Node.js is an established platform for developing server-side applications in JavaScript. One of the most fundamental concerns that arise during the development of Node.js apps is how to carry out proper logging that will be safe, secure and performant. While there are several options for configuring Node.js logging, a few specific engineering best practices still apply, no matter which option you choose.

NGINX Logging Configuration: How to View and Analyze Access and Error Logs

NGINX is one of the most widely used reverse proxy servers, web servers, and load balancers. It has capabilities like TLS offloading, can do health checks for backends, and offers support for HTTP2, gRPC, WebSocket, and most TCP-based protocols. When running a tool like NGINX, which generally sits in front of your applications, it’s important to understand how to debug issues. And because you need to see the logs, you have to understand the different NGINX logging mechanisms.

Cribl Search: Redefining Search Around Today's Reality

As CEO of Cribl, one of my greatest privileges is to spend time on the road and on calls with our customers hearing about their needs and challenges. Cribl is a focused company. We build software for observability and security. With this lens, it becomes clear the industry is neglecting to address the unique needs of our users. There are many reasons, most of which are simply that vendors tend to come at a user’s problem through the lens of their existing technology.

Cribl Raises $150M in Series D Funding, Announces New Search Product, & Innovation Lab

We are pleased to announce we have closed our Series D round of funding, led by Tiger Global with participation from existing investors IVP, CRV, Redpoint Ventures, Sequoia, and Greylock Partners. In this round, we raised $150M, bringing our total raised to over $400M. This new fundraising round further validates the value we provide to customers.

Understanding the Priorities of Data Behind Tomorrow's Business Opportunities

Many CXOs believe that Web3 will power the next paradigm shift and transform the world. As a result, they are accelerating their learning curve to spot opportunities and leapfrog to next-gen business models that will catapult their organizations to new heights. But is there really an urgency to explore what Web3 can offer?

Interview With CTO, Leonard Trigg

For the next interview in our series speaking to technology specialists from around the world, we’ve welcomed experienced CTO, Leonard Trigg. Leonard is a member of the Harbourfront executive management team and serves as the firm's Chief Technology Officer. Joining the industry in 1995 Leonard has a background in enterprise technology, finance and operations. Leonard previously served as the Chief Operating Officer and Director at Vertex One Asset Management Inc.

Collect and visualize MySQL server logs with the updated MySQL integration for Grafana Cloud

Today, we are excited to announce that the MySQL integration has received an important update, which includes a new pre-built MySQL logs dashboard and the Grafana Agent configuration to view and collect MySQL server logs. The integration is already available in Grafana Cloud, our platform that brings together all your metrics, logs, and traces with Grafana for full-stack observability.

Get more insights with the new version of the Node.js library

We’re thrilled to announce the release of a new update to the Cloud Logging Library for Node.js with the key new features of improved error handling and writing structured logging to standard output which becomes handy if you run applications in serverless environments like Google Functions!

Kubernetes vs Docker vs Docker Swarm Differences | Pros & Cons Explained - Sematext

“Kubernetes vs. Docker” is one of the most commonly asked questions by new developers. But what is docker? And what is Kubernetes? If you want to become a full-stack developer, you will need to understand both of these technologies. In this video, we will take a look at what Docker is and how it fits into a developer's workflow. We will then look at what is Kubernetes and how it relates to Docker. We will then see what the difference is between docker and docker swarm.

How to monitor MongoDB with OpenTelemetry

MongoDB is a document-oriented and cross-platform database that maintains its documents in the binary encoded JSON format. Mongo’s replication capabilities and horizontal capability using sharding make MongoDB highly available. An effective monitoring solution can make it easier for you to identify issues with MongoDB such as resource availability, execution slowdowns, and scalability. observIQ recently built and contributed a MongoDB metric receiver to the OpenTelemetry contrib repo.

How Elastic powers speed, security, and connectivity in capital markets

Speed is everything in capital markets. Success in the front and back office is dependent on the ability to provide accurate, fast responses to challenging questions. Over the past several decades, there has been a tremendous increase in the amount of information available to market participants, and trade transactions are now being carried out at a very rapid pace. In parallel, the technology which capital markets firms are developing is becoming increasingly complex.

AWS Service Observability using OpenTelemetry

Efficient use of observability statistics is essential to any microservice architecture. OpenTelemetry is a project supported by the Cloud Native Computing Foundation (CNCF) to enhance the observability of microservice projects. AWS Distro for OpenTelemetry (ADOT) is an AWS-supported distribution of the OpenTelemetry project specifically designed to improve the observability of AWS projects.

Application logging with Flask

Without logs, or a good understanding of them, debugging an application or looking through an error stack trace can be challenging. Luckily, Flask logging can change the way you understand debugging and how you interact with logs produced by the application. The Flask logging module gives you a way to record errors over different severity levels. A default logging module is included in the Python standard library, and it provides both simple and advanced logging functions.

Retrace Power User Tips and Tricks - Error and Log Management

The explosive growth of ecommerce has slowed in the last year. But the need for businesses to deliver a great digital user experience continues to grow. Companies that don’t rely on online customer purchases can still suffer blows to revenues due to a poor online experience. Market conditions are raising the importance of Application Performance Monitoring (APM) tools to ensure every digital interaction with your company is positive. APM tools vary by design, features and functionality.

Log Observer Connect: Leverage the power of Splunk Enterprise data in Splunk Observability Cloud

With Splunk Log Observer Connect it’s easier than ever to correlate all of your metric, trace and log data to deliver better customer experiences! Available now for existing Splunk Enterprise and Splunk Observability Customers. Log Observer Connect lets observability users explore the data they’re already sending to their existing Splunk instances with Splunk Log Observer’s intuitive no-code interface integrated in Splunk Observability, for faster troubleshooting, root-cause analysis and better cross-team collaboration.

How to correlate logs and metrics with the Linux Node integration for Grafana Cloud

We are pleased to announce that an upgraded version of the Linux Node integration is available in Grafana Cloud, including the capability to visualize logs that are correlated with previously existing metrics. It also includes a new pre-configured dashboard based on the USE method, which focuses on showing resources utilization, saturation, and errors.

Benefits of Logging Agents

You probably have heard of logging agents, such as Logstash or Fluent Bit, if you’ve been investigating log analysis, monitoring, and observability. If so, and you’re wondering what logging agents are and why you might need them, you’ve come to the right place. This article will look at what logging agents are for, their advantages, and what you can use instead of a logging agent.

Alerting on error log messages in Cloud SQL for SQL Server

With Cloud SQL for SQL Server, you can bring your existing SQL Server on-premises workloads to Google Cloud. Cloud SQL takes care of infrastructure, maintenance, and patching so you can focus on your application and users. A great way to take better care of your application is by monitoring the SQL Server error log for issues that may be affecting your users such as deadlocks, job failures, and changes in database health.

All things logs: best practices for logging and Grafana Loki

What’s the saddest log line in the multiverse? A log line without context. That’s according to Grafana Labs software engineer and Grafana Loki tech lead Ed Welch, who joined Grafana Labs VP of Culture Matt Toback and Engineering Director Mat Ryer for the latest episode of “Grafana’s Big Tent," our new podcast about people, community, tech, and tools around observability.

Event Reduction in Four Easy Ways with Cribl Stream

One of Cribl Stream’s selling points is the reduction of ingested log volume, which helps our customers control costs and improve system performance. This can be accomplished in two ways – either by eliminating duplicate or unnecessary fields and null values within the events, or controlling the number of specific events that actually get sent to the destinations through strategic filtering.

How to Monitor Microsoft IIS with OpenTelemetry

The OpenTelemetry members at observIQ are excited to add Microsoft IIS metric monitoring support to OpenTelemetry! You can now easily monitor your IIS web servers with the oIQ OpenTelemetry Collector. You can add the IIS metric receiver to any OpenTelemetry collector. This post demonstrates just one configuration for shipping metrics with OpenTelemetry components. This configuration and many other observIQ OpenTelemetry configurations are available in the oIQ Opentelemetry Collector.

New observability features for your Splunk Dataflow streaming pipelines

We’re thrilled to announce several new observability features for the Pub/Sub to Splunk Dataflow template to help operators keep a tab on their streaming pipeline performance. Splunk Enterprise and Splunk Cloud customers use the Splunk Dataflow template to reliably export Google Cloud logs for in-depth analytics for security, IT or business use cases.

Cyclical Statistical Forecasts and Anomalies - Part 6

At this point we are well past the third installment of the trilogy, and at the end of the second installment of trilogies. You might be wondering if the second set of trilogies was strictly necessary (we’re looking at you, Star Wars) or a great idea (well done, Lord of the Rings, nice compliment to the books). Needless to say, detecting anomalies in data remains as important to our customers as it was back at the start of 2018 when the first installment of this series was released.

Docker Log Rotation Configuration | Container Logging for Beginners - Sematext

Docker logs are one of the primary sources of information developers use to spot problems with their apps. However, Docker log files can get huge in a short amount of time. This is why you absolutely must configure Docker log rotation! 🔥Today, we will explain how to free up space on the host machine of your Docker containers. We will also look at how to set up a centralized logging solution for free to get the most out of your Docker logs.

10 things you didn't know about LogQL

For this edition of my ongoing Grafana Loki how-to series, I wanted to offer up some helpful — and perhaps surprising — facts about using LogQL, Loki’s query language. In case you’re new to Grafana Loki, it’s a log aggregation system created in 2018, and the Loki team has worked with the community ever since to introduce new features and make it easier to deploy.

Webinar Recap: Best Practices for Right-Sizing and Overhauling Your Architecture

Last week, we hosted a webinar on the easiest way to right-size – and safest way to overhaul – your architecture. One of the scenarios we’re seeing come up more and more with prospects and customers is the need to update your architecture, and particularly your security architecture, as new needs and threats arise. As I’m sure you all know, that can be a real hassle, put a strain on your resources, and put your security posture at risk if it isn’t done well.

4 Different Ways to Ingest Data in AWS OpenSearch

AWS OpenSearch is a project based on Elastic’s Elasticsearch and Kibana projects. Amazon created OpenSearch from the last open-source version of ElasticSearch (7.10) and is part of the AWS system. The key differences between the two are topics for another discussion, but the most significant point to note before running either distribution is the difference in licenses. ElasticSearch now runs under a dual-license model, and OpenSearch remains open-source.

Up the Creek Without a Paddle: Easing the Strain on Your Analytics Systems

When it comes to your analytics tools, would you say they’re getting easier to manage overall, or is it increasingly difficult? Can you easily scale to meet new compliance requirements, or is there so much custom work required that the pace of change is too much for your team to handle? Do you feel in control over how and where your observability data flows, or do you feel beholden to your vendors? This blog post will shed light on how you can ease the strain on your downstream systems.

The Power Of The Ecosystem: Intel and Splunk Help Partners Bring Data To Life

Last year, International Data Corporation released its Data GlobalSphere Forecast, 2021-25, in which it outlined the projected 23% compound annual growth in data, leaping to 175 zettabytes of data globally. So the natural question becomes, what will the world do with that much data? And, more importantly, what can your business do with your data?

Centralized Log Management for Reporting Compliance to Management

It’s 3pm on a Wednesday, and you’re really just done with the week already. You hear that “ping” from your Slack and know that you set notifications for direct messages only, which means, ugh, you have to pay attention to this one. It’s your boss, and she’s telling you to check your email. Then you see it, the dreaded audit documentation request. This will take you the rest of today and most of tomorrow.

How Sumo SREs manage and monitor SLOs as Code with OpenSLO

At Nobl9’s annual SLOconf—the first conference dedicated to helping SREs quantify the reliability of their applications through service level objectives (SLOs)—Sumo Logic shared our contribution of slogen to the OpenSLO community, as well as our commitment to OpenSLO as an emerging standard for expressing SLOs as Code. slogen is an open source, SLO-as-code CLI tool based on the OpenSLO specification.

New in Grafana 8.5: How to jump from traces to Splunk logs

The recent release of Grafana 8.5 marks the start of enabling the jump from traces directly to Splunk logs. It’s a big leap that now allows you to draw a straight line between your traces — whether they are coming from Tempo, Zipkin, or Jaeger — to even more third-party logging data, all from the comfort of your traces view. Previously, the Grafana trace to logs enablement included only Loki logs.

How Offishall Uses DigitalOcean and Papertrail to Simplify Hybrid Work

Paris-based tech startup Offishall is all about simplifying and streamlining modern hybrid work. CTO Bruno Ronzani and his team rely on reliability, speed, and simplicity from DigitalOcean Droplets and Papertrail™ log management. This foundation helps ensure Offishall delivers the seamless web experience their customers—and regional manager Dwight K. Schrute—demand.

Kubernetes Logging with Elasticsearch, Fluentd and Kibana

Kubernetes, a Greek word meaning pilot, has found its way into the center stage of modern software engineering. Its in-built observability, monitoring, metrics, and self-healing make it an outstanding toolset out of the box, but its core offering has a glaring problem. The Kubernetes logging challenge is its ephemeral resources disappearing into the ether, and without some 2005-style SSHing into the correct server to find the rolled over log files, you’ll never see the log data again.

Going Beyond CloudWatch: 5 Steps to Better Log Analytics & Analysis

CloudWatch is great – if you require very basic logging and monitoring for the Amazon Web Services (AWS) cloud, at least. However, the reality is that most teams need more than basic logging and monitoring. They may also need to perform log analytics on data sources from outside AWS, which CloudWatch doesn’t support. That’s why, although CloudWatch may be one tool in your log analytics strategy, it probably should not be the only one.

Getting Better Sysmon Data Using Cribl Stream

System Monitor, better known as Sysmon, is one of my favorite security datasets. The data is crazy detailed and offers a great way to power security detection and response since it gives cyber security teams a roadmap to understand exactly what systems or people are doing while they use any Windows operating systems. The avalanche of the data is the downside and why observability engineers need tools like Cribl Stream to manage and enrich Sysmon data to make it more useful and more cost-effective.

ElasticON Solution Series Keynote: Celebrating 10 Years of Elastic

Learn more about Elastic's origin story and how the world's most popular search engine evolved into the leading platform for search-powered solutions. Since the release of Elastic 7.0 there have been 17 additional 7.x releases. In less than 20 minutes you’ll hear the highlights from two years of Elastic 7.x and explore the latest news from 8.x — and — what’s to come in the future. Speaker: Mike Nichols, Product Lead - Elastic Security, Elastic

ElasticON Solution Seminar Customer Conversations: Putting Data to Work

Hear first hand accounts from Elastic customers on how they are using the power of search to solve for unique challenges and to reach new levels of success. Speakers: Ali Nazemian, Chief Technology Officer, Brolly Kevin Serafin, Director of Incident Response, Ecolab Matt Riley, General Manager, Enterprise Search, Elastic

Survey Review: Key Challenges of Scaling Observability with Cloud Workloads

When you migrated critical infrastructure to the cloud, what were your goals and expectations? Odds are, you hoped leaving on-premises infrastructure would produce significant organizational benefits. You probably figured you’d streamline operations and reduce management overhead. You felt you’d have an easier time meeting business goals. Perhaps most important of all, you likely expected your environment would become less complex, and even cost less to operate.

Smart, Secure and Sustainable Manufacturing - How Splunk and Google Cloud Are Helping Manufacturers to Skate Where the Puck is Going

* Co-author: Alexander Okl, Sr. Partner Development Manager EMEA | Google Cloud at Splunk “The way we look at manufacturing is this: the strategy should be to skate where the puck is going, not where it is.” - Tim Cook, CEO, Apple Inc.* So where is the puck going for manufacturers in 2022 and beyond?

What is Kibana? (Updated Guide For 2022)

Kibana is a popular user interface used for data visualisation and for creating detailed reporting dashboards. This piece of software notably makes up a key part of the Elastic Stack alongside Elasticsearch and the extract, transform and load (ETL) tool, Logstash. In this comprehensive introduction to Kibana, we are covering all of the basics that you will need to know as a user considering using Kibana for your log data visualisation and reporting needs.

Micro Lesson: Troubleshoot an Incident Using Root Cause Explorer

The video uses a scenario to demonstrate how to use Root Cause Explorer to analyse and troubleshoot an incident faster. The video shows how Root Cause Explorer helps you dig deeper into the relevant logs and traces in order to isolate the root cause using various dashboards.

Where's Open Source Observability Headed in 2022?

For the last five years, Logz.io has tracked and measured the pulse of DevOps, as well as adoption of key trends and technology, through our DevOps Pulse survey and report. One of the obvious focus areas for us, as a company whose products are based on industry-leading open source, is the increased rise of incredibly useful open-source observability solutions, in general.

Protect the Business with Cribl Packs: Webinar Recap

The second in our Feature Highlights webinar series, Protect the Business with Cribl Packs, highlights Packs and security use cases. Packs enable you to share complex Stream/Edge configurations across multiple Worker Groups/Fleets, between Stream/Edge deployments or with the Cribl Community. Packs roll up best practices to ensure Site Reliability Engineering (SRE) teams have the required data to protect the business.

CI/CD Detection Engineering: Dockerizing for Scale, Part 4

Splunk builds innovative tools which enable users, their teams, and their customers to gather millions of data points per second from an ever-growing number of sources. Together, Splunk helps users leverage that data to deliver, monitor, improve, and secure systems, networks, data, products, and customers with industry-leading solutions and expertise.

Elastic Observability 8.2: Tail-based sampling, plus more serverless visibility for AWS

As more organizations adopt cloud-native technologies and microservices-based architectures, application troubleshooting is becoming increasingly complex. With so many moving parts in an environment that is both dynamic and distributed, it is difficult to get the full picture. Yet complete visibility is crucial in order to find and fix issues quickly — especially ones that impact the bottom line.

Elastic Enterprise Search 8.2: Relevance controls for Elasticsearch

Elastic Enterprise Search 8.2 introduces new ways to ingest, search, and monitor data, giving developers the productivity benefits of using out-of-the-box capabilities along with the power and flexibility inherent in Elastic Stack tools. Operators also gain even more transparency for managing search experiences and observing search performance. For a visual walkthrough of some of the key capabilities in 8.2, check out the latest installment of What’s new in Enterprise Search on YouTube.

How to Monitor Riak Metrics with OpenTelemetry

observIQ’s OpenTelemetry members contributed Riak metric monitoring support to OpenTelemetry! You can now monitor your Riak agent performance with OpenTelemetry, and deploy simply with the oIQ OpenTelemetry Collector. You can add the Riak metric receiver to any OpenTelemetry collector. This post demonstrates a configuration for shipping metrics to Google Cloud Operations with OpenTelemetry components.

Proactive Monitoring vs. Reactive Monitoring

Monitoring is a fundamental pillar of modern software development. With the advent of modern software architectures like microservices, the demand for high-performance monitoring and alerting shifted from useful to mandatory. Combine this with an average outage cost of $5,600 per minute, and you’ve got a compelling case for investing in your monitoring capability.