Telemetry data sent from applications often contains Personally Identifying Information (PII) like names, user IDs, phone numbers, and other information that must be obfuscated before the data is sent to storage or observability tools, in order to be in compliance with corporate or government policies such as HIPAA in the US or the GDPR in the EU.

At the most abstract level, a data pipeline is a series of steps for processing data, where the type of data being processed determines the types and order of the steps. In other words, a data pipeline is an algorithm, and standard data types can be processed in a standard way, just as solving an algebra problem follows a standard order of operations.

OpenTelemetry (OTel) is an open-source framework designed to standardize and automate telemetry data collection, enabling you to collect, process, and distribute telemetry data from your system across vendors. Telemetry data is traditionally in disparate formats, and OTel serves as a universal standard to support data management and portability.

Here at Mezmo, we see the purpose of a telemetry pipeline is to help ingest, profile, transform, and route data to control costs and drive actionability. There are many ways to do that as we’ve previously discussed in our blogs, but today I’m going to talk about real-time alerting on data in motion, yes - on streaming data, before it reaches its destination.