Lexicon
Observability pipeline
An observability pipeline collects, transforms, and routes telemetry between sources and destinations. What pipelines are really for and where they fail.
Definition
An observability pipeline is the layer that moves telemetry from where it is produced to where it is stored, transforming it along the way: parsing, enriching, redacting, filtering, sampling, and routing to one or more destinations. It is infrastructure for data in motion.
What it means in observability
Pipelines exist because telemetry rarely arrives in the shape you want to keep. Raw logs need parsing into attributes, everything needs enrichment with context, sensitive values need redacting before storage, and different signals may need different destinations. A pipeline centralizes those decisions instead of scattering them across every service, and it decouples producers from backends, which is both an engineering convenience and a negotiating position: when telemetry flows through a layer you control, no single destination owns your data by default.
How it works in practice
Pipelines compose three stages. Collection receives telemetry from applications, agents, and platforms. Processing transforms it in flight: parsing text into structure, normalizing fields, adding metadata, redacting secrets, and applying volume controls like filtering and sampling. Routing delivers results to destinations, sometimes several at once: production telemetry to the observability platform, an archive copy to object storage, a compliance stream to long retention. The OpenTelemetry Collector is the open standard implementation, and platforms add ingest side processing on top.
Where it gets hard
A pipeline is another production system: it needs capacity, monitoring, and an owner, and a pipeline that silently drops data is a coverage hole nobody alerted on. Configuration complexity compounds with every route and transform. And there is a telling anti pattern: pipelines purchased primarily to throw data away, elaborate machinery for deciding which telemetry you can afford to lose, which treats the symptom of broken pricing rather than the cause.
Where Tsuga fits
Tsuga builds pipeline capability in: routes with ordered processors handle parsing, mapping, and enrichment at ingest, and the sensitive data scanner catches secrets after route processing. With flat per GB pricing, the pipeline's job is shaping data, not rationing it.
Related terms
- GrafanaGrafana is the open source standard for dashboards, and Grafana Labs builds an observability stack around it: Loki for logs, Tempo for traces, Mimir for metrics, with Grafana Cloud as the managed offering.
- Log levelsA log level is the severity label attached to a log line: DEBUG, INFO, WARN, ERROR, and FATAL in the common scheme.
- Log managementLog management is the practice of collecting, processing, storing, and searching log data at scale: every event record your applications and infrastructure emit, made findable when someone needs it.
- OpenTelemetry CollectorThe OpenTelemetry Collector is a standalone process that receives telemetry from applications and infrastructure, transforms it, and exports it to one or more destinations.
- SamplingSampling is the practice of keeping only a subset of telemetry, usually traces, so that volume and cost stay manageable.
- TelemetryTelemetry is the data software and infrastructure emit about their own behavior.