Lexicon
Sampling
Sampling keeps only a fraction of telemetry to control cost. How head and tail sampling work, and why the trace you need most is the one sampling dropped.
Definition
Sampling is the practice of keeping only a subset of telemetry, usually traces, so that volume and cost stay manageable. A sampler decides which data survives, either when a request begins or after it completes, and everything else is discarded before it ever reaches storage.
What it means in observability
Sampling exists because of economics, not engineering preference. Nobody wants less data about their systems; teams sample because full fidelity telemetry at traditional observability prices is unaffordable, and the AI era made it worse as volumes compound faster than budgets. So sampling rates climb, quietly, one cost review at a time. Each increase looks reasonable in isolation. Collectively they shrink coverage until observability describes a thinning statistical sketch of production rather than production itself.
How it works in practice
Head sampling decides at the start of a trace, typically keeping a fixed percentage, which is cheap and simple but blind: it cannot know whether the request it is discarding will turn out to be the interesting one. Tail sampling waits until a trace completes and applies policies, keeping errors and outliers at higher rates, which is smarter but operationally heavier. Logs get sampled and dropped by level or volume too, while metrics, being pre-aggregated, mostly escape the knife.
Where it gets hard
Sampling fails silently, and that is the heart of the problem. The customer escalation lands, you search for the trace, and it does not exist, because a one percent sampler discarded it along with the other ninety nine. Rare failures are precisely what samplers are statistically built to throw away. Worse, sampled data quietly distorts everything computed from it: error rates, percentiles, SLO measurements, and the AI systems now being pointed at telemetry, all reasoning over a partial record while presenting conclusions as though it were whole.
Where Tsuga fits
Tsuga's flat per GB pricing inside your own cloud account changes the equation that forces sampling in the first place. When keeping everything is affordable, sampling becomes a deliberate engineering choice rather than a survival tactic, and the trace you need is there when you go looking.
Related terms
- AIOpsAIOps is the application of machine learning and artificial intelligence to IT operations: detecting anomalies, correlating and grouping alerts, suggesting root causes, and increasingly, powering assistants that investigate telemetry conversationally.
- CardinalityCardinality is the number of distinct values, or distinct value combinations, that an attribute or set of attributes can take.
- DatadogDatadog is the largest SaaS observability platform, spanning infrastructure monitoring, APM, logs, RUM, security, and dozens of adjacent products, collected largely through its proprietary agent and priced per product.
- Distributed tracingDistributed tracing is the technique of following a request across service boundaries by recording a connected span for each operation it touches.
- Egress costsEgress costs are the fees cloud providers charge for data leaving their network, priced per gigabyte and varying by destination.
- Log levelsA log level is the severity label attached to a log line: DEBUG, INFO, WARN, ERROR, and FATAL in the common scheme.
- Log managementLog management is the practice of collecting, processing, storing, and searching log data at scale: every event record your applications and infrastructure emit, made findable when someone needs it.
- Observability pipelineAn observability pipeline is the layer that moves telemetry from where it is produced to where it is stored, transforming it along the way: parsing, enriching, redacting, filtering, sampling, and routing to one or more destinations.
- OpenTelemetry CollectorThe OpenTelemetry Collector is a standalone process that receives telemetry from applications and infrastructure, transforms it, and exports it to one or more destinations.
- SLIAn SLI, service level indicator, is the measurement behind an SLO: the specific, computable signal that stands in for whether users are having a good experience.
- SLOAn SLO, service level objective, is a target for how reliable a service should be, expressed as a measured indicator held over a time window: 99.9 percent of requests succeed over 30 days, or 95 percent of checkouts complete in under two seconds.
- Tail samplingTail sampling is a sampling strategy that makes the keep or discard decision after a trace completes, when its outcome is known.
- TraceA trace is the end to end record of one request or workflow as it moves through a system, composed of all the spans that share a single trace ID.