Lexicon
Tail sampling
Tail sampling decides which traces to keep after they complete, so errors and outliers survive. How the policies work and what the buffering really costs.
Definition
Tail sampling is a sampling strategy that makes the keep or discard decision after a trace completes, when its outcome is known. That timing lets it keep the interesting traces, the errors, the outliers, the slow ones, while discarding routine successes.
What it means in observability
Tail sampling exists to fix head sampling's blindness. A decision made at the start of a request cannot know how that request will end, so a head sampler discards failures and successes with equal indifference. Deciding at the end means the sampler can look at what actually happened and keep what matters. For teams that must sample, it concentrates the retained data where the debugging value is, which is why it has become the default recommendation for trace volume control.
How it works in practice
In practice tail sampling runs in an OpenTelemetry Collector gateway. The Collector buffers all spans of a trace until the trace completes or a decision timeout passes, then evaluates policies. Common policies come in three families: status based, keeping traces containing errors; latency based, keeping traces above a duration threshold; and probabilistic, keeping a baseline percentage of everything else. Because every span of a trace must reach the same Collector instance for the decision to see the whole picture, scaled deployments add trace aware load balancing in front of the sampling tier.
Where it gets hard
The buffering is the bill. Holding every in flight trace in memory makes the sampling tier a stateful, resource hungry system that must be sized, monitored, and scaled like the production infrastructure it is. Decision timeouts truncate long running traces, so slow workflows get judged on partial evidence. And even done perfectly, tail sampling still discards data: the routine traces it drops are the baseline that anomaly detection and honest percentiles depend on.
Where Tsuga fits
Tsuga works with standard Collector tail sampling for teams that choose it, but its flat per GB economics exist to make the choice optional. Keeping complete traces is usually simpler than operating the machinery that decides which ones to lose.
Related terms
- OpenTelemetry CollectorThe OpenTelemetry Collector is a standalone process that receives telemetry from applications and infrastructure, transforms it, and exports it to one or more destinations.
- SamplingSampling is the practice of keeping only a subset of telemetry, usually traces, so that volume and cost stay manageable.
- SLOAn SLO, service level objective, is a target for how reliable a service should be, expressed as a measured indicator held over a time window: 99.9 percent of requests succeed over 30 days, or 95 percent of checkouts complete in under two seconds.
- TraceA trace is the end to end record of one request or workflow as it moves through a system, composed of all the spans that share a single trace ID.