The complete guide to Kubernetes application monitoring

Which layer you are monitoring

Kubernetes observability spans three layers, and each answers a different question. The cluster and infrastructure layer covers nodes, the control plane, the scheduler and etcd, and it answers whether the platform itself is healthy through signals such as node readiness, resource utilization and API server latency. The workload layer covers pods, containers, Deployments, ReplicaSets and StatefulSets, and it answers whether the right number of healthy instances is running, through pod restarts, container OOM kills and rollout status.

The application layer is the subject of this guide. It covers request latency at p95 and p99, error rates, throughput and saturation, and it answers whether the code inside those pods is doing its job. The cluster and workload layers are covered separately.

A useful way to draw the line is to look at where a metric comes from. cAdvisor, embedded in the kubelet, collects CPU, memory and filesystem metrics for each container, and the Metrics API aggregates that same data for autoscaling and kubectl top rather than being a separate source. Metrics your own code emits through OpenTelemetry or Prometheus client libraries are the application layer, and that distinction matters because you can have healthy nodes, fully scheduled pods and no infrastructure alerts while every user gets a 500.

What to measure: the golden signals

The four golden signals defined in Google's SRE book work as a practical filter. When all four sit within acceptable ranges your users are probably fine, and when one crosses its threshold something is already going wrong or is about to.

Latency

Avoid alerting on average latency. Take 100 requests where 99 finish in 10ms and one takes 10,000ms: the average is 108ms, which looks reasonable, while the p99 is 10,000ms, which is what one of your users actually experienced.

Track p50 for the typical experience, p95 for most users and p99 for the slowest requests. It also helps to separate successful requests from errors, because a validation failure might return in 5ms while a successful database query takes 200ms, and combining them makes your p95 look healthier than it is.

Traffic

Request volume puts the other three signals in context, since a one percent error rate during normal traffic means something different from the same rate during a sudden spike. In Kubernetes it also shows whether horizontal pod autoscaling is behaving, because a spike should produce more replicas.

If latency rises while traffic stays flat, the cause is more likely to be in the code or a dependency than a lack of capacity. That one comparison rules out a whole branch of the investigation before you start.

Errors

Errors are more than 5xx responses. Timeouts, failed policy checks and even 200 responses carrying the wrong data all count, and separating 4xx from 5xx is worth doing because they usually point at different causes.

Break errors down by service and endpoint as well. A five percent error rate on a health check endpoint has very different implications from five percent on a payment endpoint.

Saturation

Saturation is the hardest of the four to measure, because the limiting resource depends on the service. A cache may be bound by memory while a web service is bound by its thread pool, and neither shows up as the obvious number on a node dashboard.

Node CPU can sit at 43 percent while a pod is throttled at its cgroup CPU quota. The on-call engineer sees normal node CPU while p99 latency climbs from 196ms to 288ms, which is why saturation belongs at the application level, measured through thread pool utilization, connection pool usage and CPU throttling rate rather than node metrics alone.

How to instrument applications on Kubernetes

Instrumentation options fall into a few categories, and most production clusters combine them rather than picking one. The right mix usually depends on how many services you run and how much of their behavior lives outside the frameworks an agent understands.

Manual SDK instrumentation

Adding OpenTelemetry SDKs directly to your code, creating a tracer, configuring an OTLP exporter and wrapping operations in spans, gives you the most control. It is the only approach that captures Kubernetes-specific context such as namespace, deployment name or pod template hash alongside business context such as user IDs and transaction types.

The trade-off is that every service needs code changes, and the effort scales with the number of services. There is ongoing maintenance too, since the OpenTelemetry Go SDK releases roughly every six weeks with patch releases in between, and while breaking changes are reserved for major versions under semantic versioning, experimental modules can still change in minor releases.

Auto-instrumentation through the OpenTelemetry Operator

Auto-instrumentation avoids most code changes. You deploy the Operator, create an Instrumentation custom resource and annotate a deployment or namespace, and a mutating admission webhook injects a language-specific agent for .NET, Java, Node.js, Python or Go.

Go works differently because it compiles statically, so the Operator injects a sidecar container instead of an agent. This gets you baseline coverage across many services quickly, though it only captures what supported frameworks expose, which leaves custom business logic, background workers and asynchronous code paths invisible.

Sidecar or DaemonSet

This choice is about where the Collector runs rather than how instrumentation happens. The trade-offs are set out below, and the last row is the one that tends to decide it.

Tail sampling requires every span in a trace to reach the same Collector. With a DaemonSet, spans can be routed unpredictably across pods, so teams that rely on tail sampling usually deploy backend Collectors as a StatefulSet with a load-balancing exporter that routes by trace ID.

Service mesh telemetry

Istio sidecar proxies generate traces for requests passing through the mesh, giving you request rate, latency and error rate without code changes. The limitation is that a mesh only sees network traffic, so exceptions, validation failures and business context stay invisible, which makes mesh telemetry a supplement to SDK instrumentation rather than a replacement for it.

Practices worth adopting early

Propagate trace context consistently

A trace only survives a service boundary when both sides use the same propagation format. Mixing W3C TraceContext with B3 or Jaeger breaks traces silently, leaving gaps without an obvious error to investigate.

OpenTelemetry carries context in a traceparent header, and where multiple formats appear in the same pipeline the W3C headers take priority. Asynchronous messaging needs a different approach from synchronous calls, because a Kafka producer should inject traceparent into the message while the consumer links its span back to the producer span rather than treating it as a direct parent, since consumer execution happens independently. One more detail catches people out: the kube-apiserver propagates context on outgoing requests but ignores incoming context, so traces entering through the API server do not automatically continue into internal component traces.

Keep service naming consistent

Inconsistent naming makes cross-service queries harder than they need to be, with one team using http.request while another uses HTTP_REQUEST, and user identity appearing as user.id, userId or customer.id. The semantic conventions exist to settle exactly this.

The worst of these anti-patterns is putting the service name inside the metric name. Use transaction.count with service.name=payment rather than payment_transaction_total, because the service name is already a resource attribute and duplicating it in the metric name makes aggregation across services harder than it should be.

Define SLOs per service, not system-wide

A 99.9 percent availability target for an entire system does not tell you which service needs attention when something breaks. SLOs are more useful mapped to individual services and their critical paths, defined as SLIs you can query, given a target per service and where necessary per endpoint, since a payment endpoint and a health check deserve very different objectives.

Alert on error budget burn rate rather than raw error rate, because that is what separates a brief spike from sustained degradation. Teams that ship SLOs successfully tend to start with five measures per service: availability as a success rate, p95 latency, p99 latency, a request-rate floor confirming the service is receiving traffic at all, and burn rate.

Correlate traces to pods and nodes

The Kubernetes Attributes processor attaches k8s.pod.name, k8s.node.name and k8s.deployment.name to spans, metrics and logs, which is the metadata that lets you move from an application problem to the infrastructure underneath it. Without it, a trace showing the payment service is slow does not tell you which of its twenty replicas is responsible.

One configuration detail decides whether this works. Run k8sattributes before tail sampling in the Collector pipeline, or the sampling decision happens before the Kubernetes metadata is attached and the sampler makes its choice without the context it needed.

Where it gets hard

Sampling decisions

Fixed sampling rates create an uncomfortable trade-off. Sample aggressively and you lose the errors that matter, sample lightly and telemetry volume becomes difficult to manage.

Head-based sampling decides when a trace starts and keeps a random portion, so a failure landing in the discarded ninety percent is simply gone. Tail-based sampling waits until the trace completes, which lets you keep errors and slow requests while dropping routine traffic, at the cost of additional resource overhead. There is also the requirement that is easy to overlook, which is that every span in a trace must reach the same Collector, so Collectors running as a Deployment behind a load-balanced Service will scatter spans across pods and quietly break the decision.

Distributed tracing across ephemeral pods

Pods are temporary by design. A crashed pod's termination state can be overwritten within roughly ninety seconds, scheduler placement decisions may be pruned within an hour, and even a kubectl debug session leaves no persistent record once it ends.

Tracing cannot close this gap on its own. A trace shows that a request failed, but it cannot tell you the node behind that request was under memory pressure unless that infrastructure event was captured separately and kept.

Mapping application errors to infrastructure causes

An application error such as a database connection timeout often has an infrastructure cause, for example the database pod running out of memory and restarting. These events usually live in separate systems with different data models, which is what makes the causal link easy to miss.

Two things are worth checking before you settle on a cause. Restart counts alone are a weak signal, so look at OOM kills and the resource limits behind them, and remember that an OOMKilled pod does not by itself indicate a memory leak, because the limit may simply be too low for the workload. Compare usage against a seven day baseline before drawing that conclusion.

Retention cost on trace data

Kubernetes generates far more telemetry than VM-based infrastructure did, because workloads are spread across services, pods, replicas and namespaces. The multiplication is not gradual, and the table below is the shape of it.

That kind of increase is what makes per GB and per host pricing painful, and the response is predictable. Teams shorten retention to a few days or sample down to one trace in ten, and the trace they remove is sometimes the one that would have explained the next incident.

How teams close these gaps

Ephemeral pod tracing

This is mostly an architectural problem. A common answer is two tiers of Collector, where the frontend layer runs as a Deployment and uses the load-balancing exporter to hash spans by trace ID, and the backend layer runs as a StatefulSet with stable pod DNS names exposed by a headless Service.

That arrangement sends every span from a given trace to the same backend pod, which is what tail sampling needs to make a reliable decision. Processor order still matters, so k8sattributes runs before tail_sampling and the metadata is there when the decision is made.

Tail-sampling memory pressure

Buffering traces in RAM gets difficult at scale, since a service processing 100,000 spans per second may need to hold around 12 million spans during a two minute decision window. Atlassian hit this and built a custom tail-sampling processor for the Collector that compresses spans in memory, applies policies in priority order and scales horizontally without dropping data.

They report that it substantially reduced their tail-sampling compute costs. Most teams cannot justify that level of engineering effort for a sampling processor alone, which is worth saying plainly before anyone plans a quarter around it.

Connecting application and infrastructure data

The fix for siloed data is a shared data model. Once k8sattributes adds pod name, node name and namespace to spans, metrics and logs, an engineer can move from an application error to the relevant trace, log entries and infrastructure event using the same identifiers.

Timing matters here too. Enrichment has to happen before sampling or filtering removes data, because metadata added afterwards cannot restore context that has already been discarded.

Retention cost

Retention is the one that does not yield to a configuration change. When observability costs rise with every additional gigabyte and host, teams have a financial incentive to keep less, and that incentive is what produces the shortened windows and the aggressive sampling rates.

The pricing model is the root of it rather than the technology. Change how the data is priced and the sampling decision stops being a budget question, which is the approach we take.

Where Tsuga fits

We run inside your own cloud environment, on AWS, GCP or Azure. Storage, indexing and processing stay behind your VPC instead of being shipped to a vendor's infrastructure, we operate the control plane, UI and orchestration, and the data plane stays in your account where it can be encrypted with your own KMS keys.

That changes the economics rather than the engineering. Flat per GB pricing with retention included, and no per host or per user fees, means storage and compute land on your own cloud bill at your provider's rates, and adding pod, namespace, deployment or node context does not turn into a second pricing problem. Collection is OpenTelemetry native and built on open formats, so telemetry stays readable in your own object storage rather than locked into a proprietary layer.

The practical result is that full trace retention becomes easy to justify. You are not sampling down to ten percent and hoping the trace you need survived, and you are not cutting two weeks to three days to stay inside a budget, which means an investigation starts with complete traces and the Kubernetes context around them.