Watch what your customers feel
Put monitors on the handful of signals that matter to the business, such as checkout errors, API latency or an SLO. You do not need a monitor on every service to reach every service's team.
Page the team that can fix it.
When one service fails, every service that calls it alerts too, and most of those pages wake up teams that cannot fix anything. Adaptive notifications read the traces behind each alert and page the team that owns the cause, with the context it needs to start fixing it from the first message.
Alerting that reaches the right team
Alerting on the signals customers feel is the right strategy, but it has always had one weak point: the page goes to whoever owns the system closest to the product interface rather than whoever owns the cause. Here is how that shows up during an incident, and what changes when the routing comes from your traces.
In a microservice architecture, a fault often affects surrounding systems. Take a standard e-commerce system: when the payments service fails, checkout fails with it, and so does the frontend, so every team in the call path gets a page for the same incident while the team that owns the cause is still asleep. Writing more service-level monitors to close the gap only adds to the noise, and the first twenty minutes of the incident go to working out whose problem it is.
How Tsuga helps
Configure your high priority monitors and service level objectives (SLOs) on business signals, such as checkout errors or API latency, and add an adaptive rule on top. When one fires, we find the root-cause service in the traces and notify its destination, so one incident produces one page to the team that can act on it.
One incident should mean one page.
If you already send distributed traces to Tsuga, there is nothing new to instrument. Talk to an architect about putting your most critical monitor on an adaptive rule.
How it works
You decide what is worth paging on, and Tsuga works out who should receive the page each time it fires. Here is what happens between the alert and the page.
Let the traces find the owner
When a monitor fires, Tsuga follows the failing requests through your services and identifies the one where the failure starts. Anything it cannot attribute with confidence goes to a fallback destination you choose.
Page the team that can fix it
The page goes to that service's own destination, in the on-call tool its team already uses. It arrives with the reason it was routed there and pre-built investigation context to start from.
What you get
Routing by root cause
Each alert is routed by where the failure starts, not by who owns the monitor that noticed it. The teams further up the call path are left to sleep.
Pages that explain themselves
Every adaptive notification names the service it identified and opens a Tsuga investigation that records how the decision was made. The team that receives it can check the reasoning before acting on it.
A fallback for anything unclear
When the evidence does not point to one service, or that service has no destination yet, the alert goes to a fallback destination you choose. It is still delivered to someone who can triage it.
Your on-call tools, unchanged
Pages go to Slack, incident.io, PagerDuty, Microsoft Teams, ServiceNow, Jira, email or a webhook. Renotifications and recovery notices follow the original page, so the team that was paged hears when it is resolved.
SLO alerts with an owner
Burn rate and threshold alerts on SLOs built from your traces and span metrics are routed the same way. The objective your customers feel finally reaches a team that can act on it.
Set up once per service
Configure each service's destination once, from the service page or through the API. Adaptive rules sit alongside your existing notification rules, so you can start with one monitor.
Is it right for you?
Adaptive notifications rely on distributed traces, and they are most useful in highly distributed systems where many teams share a call path. If that is not your setup yet, we will tell you upfront.
Adaptive notifications fit if you...
- Run distributed services where one failure sets off alerts across several teams
- Send distributed traces to Tsuga, or plan to
- Want to page on customer-facing signals and SLOs rather than on every service
- Already use an on-call tool such as PagerDuty or incident.io and want pages routed into it
They are not a fit yet if you...
- Build all of your alerting on log-based metrics, with no traces behind it
- Run a small number of services owned by a single team, where every page already reaches the right people
Frequently asked questions
Adaptive paging is the name engineering teams use for routing a page to whoever owns the cause, a practice large organizations have built for themselves. Adaptive notifications are our implementation of it, built into Tsuga, so there is nothing for your team to write or maintain.
Page the team that can fix it.
Adaptive notifications run alongside the notification rules you already have, so you can start with a single customer-facing monitor. We will walk you through setting it up on your own services.