At 2:47am, a payment processing service starts degrading. Within ninety seconds, your alerting stack has fired forty-three separate notifications: Prometheus Alertmanager triggers on HTTP error rate, Datadog fires on p99 latency, Kubernetes sends OOMKill events for two pods, CloudWatch flags an RDS connection pool near-exhaustion, and a PagerDuty escalation chains through three on-call rotations. Your engineers wake up to a Slack channel that looks like a fire hose.
Nobody investigates the root cause at this point. They're too busy triaging which of the forty-three alerts are actually distinct problems and which are the same problem wearing different clothes. This is alert fatigue — and it's not primarily a human attention problem. It's a signal architecture problem.
The anatomy of an alert storm
Alert storms happen when a single root cause propagates through multiple observable dimensions simultaneously. A config change deploys at 2:45am. It introduces a memory leak in one service. That service starts hitting its container memory limit. Pods begin OOMKilling. Upstream services start seeing connection refused errors. Their p99 latency climbs as retries back up. Their SLO burn rate alert fires. A dependent service starts failing healthchecks. The healthcheck alert fires. The ingress load balancer logs 5xx errors. The error rate alert fires.
One root cause. Eleven alerts. Your on-call engineer wakes up to eleven separate PagerDuty pages — or worse, three different on-call engineers each wake up to four pages, all investigating the same incident without knowing the others are active.
The individual alerts are not wrong. Each one is accurately detecting a real signal. The problem is not that your alerts are too sensitive — the problem is that your alerting system treats each signal as an independent event with no causal context. It has no model of your service topology. It doesn't know that the ingress 5xx and the payment service OOMKill happened 90 seconds apart in the same namespace. It just fires.
Why teams respond to storms by raising thresholds
The standard playbook when alert volume overwhelms a team is to reduce noise through threshold tuning. Set the error rate alert to fire at 5% instead of 1%. Increase the p99 latency threshold from 500ms to 800ms. Add a longer evaluation window so brief spikes don't trigger. The underlying logic is: if the alerts are too noisy, make them less sensitive.
This approach trades false positives for false negatives. You stop getting paged for transient blips, but you also start missing the early stages of real incidents. The first 5 minutes of an incident — where the error signal is still below your new 5% threshold — is often when intervention costs least. By the time your threshold fires, the scope has expanded.
We're not saying threshold tuning is bad. Thoughtful SLO-based alerting with burn rate calculations is genuinely better than raw metric thresholds. But threshold tuning addresses the symptom — volume — not the structural cause, which is that unrelated-looking alerts from one incident are counted as independent events.
The correlation gap in modern observability stacks
Modern observability tooling is excellent at collection. Prometheus scrapes everything. Distributed tracing with OpenTelemetry gives you request-level visibility. Kubernetes events capture every pod lifecycle transition. The signal fidelity is high — often high enough that you can reconstruct any incident from first principles if you know where to look.
What the collection layer doesn't do is connect the signals. When a Prometheus alert fires and a Kubernetes OOMKill event fires 45 seconds later in the same namespace, your alerting stack treats these as two independent notifications. They arrive in Slack as two separate messages. They create two separate PagerDuty incidents. Two engineers might acknowledge them independently and start two parallel investigation threads.
Consider a specific scenario: a fintech platform running 120 microservices. Their fraud detection service has a correlation job that runs every 15 minutes. A recent Helm chart upgrade silently changed the default JVM heap size for that service. Every 15 minutes, the correlation job runs, hits the new heap ceiling, the pod OOMKills, and restarts. The restart causes a brief connection drop to the upstream API gateway. The gateway logs a spike in upstream errors. Simultaneously, the downstream reporting service sees null responses and throws exceptions.
In this scenario, the team's alerting stack fires six times every 15 minutes: OOMKill event, pod restart alert, gateway upstream error rate, reporting service exception rate, connection pool error, and a general availability SLO burn. Twenty-four alerts per hour. The on-call engineer on first rotation spends the first 20 minutes ruling out an attack vector because the gateway alert looks like an external traffic issue. Nobody ties the OOMKill to the Helm change until someone with context pulls the deploy log.
What event correlation actually solves
Event correlation is not just alert deduplication. Deduplication collapses identical or near-identical alerts into one notification — useful, but it only solves the case where the same alert fires multiple times. Correlation addresses the harder problem: connecting alerts that are different in type but common in cause.
Effective correlation requires three things. First, a temporal window: signals that occur within a configurable time band (2 minutes, 5 minutes, 15 minutes depending on your service response latency) are candidates for correlation. Second, a topology model: the correlation engine needs to know that service A depends on service B, that they share a namespace, that pod X belongs to deployment Y. Without topology, temporal proximity is just coincidence detection. Third, a change stream: config changes, deploys, Helm upgrades, Terraform applies. A latency spike that occurs 90 seconds after a Helm upgrade is a fundamentally different event from a latency spike with no recent change context.
When these three inputs are available, the calculus changes. Instead of forty-three independent alerts, your on-call engineer receives one correlated incident: "Pod OOMKill + gateway error rate + reporting exceptions, all services in namespace payments, window 14:45–15:00 UTC. Preceding change: Helm chart upgrade fraud-detection v2.3.1 → v2.3.2, 14:41 UTC."
That's the information they needed at the start. Not forty-three separate signals to manually cluster during an active incident at 3am.
The SLO-alerting + correlation pairing
SLO-based alerting with burn rate calculations (the model Google SRE popularized, where you alert on error budget consumption rate rather than raw metric thresholds) is a significant improvement over threshold alerting. It reduces noise by nature — short spikes don't consume enough error budget to trigger alerts. But SLO alerting alone still fires per-SLO, per-service. A correlated incident can still generate multiple SLO burn alerts across dependent services.
The right architecture pairs SLO-based alerting with a correlation layer upstream of your notification channel. SLO alerts provide high-quality, low-noise signals. The correlation layer groups them by topology and change context before they reach your on-call queue. You get the precision of SLO alerting without the fan-out problem when one incident touches multiple services.
Measuring improvement: MTTD and alert-to-incident ratio
If you're diagnosing your own alert storm problem, two metrics matter most. Mean Time to Detect (MTTD) — specifically, the time from incident start to the first engineer having actionable context, not just a page. And alert-to-incident ratio: how many individual alert firings correspond to each true incident. In healthy alerting systems, this ratio should be close to 1. In untuned systems with no correlation, ratios of 8–15 are common. We've seen teams with ratios above 30.
Reducing alert-to-incident ratio is the concrete outcome of correlation. MTTD improvement follows naturally: when the first notification contains correlated context rather than a single symptom signal, engineers spend less of their incident response time in the "what is actually happening" phase and more in the "what do I do about it" phase. That's the shift that actually reduces downtime.
Alert storms are solvable. Not by making your engineers more tolerant of noise, and not by blunting your alerting sensitivity. They're solvable by building an event model that treats related signals as what they are: symptoms of a shared cause.
