Static thresholds produce 15–20% false positives in our infrastructure. Baseline learning reduces false positives by 30–40% versus well-tuned thresholds. Use a layered approach: static thresholds for hard limits (disk, memory exhaustion); baseline learning for dynamic metrics; root-cause correlation as a hint, not a verdict.
Most monitoring systems compare metrics against static thresholds. CPU above 80%? Alert. Request latency over 500ms? Alert. We found that threshold configuration alone can swing alert volume by 40–50% between strict and moderate setups—often more than the underlying signal. A 40% CPU spike at 3 AM differs from the same value during a scheduled batch job; static thresholds treat both identically.
Evidence: threshold calibration impact
False positive rates reached 15–20% with static thresholds in our cloud workload monitoring. Alert volume swung 40–50% between strict and moderate configurations.
We run infrastructure monitoring for cloud workloads. While calibrating alert thresholds, we noticed that alert volume didn't match underlying system behavior—15–20% of alerts fired for normal operational variation, unrelated to actual incidents.
The discrepancy came down to context. Our initial implementation treated each metric in isolation: we set a floor (e.g., "alert when CPU > 80%") and fired whenever the threshold was breached. But infrastructure metrics are inherently noisy. Transient spikes—a garbage collection pause, a brief network blip, a cold start—can cross a threshold momentarily without indicating a real problem. When the threshold is the only signal, there's no way to distinguish "something is wrong" from "this is normal variation we haven't seen before."
This raised a larger question: how much does threshold configuration impact what you actually measure?
To quantify the effect, we ran a baseline comparison across several threshold configurations—from strict (alert on any breach) to lenient (require sustained breach over multiple samples). Everything else stayed constant: same metrics, same systems, same time window.
In our experiments, alert volume dropped significantly as we relaxed the strictness. This was primarily driven by false positive rates declining at each step. The drop between strict and moderate configurations was on the order of 40–50%. With more headroom—e.g., requiring a metric to stay above threshold for 2–3 consecutive samples—fewer transient spikes triggered alerts.
However, we also observed that at the lenient end, we started missing real incidents. The extra filtering that reduced noise also delayed detection of genuine anomalies. There's a tradeoff: tight thresholds catch everything but train engineers to ignore alerts; loose thresholds reduce noise but can miss the signal.
What baseline learning changes
Baseline learning reduces false positives by 30–40% versus well-tuned static thresholds, with comparable detection latency. Requires 2–4 weeks of historical data.
ML-based anomaly detection learns normal behavior over time. Instead of "is this metric above threshold?" it asks "is this metric unusual given current conditions?" A baseline can learn that Tuesday afternoons see higher CPU during batch jobs; 3 AM spikes differ from afternoon spikes. The same absolute value can be normal in one context and anomalous in another.
The exact improvement depends on workload—systems with regular patterns benefit more than chaotic traffic. Baseline learning struggles with novel behavior (new workloads) and adds operational complexity (monitor baselines for drift).
Root cause correlation: promise and limitations
Automated root-cause correlation reduced mean time to identify cause by ~50% versus manual log diving—when it worked. Failure mode: wrong cause when multiple changes correlate (deployment + traffic spike + dependency outage). Use as ranking mechanism, not verdict.
Correlating signals across metrics, logs, and traces surfaces likely causes faster than manual investigation. Rank candidates by correlation strength; surface for engineers to verify rather than auto-escalating.
The Meterra approach
We've landed on a layered approach that addresses these tradeoffs without over-promising. First, we keep static thresholds for hard limits—disk, memory exhaustion, things that are always bad regardless of context. Second, we use baseline learning for everything else, but with explicit guardrails: a minimum training window before baselines go live, and a fallback to conservative thresholds when the baseline hasn't seen enough data. Third, we treat root cause correlation as a hint, not a verdict—surfacing likely candidates for engineers to verify rather than auto-escalating on correlation alone.
The key is treating each layer as a tool with known failure modes. We don't expect baseline learning to handle novel workloads; we don't expect correlation to be right every time. The solution is the combination: multiple signals, each imperfect, that together narrow the problem space enough for humans to act quickly.
What we recommend
The ideal scenario is to have both: static thresholds for known critical limits (e.g., disk at 95% is always bad) and baseline learning for everything else. However, this may not always be practical—baseline systems require more setup and maintenance.
Given how threshold-based alerting actually behaves, we recommend:
1. Separate transient from sustained.
Require a metric to breach threshold for N consecutive samples (e.g., 2–3) before alerting. This cuts spurious alerts from momentary spikes without losing real signal.
2. Use time-of-day awareness where possible.
If your traffic is predictable, consider different thresholds for peak vs. off-peak. A simple improvement over a single global threshold.
3. Treat baseline learning as an experiment.
Run it in parallel with existing thresholds for a few weeks before switching. Compare alert volume and incident detection rates. The right choice depends on your workload.
4. Document your configuration.
Threshold values, baseline training windows, and correlation logic should be versioned and reviewable. When an alert fires, engineers need to understand why.
At the margins, the difference between "we have good monitoring" and "we have alert fatigue" often comes down to configuration choices that aren't obvious from the outside. Small changes in threshold strictness or baseline sensitivity can swing results by several percentage points. Without published setup details, it's hard to tell whether a monitoring system is well-tuned or just noisy.
A few extra alerts might signal a real problem—or they might just be a threshold that's too tight for your workload.