A single global latency threshold is easy to configure and usually wrong. Services have different normal response times, network paths and user expectations. The result is either constant noise or thresholds so high that they detect nothing useful.
Choose the source of truth
Use a fixed threshold when a contract, product requirement or known timeout defines acceptable latency. Use an adaptive baseline when “unusual for this service” matters more than one universal number.
A good precedence rule is:
- explicit per-monitor threshold;
- learned per-monitor baseline;
- global fallback while the baseline is learning.
Require persistence
One slow check is evidence, not an incident. Confirm degradation across consecutive failures and a sustained-alert window. Apply the same care to recovery: one fast check should not resolve a condition that is still oscillating.
Flap detection and notification cooldowns solve a related problem. They should suppress repeated notifications without deleting the underlying history.
Make the threshold explainable
An operator should be able to answer: which threshold fired, how it was calculated, how many samples supported it and what the recent P50/P95 looked like.
Warden’s adaptive latency baseline uses successful checks from a configurable window, requires a minimum sample count and persists the result across restarts. By default, degraded begins above max(P95 × 1.5, P95 + 100 ms) after confirmation.
Review after environmental changes
Moving the monitoring host, adding a VPN or changing regions alters the measured path. Relearn the baseline deliberately. Raising every threshold to silence the transition can hide the next real regression.