alerting

Threshold alerts fire all weekend and miss the real outage Monday. The signal is decomposing seasonality first, choosing the right detector, and tuning for alert fatigue. Here is the answer that survives production.

Models fail silently; the question is whether you would know. The signal is monitoring the right layers (operational, data, prediction, outcome) and alerting on drift without drowning in false pages. Here is the system and the metrics that matter.

Anomaly detection sounds easy until seasonality, cold start, and alert fatigue hit. The signal is modeling the expected baseline (including weekly and daily cycles), choosing unsupervised methods for scarce labels, and tuning to avoid drowning users in false alarms. Here is the design.

An alert that fires constantly gets muted, and a model that fails with no alert is worse. Good ML alerting is a design problem with the same principles as SRE, plus ML-specific twists. Here is how to get it right.