The problem: every service failed in its own language
Production data-science and data-engineering services fail in different ways. A cron may never start, a pod may repeatedly restart, credentials may lose access, a table may stop refreshing, a write may become partial or a service may finish successfully while producing no usable rows. When every repository implements its own notification, silence and severity mean different things.
DS-Alerts was the central operational safety net for that estate. It was not a pricing-anomaly model—that is Watchdog. Its job was to make execution health, infrastructure readiness, dependency failures and database integrity visible in one consistent format, with enough context to reach the owner who could act.
This article is intentionally public-safe. It explains the engineering decisions and lessons without exposing customer data, proprietary prompts, internal identifiers, operational commands or confidential decision thresholds.
- IDENTIFYName service, environment, run and owner
- CLASSIFYExecution, infrastructure, dependency or data
- CONTEXTAttach symptom, timing and safe diagnostics
- DEDUPLICATEGroup one continuing incident
- ROUTESend the right severity to the right team
A common event instead of common prose
Every signal was normalised into a small event contract: service and component, environment, execution or correlation identity, observed time, severity, failure category, owner, concise message and safe diagnostic context. Producers could keep domain-specific checks while consumers received one predictable shape.
The distinction between an event and an incident mattered. Five retries of the same database outage are five observations but usually one incident. A stable fingerprint grouped repeats, maintained first- and last-seen time and counted occurrences without flooding the channel. Recovery could close the same incident rather than producing an unrelated green message.
Four families of checks
Execution checks answered whether a scheduled job started, completed and stayed within its expected window. Infrastructure checks covered process or pod readiness, restarts and the ability to serve. Dependency checks verified that databases, warehouses, model providers and other required services were reachable with the expected permissions. Data-integrity checks examined freshness, row presence and contract-level invariants after execution.
These checks were deliberately layered. A failed job should not also emit dozens of missing-output symptoms as independent root causes. The alert carried the most useful known level and included downstream effects as context. Unknown remained a valid category when the service could observe a symptom but not prove its cause.
event = normalise(service, environment, run, check, severity, context)
incident = fingerprint(event)
if new_or_escalated(incident):
route(owner, event)
else:
increment_occurrence(incident)
Severity and routing are operational decisions
Severity was based on consequence and urgency, not how alarming an exception string looked. A transient non-critical retry and a silent failure in a customer-facing production service deserved different routes. Environment, blast radius, persistence and the availability of a fallback informed escalation.
Messages included what failed, when it was last healthy, whether retry or fallback was active, the safe first diagnostic and the owning service. They excluded credentials, raw customer data and oversized stack traces. The goal was to shorten time to useful action, not copy an entire log into a notification.
Evaluation: a delivered message is not automatically useful
I tested producer contract validation, severity mapping, fingerprints, repeat suppression, escalation, recovery and routing failure. Scenario tests covered a job that never started, a provider outage, stale output after apparent success, partial writes, repeated retries and an alerting dependency that was itself unavailable.
Operational quality was measured through volume, duplicate suppression, unresolved duration, acknowledgement and sampled actionability. False positives train engineers to ignore the channel; false negatives make the framework decorative. The system needed both heartbeat evidence and independent monitoring so the absence of an alert did not have to prove that alerting itself was alive.
Impact
DS-Alerts gave production DS–DE services one reliability language without forcing them into one codebase. Teams could add domain checks while retaining common ownership, routing, deduplication and incident semantics.
The quality of an operational alert system is not how many errors it forwards. It is whether the right owner learns about one real incident early enough to act, with enough context to begin, and without being trained to ignore the next message.