The simple question Watchdog answers
Every morning, Watchdog asks: did conversion, bookings, GBV or a pricing component move enough that a person should investigate? Then it asks the more important follow-up: is the movement real, or did we compare the wrong days, use too little volume or read data before an upstream job finished?
Pricing metrics have weekday seasonality, changing city and duration mix, sparse cohorts, refunds and sources that complete at different times. One flat threshold can flag a perfectly normal Monday and miss a slow decline that lasts a week. Watchdog turned the judgement previously performed in a workbook into a replayable daily service.
This article is intentionally public-safe. It explains the engineering decisions and lessons without exposing customer data, proprietary prompts, internal identifiers, operational commands or confidential decision thresholds.
- READINESSConfirm grain, coverage and source state
- BASELINESelect comparable historical periods
- SCOREEvaluate point movement and persistence
- STATECompare only with an earlier successful run
- PUBLISHArchive the run, then expose the snapshot
One example before the machinery
Suppose a city’s conversion falls 12% yesterday. That number alone is not an alert. If yesterday was a Monday, Watchdog compares it with earlier Mondays at the same geography and duration grain, verifies search and booking volume, checks whether GBV and price moved, and confirms that the data sources are complete. A large single-day residual becomes a point anomaly.
Now suppose conversion is only 3% below expectation—not dramatic on any one day—but the residual has been negative for several comparable observations and the slope keeps worsening. That becomes a trend anomaly. Point detection catches shocks; trend detection catches drift. The receiver sees which kind occurred and the business volume attached to it.
Data contract: grain and eligibility first
Every evaluated row carried date, geography, metric identity and processing run. Before anomaly scoring, the service checked source readiness, expected coverage, valid geography and sufficient comparable history. Rows that failed these checks became data-quality states rather than business anomalies.
Revenue and conversion were evaluated independently because they have different denominators and failure modes. Negative aggregate values were preserved in reporting and impact calculations; they were constrained only inside statistical operations that could not accept them. Convenient preprocessing was not allowed to erase a real financial event.
Method: comparable history, points and trends
Expected behaviour came from comparable weekdays rather than an undifferentiated trailing mean. I evaluated robust baselines, dispersion bands, residual magnitude, consecutive misses and short-window slope. The released logic combined interpretable same-weekday expectation with separate point and persistence components instead of hiding everything inside one anomaly score.
Volume gates prevented a tiny cohort from receiving the same treatment as a broad market movement. Conversion, bookings, searches, GBV and price components were evaluated on their own valid denominators, then read together for diagnosis. A statistically unusual percentage on ten searches could not outrank a moderate deterioration across a major market merely because its z-score was larger.
State transitions referenced only the latest earlier successful run. A failed or partial execution could not convert yesterday’s red state into green or become the baseline for tomorrow. Historical runs remained append-only; the latest published snapshot changed only after successful evaluation.
Evaluation: replay the past before trusting tomorrow
I replayed the service across a long historical window and compared its outputs with the analytical workbook and known business periods. Regression cases covered sparse segments, negative values, missing history, state lookup and persistence. The goal was not to reproduce every historical alert exactly, but to explain every difference.
I also reviewed alert concentration: how many markets were red, whether one upstream problem created a flood, and whether severity tracked business impact. These checks catch a system that is technically functioning but operationally unusable.
Result and impact
Watchdog converted a useful analysis into a deterministic, stateful daily evaluator for business metrics, point anomalies, persistent trends and data quality. It gave Pigeon a trustworthy run boundary and ranked snapshot instead of a table that might represent partial work.
One limitation remains important: when an upstream system has no authoritative completion signal, structural checks can only infer readiness. The service documents that boundary. Honest observability includes what the system cannot prove.