Osiris: prediction is not intervention

Iteration one: find a real signal

Osiris began with a narrow question: among checkout drop-offs, who is more likely to book soon? The first version intentionally used a small behavioural surface—recent searches, checkout activity, duration and geography—plus careful user/device identity rules.

I compared logistic regression, trees, random forests and gradient boosting. A non-aggregated XGBoost pipeline became the first production version. It refreshed recent activity more than once a day and published a probability plus a low, medium or high intent tag for downstream use.

This is a public-safe account of the work. Internal identifiers, proprietary thresholds, prompts, infrastructure details and operational commands are intentionally omitted.

Iteration two: the modelling rabbit hole

The second iteration added broader app activity, past booking behaviour and multiple lookback windows. Recall moved, precision resisted. I tried robust and power transformations, feature reduction, engineered ratios, several model families, class-imbalance strategies, city tiers and outlier removal.

Most changes produced marginal or worse results. That was useful evidence. The limitation was not a missing hyperparameter; the available features only captured a small part of real human behaviour. Iteration two became modelling-heavy when it should have become feature-heavy sooner.

Ranking changed the interpretation

A single classification threshold made the model look less useful than it was. Decile analysis showed that the highest-scored 10% converted at 3.16 times the overall baseline. The KS statistic was about 0.39, and permutation tests suggested the weak signal was real rather than an artefact.

For an outreach system, ranking can be the product. A constrained team may care more about finding a concentrated top segment than labelling every user correctly. Lift, temporal validation and stability therefore mattered alongside precision, recall and AUC.

Then the experiment disagreed

The intended action was a targeted incentive. The hypothesis was sensible: high-intent users might not need help, medium-intent users might respond to a nudge, and low-intent users might remain insensitive. Controlled campaigns did not support that clean story. In several comparisons, untreated users performed as well as or better than contacted users.

That result did not invalidate the model. It separated two causal questions that are often confused: who will book, and who will book because of this intervention? Osiris answered the first. The campaign needed uplift modelling—or a different intervention—to answer the second.

The resurrection and the same lesson

A later first-time-user version removed features unavailable to new customers and shipped as a deliberately quick baseline. Its discrimination remained usable despite lower precision in a heavily imbalanced problem. It supported both conventional outreach tests and a three-way comparison involving automated calling, messaging and control.

Again, the control was difficult to beat consistently. The project paused, not because the pipeline failed, but because the organisation was not ready to change the baseline intervention. That is the most honest ending: a solid production model can reveal that the next bottleneck is product strategy, not model quality.