The simple version
Osiris ranked checkout drop-offs by how likely they were to book soon. Technically, it worked: the highest-scored decile converted at about 3.16 times the overall baseline, the pipeline ran in production and the ranking survived several iterations.
The business then used those bands for messages and discounts. Controlled tests showed that contacted users were often no better—and sometimes worse—than comparable users left alone. The model had answered 'who will book?' while the decision required 'who will book because we intervene?' I had taken the requester's problem statement at face value instead of reformulating the problem myself. That is why I consider Osiris my best failure.
This article is intentionally public-safe. It explains the engineering decisions and lessons without exposing customer data, proprietary prompts, internal identifiers, operational commands or confidential decision thresholds.
- BEHAVIOURBuild a point-in-time feature snapshot
- SCOREEstimate near-term booking propensity
- RANKConcentrate likely bookers into intent bands
- INTERVENEApply outreach under randomised control
- LEARNSeparate predictive lift from causal lift
Data: point-in-time behaviour and identity
The first iteration deliberately used a small feature surface: counts and durations around search and checkout, geography and stable identifiers. The label came from future bookings, and every feature had to be available at scoring time. This avoided leakage from actions that occurred after the checkout event.
Identity was not trivial. A user could appear on more than one device; a device could be shared; some activity had no signed-in user. I defined precedence and retention rules so the training set and production publisher described the same entity rather than quietly changing grain between notebook and service.
Iteration one: establish a serious classical baseline
I compared logistic regression, decision trees, random forests and gradient boosting on the same point-in-time frame. XGBoost gave the strongest practical ranking and became the first production version. The service converted the notebook into timestamp-driven queries, reproducible preprocessing and scheduled publication of both probability and intent band.
When the original environment could not support current dependencies, I packaged the job rather than forcing a fragile host upgrade. The production schedule refreshed recent behaviour more than once a day, which made time boundaries and rerun safety part of the design.
More than 100 runs: preprocessing, features and model families
The second phase became a systematic classical-ML programme with more than 100 training and evaluation runs. I varied behavioural lookbacks across short and long windows, time periods across one week to several months, feature subsets, interaction ratios, city encodings and the treatment of app clicks and past bookings. I tested log transforms, robust scaling, Yeo-Johnson power transforms, feature reduction, outlier removal and city-tier aggregation.
The model sweep included logistic regression with multiple solvers, random forests with randomised search, XGBoost, LightGBM and Gaussian Naive Bayes. Optimising ROC-AUC produced models with impressive-looking precision and almost no recall; optimising F1 moved the trade-off but not the frontier. Outlier removal and broad city grouping sometimes made the model worse. The negative results were recorded rather than discarded because they mapped the limits of the feature space.
A separate explanatory diagnostic suggested that the existing features accounted for only a small share of variation in human booking behaviour. Permutation tests still showed a weak but genuine signal. The conclusion was not 'try a more exotic classifier'. It was to add recency, velocity and sequence features: hours since last search, search acceleration, checkout-to-search ratio, number and order of cars visited, time since the last booking and the progression through high-intent screens.
What made the workflow production-grade
Class imbalance made accuracy almost meaningless. Evaluation combined ROC-AUC with precision-recall behaviour, calibration, KS separation and decile lift. Thresholds were treated as operating policies, not properties of the model; a top-k campaign can be valuable even when a global 0.5 threshold is not. Temporal validation trained on earlier behaviour and tested on later periods so nearby sessions could not make a random split look safer than production.
I also reconciled notebook and service semantics: event cut-offs, lookback windows, user-device precedence, null identities, duplicate snapshots and label maturity. A sophisticated estimator over a leaking or grain-shifting dataset is not state of the art. The state of the art begins with a dataset that represents what the model will know when it acts.
Evaluation: rank the population, then validate through time
A fixed classification threshold understated the system’s operational value. Decile analysis showed that the highest-scored tenth converted at about 3.16 times the overall baseline, and the score separated positive and negative populations meaningfully. For a constrained outreach channel, that concentration can matter more than maximising one thresholded metric.
I therefore evaluated AUC, precision and recall alongside lift, KS separation, calibration and temporal stability. The correct validation split followed time: train on the past and test on a later period. Random splits can make behavioural models look stable by allowing nearby patterns to leak across the boundary.
Experiment: the intervention did not inherit the model’s quality
The initial business hypothesis was that medium-intent users might respond most to a discount, while high-intent users would book anyway and low-intent users would remain hard to move. Randomised outreach comparisons did not support that simple story. In important cuts, untreated users performed as well as or better than contacted users.
A later first-time-user version removed history unavailable to new customers and supported both conventional messaging and a multi-arm test involving automated calling, messaging and control. Again, the intervention did not create consistent conversion improvement. The experiments were more informative than a flattering offline metric: they showed that broad discounting could optimise neither conversion nor cost efficiently.
Why this is my best failure
Osiris produced a stable production ranker, clear top-of-list concentration and disciplined experimentation. It also showed that discounts were not creating the expected incremental bookings. The project paused not because the model could not rank, but because the intervention and business assumptions became the limiting system.
My mistake was upstream of modelling. I accepted 'find users likely to book' because that was the request, when I should have asked what action the score would trigger and what counterfactual decision mattered. The correct long-term formulation was uplift: estimate who changes because of a message, call or discount, using treatment assignment and mature outcomes—not only future-booking propensity.
That lesson is worth more to me than another lift point. Osiris taught me that a model can be technically strong, deployed and measurably predictive while the product remains wrong. The best data scientist in the room is sometimes the person who refuses the supplied target and rewrites the question.