Nemo: the five-month one-day task

The deceptively small brief

On my third day as an intern, my manager asked me to study review summarisation. My first reaction was embarrassingly simple: this sounds like one prompt and one model call. I was right about the model call and wrong about almost everything surrounding it.

Nemo eventually took roughly five months from investigation to a production system. It also marked my transition from treating data science as modelling to treating it as a service with users, contracts, failure modes and ownership.

This is a public-safe account of the work. Internal identifiers, proprietary thresholds, prompts, infrastructure details and operational commands are intentionally omitted.

First prove that it should exist

Before writing the summariser, I analysed how users moved from search to checkout, from checkout into reviews, and onward toward payment. Review readers were a smaller cohort, but their path-level conversion was materially stronger. That did not prove reviews caused conversion—intent bias was an obvious alternative—but it established a useful hypothesis: social proof mattered and the interface made it expensive to consume.

I then read and profiled a large review sample. The data ranged from useful descriptions to two-word comments such as ‘good car’. Some reviews were too long to scan, many were too thin to summarise, and newer cars often lacked enough evidence. The correct input policy mattered as much as the prompt.

An eligibility rule, not a text dump

A car qualified only when it had at least three reviews containing more than three words. Once a car qualified, all of its valid review context was retained—not only the three reviews that crossed the threshold. This filtered the worst noise while avoiding a summary built from a misleading fragment.

The threshold came from examining the actual distribution rather than choosing a round number. About 56% of cars qualified at the time. Cars without enough evidence returned no generated story; absence was safer than confident filler.

qualifies = count(review.word_count > 3) >= 3
if qualifies:
    context = join(all_reviews, separator='---')

Generation is only the first draft

The offline pipeline generated a structured summary and ran two independent evaluations in parallel. A custom guardrail checked product-specific style and output rules. A groundedness evaluation looked for claims unsupported by the source reviews. They answered different questions: is this acceptable, and is this faithful?

A failed output was sent to a re-summariser together with the source reviews and evaluator feedback. The loop had a strict attempt limit. Passed summaries moved into current and historical stores; failed attempts were retained for analysis. If normal persistence failed, a fallback file preserved enough context for recovery rather than silently dropping work.

Two systems sharing one name

Nemo was deliberately split. Scheduled generation jobs handled expensive work: a daily incremental path for cars with changed reviews and a controlled full refresh for exceptional cases. Work was parallelised across a small number of cars so throughput improved without turning downstream services into an accidental load test.

The serving side stayed boring. One API returned the latest accepted summary for a car; another recorded user feedback with duplicate protection. The checkout path never waited for an LLM. It read a prepared result and returned an empty object when none was available or when a kill switch was active.

What I actually learned

The headline result—more than 200K daily requests at sub-150ms p95—is mostly a consequence of moving uncertainty out of the request path. The deeper lesson was that an LLM feature is a chain of evidence policies, evaluation, storage, observability, contracts and graceful absence.

Nemo began as ‘easy, one day, LLM calls’. It ended as the project that taught me why production data science is largely the work around the model—and why that work is the interesting part.