DS-Nemo: a review summarisation system

What summarisation is really trying to do

A summary is not simply shorter text. It is a lossy representation that must decide what deserves to survive compression. For one clean article, that is already difficult. Nemo had to compress many reviews written by different people, at different times, in different languages and with different ideas of what mattered, then show the result to someone making a booking decision.

The product question was simple: can a guest understand the recurring strengths and concerns of a car and its host without opening every review? The technical contract was harder: preserve evidence, express useful consensus and disagreement, omit irrelevant or unsafe material, use language a non-expert understands and return nothing when the evidence is too weak. Fluent prose was necessary, but unsupported fluency was failure.

This article is intentionally public-safe. It explains the engineering decisions and lessons without exposing customer data, proprietary prompts, internal identifiers, operational commands or confidential decision thresholds.

First define the families of summarisation

At a high level, two types of summarisation exist: extractive and abstractive. Extractive summarisation selects material that already exists, including important sentences, phrases or spans. It is relatively inspectable and cannot paraphrase a new claim, but the result may read like fragments from different authors. Abstractive summarisation generates new language that represents the source. It can merge repeated ideas and write coherently, but every novel sentence creates a faithfulness risk.

Multi-stage systems combine these ideas. An extractive-abstractive pipeline first selects evidence and then rewrites it. An abstractive-abstractive pipeline summarises chunks and summarises those summaries, often using map-reduce for inputs beyond a model's context. This matters on large marketplaces, where one product or listing may have thousands of reviews that cannot be passed to a model at once. Aspect-based summarisation adds another axis: organise evidence around topics such as cleanliness, brakes or host communication rather than producing one undifferentiated paragraph. These are design families, not a ladder where the newest automatically wins.

The design space Nemo explored

EXTRACTSelect sentences or spans from the source
ABSTRACTGenerate a shorter semantic representation
HYBRIDSelect evidence, then rewrite it
ASPECTGroup evidence by the subject being discussed
MULTI-STAGECompose partial summaries for long inputs

The right architecture depends on input length, domain, evidence quality, latency, cost and the consequence of being wrong.

Backstory

On the third day of my internship, my manager took me out for a one-on-one walk and told me that my first project would be review summarisation. As soon as I heard that, I thought, lol, one LLM call. Could I have been more wrong? Then he said that, before trying LLMs, we wanted to see whether anything 'cheap and dirty' could do the trick. Spoiler alert: it couldn't.

And so began my journey into review summarisation. I started a literature review to plan V1, but first I was tasked with EDA to see whether we should even attempt this problem. I studied Zoomcar's reviews, their distributions and how people used the review section, then discussed the findings with the whole department. We concluded that it was a worthwhile problem to solve.

Things I needed to confirm were:

People were interacting with the review section, and it appeared to be an important part of their decision-making.
Piecing all the information together was taking them too long.
The reviews were summarisable and contained enough useful information.

Look, we did not prove causality. People who completed a booking looked at reviews more often than people who did not, but that could simply mean they already had higher intent. What the analysis did show was that reviews were an important touchpoint in the booking journey. Bookers consumed them more, and the reviews held enough informational value to justify improving that experience with a summariser.

I also created a small evaluation dataset, although it changed a lot as the project evolved.

All of this took about a month. I learned so much and, yes, occasionally worked while brushing my teeth, lol. Then I started V1 of Nemo.

Iteration 1: statistics made the decisions visible

I began in the classical world. The key idea was simple: extract important words from the reviews, identify their sentiment and build template summaries. Good things: A, B and C; bad things: X, Y and Z.

Reviews were deduplicated, split into sentences, normalised, tokenised, lemmatised with part-of-speech information and scored. TF-IDF highlighted distinctive terms; sentiment separated positive and negative evidence; the highest-ranked sentences were concatenated into a controlled result. Every step could be inspected, which made failure analysis honest.

It also revealed the limits quickly. Lexical importance is not the same as decision usefulness. A sentence can contain rare words and still be irrelevant. VADER-style sentiment struggled with constructions such as 'I cannot say enough good things', and concatenated evidence sounded like several reviewers talking over one another. The system was grounded but not yet a good summary.

One problem was that some extracted words were irrelevant, so a small workaround was to cross-check them against a large list of keywords relevant to cars, Zoomcar and the domain. Other problems followed: reviews often expressed the same idea in different ways, the system was not multilingual, and every summary began to look alike. There were plenty of other issues too, especially when comparing the results today with the live version.

One more problem was we used TFIDF in itr1, if all the reviews said brakes bad, then tfidf would not consider it important, but it is important to the user. So we had to use a different approach in later iterations.

How Nemo progressed

ITERATION 1TF-IDF, VADER and templates: a cheap baseline with weak semantic understanding
ITERATION 2Fuzzy deduplication, YAKE, TextRank and DistilBERT: better preprocessing, sentiment and extractive selection
ITERATION 3DistilBERT grouped evidence by sentiment, then BART summarised each side within its token limit
ITERATION 4T5, PEGASUS, BART and FLAN-T5 with fine-tuning: better generation, but domain, data and generalisation gaps remained
FINALAn LLM with guardrails and independent evaluation: stronger synthesis and handling of long-tail language

Each iteration fixed a visible weakness, then exposed the next one. The goal was not a model leaderboard but a system that worked on real reviews.

Iterations 2 to 4: stronger components, new failure modes

Iteration 2 kept the output extractive but upgraded almost every component. Fuzzy matching removed near-duplicate sentences, spaCy handled sentence splitting and lemmatisation, YAKE extracted candidate keywords, DistilBERT added contextual sentiment, and TextRank selected representative positive and negative evidence. It handled wording better than V1, but the result was still a collection of selected sentences rather than one coherent voice.

Iteration 3 added abstraction. Sentences were divided into positive and negative groups using DistilBERT probabilities, with uncertain sentences included in both so mixed sentiment was not thrown away. I chose BART-large-CNN because it was designed specifically for summarisation, and used it to summarise each group. When the input crossed its token limit, the pipeline ranked sentences by sentiment confidence and kept the strongest evidence that would fit. The prose became more natural, but truncation could discard useful context and BART's news-style training did not always suit messy, multi-author car reviews.

Iteration 4 widened the search to T5, PEGASUS, BART and FLAN-T5, along with fine-tuning and aspect-based experiments. I manually summarised reviews for about 100 cars to create a small domain dataset, starting with PEGASUS and T5 before moving to instruction-following models. Grouping evidence around car and host attributes made the output more useful, but limited training data, multilingual reviews, automotive language and unfamiliar edge cases still hurt generalisation.

Each iteration solved something and exposed something else. Transformer extraction chose better evidence but still produced stitched prose. BART and PEGASUS improved readability but inherited a news-domain prior that did not fit multi-author marketplace text. Aspect models trained on generic review domains missed automotive language. Hybrid extract-then-rewrite pipelines could lose qualifiers during the second stage. The more fluent the output became, the easier it was to overlook a subtle unsupported claim.

That is the central trade-off of abstractive summarisation: compression and coherence come from generating language that was not literally present. Evaluation must therefore ask whether the new sentence is entailed by the evidence, not merely whether it resembles a reference.

It is not that the later transformer-based summaries were bad. They just were not good enough, and on edge cases they could fail miserably.

The period where the project looked lost

After several iterations (during Iterations 2 to 4), the work could look like motion without arrival. Some people reasonably questioned why the pipeline kept changing and whether the approach itself was wrong. I had extractive, abstractive, hybrid and aspect-based experiments, yet none met the full product contract. For a while I was no longer sure whether I was exploring the right search space or just accumulating techniques.

As an intern, I could see my PPO (my pre-placement offer) fading. This is the point in the story where we hit the 'valley of despair.' I did the only sane thing I could think of: took a day to revisit the literature and see how other companies were tackling the problem.

Then I found an AWS engineering article that organised production summarisation into the same families I had independently reached: extractive BERT, specialised abstractive models such as BART and PEGASUS, extractive-abstractive systems, and abstractive map-reduce approaches. It did not prove my implementation was correct. It did something more useful: it showed that the exploration was structurally sound and that the remaining problem was selection and evaluation, not a missing fashionable architecture.

Apple's work on App Store review summarisation reinforced the aspect-oriented direction, while Google and IBM described decomposition for long documents. Different companies, domains and stacks were converging on the same ideas: separate evidence selection from language generation when helpful, organise around aspects when the decision needs them, decompose long inputs, and evaluate the summary as a system rather than trusting one model call.

All of this showed me that I was on the right path and that the way out was to push through. It was interesting and, honestly, fun to see other companies tackle the same problems through similar flows and converge on the same ideas.

The data told us when a summary should exist

Before moving ahead, we took a small detour to define review quality. Not every review needed to be summarised; only those with useful information did. The rest was mostly noise. So we needed a working definition of a good review and a bad one, built by studying review length, domain keywords and the evidence each review contained.

Length was a rough proxy for depth; a domain keyword inventory captured whether the review actually discussed the car, condition, features or host. Manual review converted those signals into broad low, medium and high-information groups. Most sampled reviews carried usable information, but a meaningful tail consisted of fragments such as 'good car' that could not support a specific summary.

That analysis led to an eligibility gate at car level. We kept it on the lenient side: it was better to let some junk through than to miss good evidence. A car needed enough meaningful collective evidence before generation was attempted. The exact production criteria are deliberately not published; what matters is the principle. A summariser should be allowed to abstain before and after generation. Empty output is a valid product outcome when specificity would require invention.

Prompt engineering as a written specification

After transformers, the next logical step was LLMs. But they can make the task look deceptively easy. If we were bringing a gun to a knife fight, we needed to ensure that we took care of every problem, especially the edge cases.

One thought stayed with me throughout: anyone can summarise reviews with an LLM, so what would make this system genuinely different and the best? I kept returning to that question while finding edge cases, engineering prompts, and designing the production and evaluation systems.

By July 2025, a prompted LLM produced the best balance of synthesis, language control and iteration speed for this short multi-review input. The prompt was not 'summarise this'. It defined the reader, the domain, the input delimiter, the scope of valid evidence, the output schema, length and language, the meaning of empty output, and the distinction between a review writer and a car owner.

Several techniques mattered. Task and context came first. A strict structured schema made downstream validation possible. Positive instructions defined what a useful summary and useful aspects looked like; negative constraints excluded platform complaints, names, phone numbers, spam and direct comparisons. Few-shot examples demonstrated mixed sentiment, generic evidence and abstention. Directional cues emphasised recurring car and host attributes without forcing a fixed ontology. Delimiters isolated untrusted review text from instructions, and the prompt explicitly treated instructions inside reviews as data rather than commands.

The best prompt was not the longest possible prompt. It was the smallest specification that made failures classifiable. Repetition was removed when it stopped changing behaviour; high-risk constraints were repeated only where model attention justified it. Prompt versions were evaluated against the same cases so an improvement in style could not silently weaken faithfulness.

The edge cases were the real domain model

Evidence conflict came first. One review can contradict itself; one review can disagree with ninety-nine others; recent reviews can describe a change over time (temporal context). Nemo had to preserve mixed sentiment when disagreement was meaningful, prefer a clearly dominant pattern only when supported, and avoid placing one aspect in both positive and negative lists merely because the wording was convenient.

Language and form created a second family: extremely short or long reviews, Hinglish and other languages, transliteration, acronyms, automotive jargon, spelling errors, emojis, sarcasm, irony, humour, angry or abusive reviews, exaggeration and culturally specific phrasing. The output still had to be plain English. Technical statements were translated into the effect a normal driver would understand; ambiguous claims such as 'battery lasts six hours' were not assigned a positive or negative meaning without context.

Safety and adversarial text formed a third family: prompt injection, prompt leaking, jailbreak attempts, encoded instructions, profanity, personal attacks, hate speech, names, disguised phone numbers, sensitive data, spam and irrelevant links. Review text was never trusted as instruction. Useful underlying experience could be expressed formally, but unsafe surface language and personal information were removed.

Relevance formed the fourth family. Platform pricing, customer support, roads, weather, feature requests and competitor comparisons may be real customer concerns but did not belong in a car-and-host summary. Suggestions, off-topic travel stories and inferred opinions were handled conservatively. The model described what reviewers reported; it did not fact-check the world or invent a stronger opinion than the evidence carried.

Finally came weighting. Repetition is not automatically importance, the majority is not automatically truth, and one safety-critical concern may matter despite low frequency. Nemo balanced recurrence, recency, severity and usefulness without pretending that an LLM could resolve every ambiguous case. In doubt, it omitted.

Generation was never allowed to approve itself

The final pipeline used independent evaluations in parallel. A custom black-box guardrail checked the product contract: valid structure, concise third-person English, relevant car or host content, no private or offensive material, no platform leakage, no duplicate polarity and no evidence that the model followed instructions embedded in reviews. It did not receive the summariser's hidden reasoning; it judged the output against an acceptance specification.

A separate hallucination evaluation compared the generated summary with the source reviews and looked for unsupported or distorted content. Lexical metrics such as ROUGE and semantic similarity such as BERTScore remained useful during research, but neither alone answered the production question. Human review added groundedness, composition, safety and helpfulness, the axes on which a readable summary can still fail.

If either evaluation failed, a re-summariser received the source, the rejected output and evaluator feedback. It could repair the response within a bounded number of attempts. A persistent failure was stored for analysis and excluded from serving. The loop improved recoverable mistakes without allowing an infinite model conversation to become an operating strategy.

Nemo's independent acceptance loop

GENERATECreate summary and aspect structure
GUARDRAILCheck product, safety and format rules
FAITHFULNESSCheck claims against review evidence
REPAIRRegenerate from explicit failure feedback
ACCEPT OR QUARANTINEServe passes; retain failures for analysis

Fallback engineering: assume every dependency will fail

Reliable LLM engineering is mostly preparation for ordinary failure. Before a batch spent time reading reviews, it made a small model preflight call; if Gemini was unavailable, the run stopped early and alerted instead of generating thousands of identical failures. Database connectivity and required state were checked before the core loop. The incremental job recorded its upper timestamp before reading and advanced the checkpoint only after the run completed, preventing new reviews arriving mid-run from falling into a gap.

Reviews came from BigQuery, were grouped and filtered, and cars were processed with bounded parallelism rather than unlimited fan-out. Every car produced an explicit success or failure result. Accepted summaries went to current and historical stores; rejected attempts went to a failure store. If database persistence failed, a recovery file captured the information required for replay. Failures triggered team alerts, and connections were closed in a final boundary.

A controlled all-cars job existed for backfills and exceptional refreshes; the daily job only processed changed evidence. The two paths shared the same core generation and evaluation logic so a recovery operation did not become an untested second product.

The offline production run

PREFLIGHTVerify model and data dependencies
CHECKPOINTFreeze the safe incremental window
BATCHFetch and group changed review evidence
PROCESSGenerate and evaluate with bounded parallelism
COMMITStore results, recovery data and the new checkpoint

Langfuse made one summary explainable

Aggregate dashboards tell you that a pipeline is healthy; they do not explain why one sentence appeared. Langfuse tracing connected the review input, prompt and model version, generation attempt, evaluation results, repair attempts, latency and final acceptance state. Sensitive content and access still required production-safe handling, but the trace identity made an individual investigation possible.

This changed debugging from 'try the prompt again' into provenance analysis. Was the input weak? Did the prompt version change? Did generation omit a qualifier? Did the hallucination evaluator disagree with the guardrail? Did a retry repair the right failure? Observability became part of evaluation because it preserved the evidence behind a decision.

Keep probabilistic work away from checkout

Nemo became two cooperating products. The offline system owned uncertain generation, retries, evaluation and recovery. The online API owned a stable lookup of the latest accepted summary and a duplicate-safe feedback write. Checkout never waited for Gemini and never received an output that was still being evaluated.

The read endpoint validated the car and feature state, respected an immediate kill switch and returned an empty object when no accepted summary existed. The feedback endpoint treated the relevant identifiers as a uniqueness boundary so one device could not record the same judgement repeatedly for one summary. Later database and connection work reduced the uncached API p99 from roughly 150 ms to 17 ms without changing this contract.

Evaluation does not end at model acceptance

I kept three scorecards separate. Model quality covered faithfulness, content coverage, readability, structure and edge-case behaviour. Pipeline quality covered acceptance, repair, failure, fallback and batch-completion rates. Product quality covered how customers interacted with reviews and checkout after the feature appeared.

Checkout-to-payment conversion was deliberately not a P0 metric for Nemo. We had two useful signals: people who completed a booking read more reviews, and cars with more reviews were booked more often. But neither was enough to say that reviews caused conversion. People who booked may already have had higher intent, while cars with more reviews may also have been more popular for other reasons. Conversion sat too far downstream to make that claim honestly. The closer question was whether Nemo improved the review experience by helping people understand useful evidence with less effort.

That separation prevented a common mistake: declaring success because an offline semantic score improved. In product measurement, Nemo was associated with a 46% reduction in review clicks and a 12% reduction in checkout-session duration. Those outcomes suggested that users could obtain useful context with less effort. They did not eliminate the need for continuous sampled review, feedback analysis and regression cases.

Historical versions retained the prompt, model and evaluator context required to compare changes. A new prompt was released against a fixed adversarial set before production observation. A fluent output that looked suspicious could be traced to its evidence and decisions rather than sent to another model for an ungrounded opinion.

A practical checklist for production summarisation

Define the reader and the decision before the model. Profile whether the source contains enough information. Choose extractive, abstractive, aspect or multi-stage architecture from input and risk. Make abstention a first-class output. Write the prompt as a testable contract. Build adversarial cases from the real domain. Separate product guardrails from evidence faithfulness. Bound retries. Version prompts, models and evaluators. Keep generation offline when the user path needs predictable latency. Give every dependency a recovery path and every accepted output provenance.

Nemo eventually served more than 200,000 daily requests and ran for more than ten months without reported downtime. The research history was not wasted scaffolding. Statistical methods taught inspectability; linguistic methods taught domain structure; transformer failures exposed domain mismatch; LLMs improved synthesis; independent evaluation governed trust; and production engineering made the feature usable.

Nemo was never one clever prompt or one model. It was a system that knew what evidence it had, what it was allowed to say, how to check itself, how to fail safely and when a summary deserved not to exist. That is the difference between a summarisation demo and a summarisation product.

Nemo: The Gift That Keeps Giving

The name was inspired by Captain Nemo from Twenty Thousand Leagues Under the Sea, the legendary explorer of the deep. Just as he navigated a vast and largely unexplored ocean, Nemo navigates an ocean of customer reviews, uncovering the opinions, themes and insights beneath the surface and distilling them into concise summaries.