The problem statement
A support decision can look like one ticket while the evidence lives in five systems. The booking and transaction history say what happened, calls and chats say what each person reported, trip events establish chronology, and policy determines which facts are relevant. A human agent had to assemble that story before deciding whether a case could be resolved or needed escalation.
Cipher's job was to automate only the cases for which that evidence supported a defensible outcome. It was not a chatbot and it was not allowed to invent missing facts. Eligibility rules selected the decision type; the service assembled a time-ordered evidence package; an LLM produced structured reasoning; deterministic rules enforced hard policy boundaries; and every automated result retained enough provenance for audit.
This article is intentionally public-safe. It explains the engineering decisions and lessons without exposing customer data, proprietary prompts, internal identifiers, operational commands or confidential decision thresholds.
- CASEResolve eligibility and the decision being asked
- EVIDENCEAssemble calls, chats and trip context
- MODELProduce structured reasoning and support
- POLICYApply deterministic business boundaries
- AUDITStore outcome, evidence version and review state
Failure analysis came before prompt editing
When the escalation team flagged incorrect cases, I manually read each ticket and its available evidence. I tagged the actual failure: missing communication, incorrect chronology, ambiguous language, policy mismatch, unsupported inference, stale evidence or output-format failure. This converted a list of complaints into an error dataset.
I then reviewed the cases with stakeholders from support, operations and product. We discussed disagreements one by one because some apparent model errors were unclear policy, inconsistent human practice or missing source data. The sessions improved the labels and clarified which failures belonged to the model, the evidence pipeline or the product rule.
Method: iterate on the failing layer
Evidence problems were fixed in retrieval and timeline construction. Reasoning problems informed prompt examples, output structure and evaluator cases. Policy disagreements moved into deterministic rules or explicit escalation. This prevented the prompt from becoming a pile of exceptions for problems it could not solve.
The resulting model input used a consistent evidence schema and visible provenance. The output separated summary, responsibility signals, supporting evidence and uncertainty. Cases without adequate support were routed away from automation rather than encouraged to complete the template.
Freshness and duplicate-safe work
Cipher reused a completed analysis only when it represented the newest communication evidence. New calls or chats moved the case back into processing. This semantic cache was safer than a fixed time-to-live because the invalidation event was the thing that changed the answer.
A later workflow exposed a concurrency race: two requests could both observe no completed row and start the same expensive model work. I moved the claim before the external call. A normalised business identity was protected by a database uniqueness rule; one worker received a lease and owner token, while duplicates saw the stored result or an in-progress state.
request_key = hash(case, subject, normalise(reason))
owner = claim_once(request_key, lease)
if owner:
evaluate_newest_evidence()
else:
return existing_or_processing_state()
Evaluation: accuracy, coverage and the cost of the wrong error
The system used a labelled historical set, newly flagged failures and continuing sampled audits. Evaluation reported both selective accuracy and automation coverage: a system that abstains on everything is accurate but useless, while one that automates everything transfers hidden error cost to customers. Roughly 51% eligible automation at about 94% audited accuracy describes both sides of that operating point.
I reviewed confusion by decision and failure type, not just one aggregate. False automation could be more costly than unnecessary escalation, so thresholds and eligibility were asymmetric. Slices covered evidence availability, chronology length, language ambiguity, decision subtype and new versus previously seen policy patterns. Prompt or retrieval changes had to pass the fixed regression set and fresh blind review.
System evaluation mattered too. Tests covered missing and stale evidence, malformed model output, deterministic policy disagreement, provider failure and audit persistence. Concurrency tests used genuinely overlapping requests because sequential retries cannot prove duplicate prevention; lease expiry and stale-owner cases ensured an old worker could not overwrite a recovered request.
Result and impact
Across eligible flows, the broader system reached roughly 51% automated resolution at about 94% audited accuracy. The system improved through a loop of ticket review, explicit failure tagging, stakeholder adjudication and layer-specific fixes—not through one prompt rewrite.
Cipher’s most reusable lesson is that production LLM quality is organisational as well as technical. Someone has to read failures, agree on policy, repair evidence and encode concurrency semantics. The model sits inside that system; it does not replace it.