Nemo API: taking uncached p99 latency from 150 ms to 17 ms

The simple version

Nemo does not call an LLM when checkout asks for a summary. The expensive work has already happened offline; the API only validates a request and reads the latest accepted result. That should be cheap. It was not cheap enough.

The uncached p99 was roughly 150 ms. After changing connection ownership, the serving-table shape and the hot query together, it fell to roughly 17 ms. The response contract did not change and no cache was required. This matters because a cache can hide inefficient work; removing the inefficient work makes every path easier to reason about.

This article is intentionally public-safe. It explains the engineering decisions and lessons without exposing customer data, proprietary prompts, internal identifiers, operational commands or confidential decision thresholds.

The final uncached serving path

REQUESTValidate identity and feature state
BORROWTake a health-checked pooled connection
SEEKUse the lookup-aligned index
RETURNEmit the existing JSON contract
RELEASEClose the request session deterministically

First principle: profile the lifecycle, not only the SQL

I decomposed one request into validation, engine and pool access, session construction, database wait, query execution, row materialisation, serialization and cleanup. Tail latency is often a queueing story: an individually fast query can still sit behind connection churn or a pool that is being used badly.

Engine construction was too close to request scope and cleanup was not equally explicit on every return path. That created more connection activity than the endpoint's work justified. The question became: which objects should live for the process, which for one request and which for one statement? Once those lifetimes were explicit, the Python side became straightforward.

Process-scoped engine, request-scoped session

Each worker now owns one SQLAlchemy engine and its pool. A request borrows a connection through a short-lived session, performs its read and releases it. Pool health checks protect against stale connections; a bounded pool prevents the database from becoming an accidental extension of application concurrency.

I did not share sessions across requests. A session is mutable unit-of-work state, not a global database client. Success, validation failure and exceptions all cross the same cleanup boundary, which made connection return deterministic and testable.

engine = create_engine(url, pool_pre_ping=True)
Session = sessionmaker(bind=engine)

def read_summary(car_id):
    session = Session()
    try:
        return repository.latest(session, car_id)
    finally:
        session.close()

The database was part of the optimisation

Connection reuse removed waste, but it did not explain the whole tail. The earlier schema was optimised for preserving generation history, while the API needed one current accepted payload for one car. Asking a historical shape to behave like a serving index forced unnecessary lookup and ordering work.

I separated those responsibilities. History remained append-oriented for audit and analysis; the serving table represented current state. The hot lookup and its composite index were aligned with the endpoint's filter and latest-state semantics, and fields needed by the response lived on the read path instead of being reconstructed through repeated joins. The schema change was small, but it let the database seek directly to what the API meant.

Proving speed without changing meaning

I locked response compatibility first: successful payloads, empty-summary behaviour, validation failures and the feature kill switch still behaved the same. Sequential tests proved infrastructure reuse. Concurrent tests proved session isolation and connection return. Failure injection covered database exceptions and early exits.

Latency measurements distinguished uncached requests from any warmed or repeated lookup. Production observation remained the authority for p50, p95, p99, error rate, pool pressure and database connections. That is how the 150-to-17 ms result was evaluated: as a tail-latency change on the real uncached read path, not a best-case local number.

What I would reuse

Performance work should start with ownership and access patterns. Make object lifetimes explicit, make the table serve one dominant read honestly, and make indexes match the query's equality and ordering semantics. Only then ask whether a cache is still necessary.

Nemo's serving path became boring in the best way: validate, borrow, seek, return, release. At more than 200K daily requests, boring is a serious engineering achievement.