Benchmarks

the numbers, and how to run them yourself

Every benchmark here is seeded, runs in seconds, never touches your real index, and prints whatever the numbers are. The corpus generators are ordinary reviewed Go — audit what "relevant" means before trusting any figure.

Recall quality

deja bench recall

Builds a seeded corpus of 500 sessions and 100 queries — half of them Russian, because the ranking it guards runs mostly on stores that are not English — runs them through the real search pipeline, and reports recall@1/@5/@10 and MRR with median latency (plus a hybrid column when an embedding endpoint is up). Each query names one session, so a miss is a ranking error rather than an ambiguous corpus: recall@1 sits at 1.00 and a test pins it, which is what CI fails on.

Context readiness

deja bench context

The interesting number. Across seeded multi-session task chains, it compares four ways to assemble the context a task needs — deja recall, replaying full history, naive grep, and cold — reporting median tokens with spread and verbatim fact coverage. On the default corpus, deja reaches the same fact coverage as grepping the raw logs for roughly 250× fewer tokens — and about 70× fewer than replaying the matched sessions in full — while injecting nothing on the negative-control chains where no prior fact is relevant.

Does the block carry the answer

deja bench block
deja bench prompt

bench block asks what the other two cannot: with the right session in hand, does what deja hands over still hold what that session settled. Eight sessions discuss each subject and one settles it, in the middle of its own transcript rather than at the end, so the baseline arm — the newest turns of the top hit — scores zero. The session-start block carries the answer on 1.00 of the chains at a median of 665 tokens; the digest does the same at 1,656.

bench prompt is the other side: what the per-prompt hook fires on, and what it fires on wrongly. Real questions, long sessions, rewordings and ties on one side; negative controls and cross-paired questions — a question from one chain asked against another's sessions — on the other, where a fire is a false one. The arms ask through the hook itself, so a gate that only exists there is measured.

What an update costs

deja bench ingest

The three above score what comes back. This one asks what it costs to keep the index current, which is the half nobody was measuring: a path that rewrote and re-tokenized the whole store every time a new conversation appeared survived two months and twenty releases before anyone produced a number for it.

Five classes, because they are the things that happen to a store — nothing changed, a turn was appended to a transcript, a transcript appeared, a transcript was renamed, a transcript was rewritten — and for each one the wall time, the records written, and whether the records already on file were replaced. None of them should replace them: each must cost what it changed, a rewritten transcript appends its records, marks the old ones dead and clears them where they stand, and a rename, which some harnesses do when a session is resumed, must not cost a second copy of the log. That column is the regression gate, because it is exact where a wall time on a shared runner is not.

What a store costs to read

deja bench read

Every other benchmark here runs against a corpus that is already indexed, so the readers — the stage that turns a harness's own store into sessions — had no number at all. That is where the sqlite3 shell's -json mode sat for two months: quadratic in the characters it escapes, invisible on kilobyte fixtures, and 2,287 seconds on one 6.16 MB value from a real store.

Two arms, because the cost does not follow the store's size: an ordinary database-backed store, and the same store with one long escape-heavy tool result in it. The ratio between them is the gate — a reader that is linear in what it escapes keeps them within a few times each other, and one that is not blows the second arm up without touching the first.

Public benchmarks

go run ./scripts/longmemeval -skip-abs -data longmemeval_s.json -out docs/benchmarks/longmemeval-s-cleaned.json
go run ./scripts/longmemeval -data longmemeval_s.json -out docs/benchmarks/longmemeval-s-full.json
go run ./scripts/locomo -data locomo10.json -out docs/benchmarks/locomo.json

-out writes the run itself — every slice, the dataset's own sha256, the wall time and the median search — and the files it wrote are in the repository: longmemeval-s-cleaned.json, longmemeval-s-full.json and locomo.json. Each record names the flags it ran with, which is the part that matters here: without -skip-abs the same code scores 87.4%, and a number quoted without its question count looks like drift. The numbers below are read from them, and a test fails if the two disagree.

The driver scores somebody else's ranking too: go run ./scripts/longmemeval -score yours.json takes a file mapping each question id to the session ids a system ranked and prints the same table, with no index and no deja in the loop. The format is here, and a number that beats the rows below is worth more to whoever is choosing a tool than another paragraph from us.

Two published long-term-memory benchmarks run through the exact production path: haystack sessions are written as real transcript files, indexed by the normal pipeline (parsing, redaction, tokenization), and queried with the verbatim question text. No LLM, no embeddings, no question rewriting. Both harnesses ship in the repo and finish in minutes on the public datasets.

LongMemEval-S (session-level retrieval, 470 questions on the cleaned set): hit@1 88.1%, hit@5 97.4%, hit@10 98.7%, hit@20 99.6%, MRR 0.920, median search ~15 ms. hit@k credits a question when any of its evidence sessions ranks in the top k; the stricter official per-evidence metric is reported alongside: evidence-recall@1 56.6%, @5 91.5%, @10 95.3%, @20 97.7%. Weakest slice: preference questions (43.3% hit@1 — pure paraphrase is where lexical retrieval loses to semantic indexes, by design; hit@20 reaches 100%). Full-set numbers (500 questions, abstention included): hit@1 87.4%, MRR 0.915 — that run is committed too.

LoCoMo (session-level retrieval, 1,982 questions): hit@1 70.5%, hit@5 90.9%, hit@10 95.6%, MRR 0.794, median search ~4 ms; evidence sessions carry their real dates, questions anchor relative time at their question date. Note: most published LoCoMo numbers are end-to-end QA accuracy with an answering LLM — a different metric; deja reports retrieval only.

Both re-measured on main after 0.20.2, against the public datasets, on an M-series laptop. Each LongMemEval question is asked against its own haystack of about fifty sessions; asked against all 19,195 at once, the same kind of question lands at 19% hit@1, and the day-zero page runs that.

Found this useful? Star deja-vu on GitHub.