Benchmarks
Both benchmarks are seeded, run in seconds, never touch your real index, and print whatever the numbers are. The corpus generators are ordinary reviewed Go — audit what "relevant" means before trusting any figure.
Recall quality
deja bench recall
Builds a seeded multi-session corpus and query set, runs them through the real search pipeline, and reports recall@5/@10 with median latency (plus a hybrid column when an embedding endpoint is up). CI fails if recall regresses.
Context readiness
deja bench context
The interesting number. Across seeded multi-session task chains, it compares four ways to assemble the context a task needs — deja recall, replaying full history, naive grep, and cold — reporting median tokens with spread and verbatim fact coverage. On the default corpus, deja reaches the same fact coverage as grepping the raw logs for roughly 200× fewer tokens — and about 60× fewer than replaying the matched sessions in full — while injecting nothing on the negative-control chains where no prior fact is relevant.
Public benchmarks
go run ./scripts/longmemeval -skip-abs -data longmemeval_s.json go run ./scripts/locomo -data locomo10.json
Two published long-term-memory benchmarks run through the exact production path: haystack sessions are written as real transcript files, indexed by the normal pipeline (parsing, redaction, tokenization), and queried with the verbatim question text. No LLM, no embeddings, no question rewriting. Both harnesses ship in the repo and finish in minutes on the public datasets.
LongMemEval-S (session-level retrieval, 470 questions on the cleaned set): hit@1 84.9%, hit@5 94.3%, hit@10 95.7%, hit@20 97.0%, MRR 0.890, median search ~40 ms. hit@k credits a question when any of its evidence sessions ranks in the top k; the stricter official per-evidence metric is reported alongside: evidence-recall@1 55.0%, @5 83.6%, @10 86.9%, @20 89.7%. Weakest slice: preference questions (36.7% hit@1 — pure paraphrase is where lexical retrieval loses to semantic indexes, by design; hit@20 still reaches 100%). Full-set numbers (500 questions, abstention included): hit@1 84.2%.
LoCoMo (session-level retrieval, 1,982 questions): hit@1 69.8%, hit@5 85.6%, MRR 0.766, median search ~6 ms; evidence sessions carry their real dates, questions anchor relative time at their question date. Note: most published LoCoMo numbers are end-to-end QA accuracy with an answering LLM — a different metric; deja reports retrieval only.