Parity on accuracy. The win is cost.

The result, stated honestly

Titen 0.6.0 with lexical FTS5 retrieval and nothing else scores recall@1 0.880 and MRR@100.9147 on all 500 LongMemEval-S instances, using zero LLM calls, zero embedding calls, and 816 seconds of ingest. That is the top of the table.

The lane directly below it is a control we wrote ourselves: roughly a hundred lines that embed each session and rank by cosine similarity. It scores 0.854. On the same 500 instances, the paired sign test between the two is 44 wins, 31 losses, 425 ties, p = 0.165. That is not a difference. A lead in the ordering is what an unmeasurable gap looks like when 425 of 500 instances answer the same way.

Governance, provenance, conflict handling, leases, and handoffs are not in this benchmark at all. No external suite measures them. They are capabilities we describe, never things we have proven better.

The table, with its significance

LongMemEval-S, 500 instances, 246,930 turns. The task is to rank an instance's roughly 50 haystack sessions for its question; ground truth is the instance's answer_session_ids. One scorer ran every lane, and failures stay in the denominator.

W / L / T and p are paired sign tests on recall@1 over the same instances, read as Titen wins, Titen losses, ties. An em dash means the figure is not in our published record for that lane — not zero.

Read the p column before the recall column. Titen's wins over the MiniLM and fastembed lanes are significant; its lead over the router-embedding control is not. Two of those significant wins are partly a context-length artifact rather than a clean retrieval result — seewhere Titen loses.

recall@5 and recall@10 are deliberately absent. On this corpus they are saturated between 0.97 and 0.99 across every serious lane, so publishing them would imply a discrimination that is not there. recall@1 and MRR@10 are the primary metrics here.

The 60-instance subsample

One lane could not be run at n=500 on any reasonable budget. Mem0 OSS 2.0.15 was measured on a deterministically stratified 60-instance subsample, 10 per question type, and every lane below ran on those same 60 instances. W / L / T is read as the row's wins, losses, and ties against Mem0 OSS.

n=60 is small and every p-value above reflects that. Note also that the Titen lane here is FTS+vector, which does use an embedding provider — it is not the zero-provider lane from the first table, and it has no 500-instance run yet.

What the LLM calls bought

Mem0 OSS 2.0.15 spent 2,981 LLM calls and 288,021 seconds of summed ingest — about 80 hours— to score recall@1 0.8333 on those 60 instances. On the same instances, Titen's FTS+vector lane scored 0.8833 and a verbatim-RAG control scored 0.8500, both with zero LLM calls.

The honest reading has two halves, and the second one matters as much as the first.

• Extraction bought no measurable retrieval advantage over embedding the raw sessions, at hundreds of times the ingest cost. That reproduces the direction of MemDelta (arXiv:2606.29914) on a different subsample with a different scorer.
• The gap is not significant either. Mem0 OSS against Titen FTS+vector is 2 wins, 5 losses, 53 ties, p = 0.45. Against the verbatim-RAG control it is 3 / 4 / 53, p = 1.0. Nobody separated from anybody on accuracy. What separated was the bill.

Two things must travel with any comparison to Mem0. First, write Mem0 OSS 2.0.15, never bare "Mem0" — the open-source SDK and the managed platform are different products with different numbers. Second, their README says so itself, immediately under their headline table:

All benchmarks run on the same production-representative model stack. Single-pass retrieval (one call, no agentic loops) at a top_200 retrieval budget. Scores reflect Mem0's managed platform, which includes proprietary optimizations not available in the open-source SDK; open-source users should expect directionally similar gains but not identical numbers.

mem0ai/mem0 README, read 2026-08-05. Their evaluation framework is published separately atmem0ai/memory-benchmarks.

We quote that ourselves rather than let someone else surface it. Their platform numbers are not what we measured, and this page does not compare against them.

What would have falsified this

The rules this run was held to were written down before it ran. They are indocs/testing/EVALS.md at commit 88935ba, dated2026-08-01 — three days before the first instance was scored. Verbatim:

Each one had a price, and each one was paid:

• The mandatory FTS-only baseline is why we can tell you that a hundred lines of cosine matches us. A run that only reported the hybrid lane would have looked much better and said much less.
• "Publish every trial, timeout, retry, exclusion, and failure" is why the voided 500-instance FTS+vector run appears above as a gap rather than being quietly re-run until it produced a number.
• The hosted-product rule is why the Mem0 comparison names an exact OSS SDK version and quotes their own platform caveat instead of comparing against their headline figures.
• "Never publish a best run without distribution or run count" is why an earlier internal record was corrected: a figure that had been quoted as a median turned out to be the top of its range.

The pre-registered control very nearly beat us. That result is in the first paragraph of this page rather than in a footnote, which is the only thing a pre-registration is actually for.

Where Titen loses

Five limitations, all of them ours, all of them measured or explicitly unmeasured.

• The corpus is nearly out of headroom. recall@10 sits between 0.97 and 0.99 across every serious lane, so only recall@1 and MRR@10 discriminate at all. A ranking improvement that does not move those two has not been measured on this corpus.
• FTS-only degrades badly as the store grows. On a synthetic corpus, recall@1 falls from 1.00 at 10³ active claims to 0.49 at 10⁵. The absolute values belong to that corpus's generator, not to Titen — but the shape is the result, and it means any FTS-only quality figure has to be quoted with the corpus size it was measured at. The 0.880 above is a LongMemEval-S number, not a promise at any scale.
• One process saturates one core at 10,000 claims. That is the current single-node ceiling.
• There is no reranker, and the ceiling for one is small. An oracle over Titen's own top-10 would add 10.2 points, 0.880 to 0.982 — that is the entire addressable gain. The best cheap reranking signal we tested captured 0.6 of those points at p = 0.61, and two other signals were significantly worse. MemPalace ships a reranker and scored lower with it in both embedding arms.
• The embedding arms are confounded, and it contaminates our largest effect. A budget control found the router arm at 0.804 with a 512-token budget and 0.858 at 2,048. Roughly five points of the router-versus-fastembed gap is context length, not model quality — so the significant wins over the weaker-embedder lanes are partly a context-length result.
• Governance and collaboration are unmeasured. Authorization before retrieval, provenance, visible conflicts, leases, checkpoints, handoffs and audit are not scored by any external suite, including this one. Nothing on this page is evidence about them.

Reproduce it yourself

The corpus is externally authored, public, and MIT-licensed. The protocol is fixed and written down. The system under test installs in one command and runs the scoring configuration by default.

# 1. the corpus is externally authored, public, and MIT
gh api repos/xiaowu0162/LongMemEval --jq .license.spdx_id      # MIT

# 2. the system under test, at the exact operating point that scored 0.880
curl -fsSL https://titen.dev/install.sh | bash
titen bootstrap --org 'Bench'
titen serve                        # FTS-only is the default: no key, no provider

# 3. the rules the run was held to, three days before it ran
git clone https://github.com/RamaAditya49/titen && cd titen
git log --format='%h %ad %s' --date=short -- docs/testing/EVALS.md
git show 88935ba:docs/testing/EVALS.md

One thing this page will not pretend to. The harness is a working directory, not a release. It lives at ~/titen-bench-20260804/ on a single machine, and the per-lane artifacts areresults/*-500.json plus the raw *.ranked.json each score was recomputed from. It is not packaged, versioned, or installable today, and calling it a reproducible harness would be the same overstatement this page exists to avoid.

What you can check right now, without our machine:

• the corpus, its licence, and its ground truth, at xiaowu0162/LongMemEval;
• the protocol and the claim rules, in docs/testing/EVALS.md, with their commit dates;
• the full competitor record, including two corrections that go against us, inthe 2026-08-04 landscape note;
• Titen itself, at the operating point that produced 0.880, on your own data.

Method

• Corpus. LongMemEval-S, 500 instances, 246,930 turns, externally authored by Di Wu et al. (ICLR 2025), MIT. Not written by us, not tuned by us.
• Task. Rank the instance's roughly 50 haystack sessions for its question. Ground truth isanswer_session_ids. Primary metrics are recall@1 and MRR@10.
• Pre-registration. The lanes, the mandatory baseline, and the claim rules were fixed in the repository before the run and are quoted above at their commit.
• One scorer. The same scorer ran every lane, including ours. Failures are kept in the denominator rather than dropped.
• Significance. Paired sign tests on recall@1 over identical instances, two-sided. Reported for every comparison the page makes a claim about, including the ones we lose or tie.
• Two embedding arms. A router arm at a 2,048-token limit and a fastembed arm at 512, to test MemDelta's finding that swapping the embedder alone can flip a verdict. Currently confounded with context length; see issue #268.
• Subsample. 60 instances, 10 per question type, deterministically stratified, shared by every lane in that table.

Measured on one host on 2026-08-04 and 2026-08-05. One corpus, one language, two days. Every number here expires the moment a lane is re-run, and the re-runs in flight are named inthe table.