Resonance Lattice, measured

At a glance

How we measure

Every benchmark is a paired comparison against named baselines, run against a committed test set, runnable from a single python -m benchmarks… invocation.

We use Claude Sonnet 4.6 as the canonical answer model and judge for the LLM-judged benchmarks (hallucination, token spend). Where LLM-judge variance affects results, we run inter-rater 10% spot-validation. Test sets are committed in full to the repository — auditability over compactness.

Concepts you need to read these benchmarks

If you've never used rlat, the tables below talk about three retrieval shapes, three grounding modes, two question types, and two judge rubrics. None are jargon — read this once and the rest of the page reads in plain English.

What rlat actually does

rlat packages your corpus (a codebase, docs, a knowledge base) into one .rlat file — a knowledge model — and ships commands to query that file. It is an answer-improvement layer: retrieval surfaces passages with full citation provenance and a live drift status against the source bytes, and the deep-search loop plus the insight layer go further and produce faithfully-grounded answers — every claim traced back to a passage in the corpus. rlat owns grounding, not truth: it represents what the corpus says, never what is true in the world.

Two CLI verbs do the retrieval work this page tests:

• rlat search km.rlat "<question>" — one query, one round of retrieval, top-k passages back. Fast, cheap, simple.
• rlat deep-search km.rlat "<question>" — internally runs a multi-hop loop (plan → retrieve → refine → maybe re-retrieve → synthesize) and returns a final answer plus the union of evidence. Slower, roughly 2.5× pricier, much higher accuracy.

Retrieval shapes (the columns in the tables)

The benchmarks test three retrieval shapes against the same corpus:

• Single-shot — rlat search once. Top-k passages. The cheapest rlat tier (~$0.003–0.004 per question with Sonnet downstream).
• Multi-query rewrite — Sonnet generates three query variants of the user question, rlat search runs each, and results dedupe. A middle ground; rarely the right pick now that deep-search exists.
• Deep-search — the productionised multi-hop loop. Pricier (~$0.009/q) but consistently the highest-accuracy lane.

Grounding modes (how rlat tells the LLM to use the evidence)

rlat search --format context and rlat skill-context stamp a one-paragraph directive at the top of the markdown they emit. The directive instructs the consumer LLM how to weight the passages against its training knowledge. Three modes:

The mode applies to all three retrieval shapes — you can run single + augment, multi + constrain, deep + knowledge, and so on. Three modes × three shapes is nine lanes; we test all nine plus two baselines.

Question types (the rows of the test set)

The hallucination benchmark has two kinds of questions, both required for an honest evaluation:

• Answerable (51 questions) — the answer is genuinely in the corpus. Tests accuracy: did the LLM find and use the right passages?
• Distractor (12 questions) — the question asks about something that doesn't exist in the corpus. Fake product names ("Materialized View Express"), made-up SKUs ("F4096"), Power-BI-only features asked as if they were Fabric features. The correct answer is refusal. Tests safety: does the LLM correctly say "this corpus doesn't cover that", or does it hallucinate an adjacent answer?

Distractors are the harder axis. A confident wrong answer to "How do I configure F4096 SKU pricing?" is the failure mode that erodes trust. On distractor rows, any non-refusal counts as hallucination — even an answer that's technically correct about an adjacent entity, because the user asked about something else.

Hallucination

In these benchmarks, a hallucination is a confidently-wrong answer (judged wrong by the rubric: a different fact, a contradiction of ground truth, or invented content). The hallucination rate is the share of questions that got a wrong verdict. Lower is better.

Strict vs relaxed judge rubric

Every trial is graded twice by the same Sonnet judge with two different rubrics:

• Strict rubric — "missing a critical detail" downgrades to partial. Counts only fully-detailed answers as correct.
• Relaxed rubric — "load-bearing fact captured" counts as correct, even if secondary details are missing or imprecise. Closer to a human user's "did this help?" judgment.

The headline numbers on this page are the relaxed rubric (closer to user-perceived usefulness). Both result JSONs ship — pick the framing that matches your evaluation needs.

Hallucination reduction

The question this benchmark answers: when you hand an LLM passages from a corpus, does it actually ground its answer in those passages — or does it confidently invent? Across 11 lanes (three modes × three retrieval shapes, plus LLM-only and LLM+grep/glob) on the Microsoft Fabric documentation, here's what we measured.

The setup: a real-world corpus the LLM partially knows

We built a knowledge model from the public Microsoft Fabric documentation (2,261 markdown files, 62,914 passages, all ms.date from 2019 through 2026-03-28). This is the right corpus shape for the test: Microsoft Fabric has been in market since 2023, so Sonnet 4.6 has substantial Fabric training data — but the corpus contains 559 files dated after Sonnet's January 2026 cutoff and another 679 from 2025-09 to 2026-01 (Sonnet's training fuzzy zone, where it may have older versions). Asking Sonnet about Fabric is exactly the kind of question where it confidently gives a slightly out-of-date answer.

The test set is 63 hand-written questions, ground-truth sourced verbatim from specific dated Fabric docs:

• 51 answerable questions across recency tiers (post-cutoff 2026-02/03 features, fuzzy-zone 2025-09 to 2026-01, stable pre-2024 concepts).
• 12 distractor questions about plausible-sounding things that don't exist in Fabric (fake F SKUs, made-up product names, Power-BI-only features asked as Fabric features).

Each question runs four ways: rlat-constrain, rlat-augment, rlat-knowledge, and no-retrieval (Sonnet alone, no rlat in the loop). A separate Sonnet judge call grades on a four-state rubric (correct / partial / wrong / refused) — wrong is the hallucination signal.

The numbers (11-lane matrix, relaxed rubric)

We benchmark every combination of grounding mode (augment, constrain, knowledge) × retrieval shape (single-shot, multi-query rewrite, the rlat deep-search 4-hop loop), plus two baselines: LLM-only (no retrieval) and an LLM with grep / glob / read_file tools on the same corpus. 63 questions × 11 lanes × 2 LLM calls is 1,386 inference plus 693 judge calls. Total benchmark cost: $8.93.

What this means

Deep-search is the high-quality, low-spend lane. 92.2% answerable accuracy at 0% hallucination on the Fabric corpus (the --mode knowledge variant) matches an 8-tool LLM+grep/glob baseline (94.1%) at 6.5× lower spend ($0.009/q vs $0.060/q) and lower wall-time (~12s/q vs ~15s/q). The deep-search loop runs in one call — plan → search → refine → synthesize — with citations and a name-verification check on the union of all hops' passages.

Single-shot rlat search already adds ~20 pp over LLM-only. Even without the loop, the simplest invocation (rlat search --format context --mode augment + Sonnet) reaches 76.5% accuracy at 3.9% hallucination, against the LLM-alone floor of 56.9% / 19.6% — roughly 5× lower hallucination, at ~$0.004/q. The cheapest tier of the matrix.

Constrain is the compliance floor. Single-shot --mode constrain hits the lowest answerable hallucination in the suite (2.0%), the highest distractor refusal (91.7% — it invents nothing 11 of 12 times on made-up product names), and the lowest distractor hallucination (8.3%). The trade is 10 pp of answerable accuracy. Pair it with --verified-only and --strict-names for fact-extraction, regulatory, or audit work where wrong-but-confident is worse than no answer.

Multi-query rewrite is dominated by deep-search. The middle column (multi-query: generate three query variants, retrieve each, merge and dedupe) adds modest accuracy over single-shot but is dominated by deep-search on every cell. Use deep-search when you want a multi-hop synthesis, single-shot when you want speed; multi-query is only worth it for query-rewrite ablations.

LLM-only loses across every metric. 19.6% hallucination on a broad documentation corpus the LLM partially knows is the safety ceiling without retrieval. That is the cost of trusting training data alone on questions a user might genuinely ask.

Mode trade-offs

• augment (default) is the value pick on broad domain corpora. It blends the corpus with the LLM's prior knowledge — highest single-shot accuracy at modest hallucination, and the deep-search default.
• constrain is the safety floor. 2.0% hallucination single-shot, 91.7% distractor refusal — pick for compliance, regulatory, or audit. Pair with --verified-only and --strict-names.
• knowledge sits between augment and constrain. Under deep-search it ties augment on accuracy at zero hallucination — the recommendation for "I want the assistant to lean on training where the corpus is thin, but never invent." Under single-shot it underperforms augment; the lighter gate doesn't pay off without the multi-hop loop.

Test set: 63 hand-written questions (51 answerable / 12 distractors), ground truth quoted verbatim from dated Microsoft Fabric documentation across recency tiers. Total cost $8.93 ($7.47 inference + $1.46 relaxed-rubric judge). Both rubric result JSONs are committed to the repository.

Result artifacts: hallucination_fabric_11lane.json (strict) · hallucination_fabric_11lane_relaxed.json (relaxed)

pip install rlat[bench]
rlat install-encoder
# Build the Fabric corpus locally OR pull a prebuilt knowledge model:
rlat build path/to/fabric-docs/docs -o fabric-docs.rlat
export CLAUDE_API=sk-ant-...
python -m benchmarks.user_bench.hallucination.run \
  --km fabric-docs.rlat \
  --tasks-file benchmarks/user_bench/hallucination/fabric_tasks.jsonl \
  --output benchmarks/results/user_bench/hallucination_fabric_11lane.json \
  --budget-usd 12
python -m benchmarks.user_bench.hallucination.rejudge_relaxed \
  --input benchmarks/results/user_bench/hallucination_fabric_11lane.json \
  --output benchmarks/results/user_bench/hallucination_fabric_11lane_relaxed.json \
  --tasks-file benchmarks/user_bench/hallucination/fabric_tasks.jsonl \
  --budget-usd 3

Token spend

The question this benchmark answers: how many LLM tokens (and dollars) does it take to reach a correct grounded answer about a corpus, versus the alternatives a developer would actually consider — letting the LLM grep and read files itself, or dumping the whole corpus into the prompt?

Tested on 20 hand-written questions about the rlat repository itself — a different, harder corpus than the Fabric one above: the questions are about a brand-new private codebase Sonnet has never seen, and the answers live across multiple files. The retrieval surface used here is rlat skill-context — same retrieval as rlat search --format context, shaped for an Anthropic skill !command injection block.

rlat single-shot vs LLM-only

The same 20 questions, four ways:

The single-shot finding: on a code-heavy corpus the LLM has never seen, single-shot rlat is enough to take Sonnet from 0% correct to ~35% at $0.012 per right answer. The accuracy ceiling is low here because some questions need cross-file synthesis that single-shot can't do — exactly the case rlat deep-search was built for, but those numbers haven't been measured on this corpus yet. On the Fabric corpus where deep-search has been measured, it adds ~16 pp accuracy over single-shot (see the Hallucination section above). Total cost of this run: $0.37 (4 lanes × 20 tasks + 80 Sonnet judge calls).

Cost-comparison rows

The grep+read agent loop and the full-corpus dump are measured against the same test set as a baseline. These approaches don't depend on the grounding-mode directive — the LLM either has tools or has the whole corpus dumped in context.

The cost-ratio headline:

• rlat-constrain is 3.7× cheaper per correct answer than the grep+read agent loop ($0.012 vs $0.044).
• rlat-constrain is 67× cheaper per correct answer than dumping the full corpus into context ($0.012 vs $0.796).

The grep+read loop wins on accuracy (85% vs 35%) because it can iteratively dig into the codebase when the first retrieval misses. For thoroughness-critical workflows, the agent loop is a real alternative; for volume-critical workflows where each query budget is small, rlat is the order-of-magnitude cheaper option.

Result artifact: token_usage_v2.json

python -m benchmarks.user_bench.token_usage.run \
  --output benchmarks/results/user_bench/token_usage_v2.json \
  --budget-usd 16

Retrieval quality (BEIR-5 floor + encoder comparison)

rlat locks a BEIR-5 floor at the base recipe. Anyone using the default install with no optimisation gets these numbers, and we benchmark the encoder choice against the named alternatives a practitioner would consider on the same stack.

Result artifacts: beir5_encoder_comparison_v1.json · aggregate_5beir_qwen3_8b_v1.json

Headline: gte-modernbert-base wins the stack apples-to-apples against BGE-large (+0.026 nDCG@10) and E5-large (+0.081 nDCG@10), at roughly half the embedding dimensionality — and also edges Qwen3-Embedding-8B (+0.014 nDCG@10) at 1/5 the dimensionality. Per-corpus is mixed — nfcorpus and scidocs go to BGE (short keyword / abstract corpora); scifact, arguana, and fiqa go to gte-mb. Arguana (paraphrase-heavy argument retrieval) is where gte-mb's masked-language- model training pays off most: +0.10 nDCG@10 vs BGE, +0.29 vs E5.

What 0.5144 BEIR-5 nDCG@10 means in context. BEIR is the canonical academic benchmark for zero-shot dense retrieval (Thakur et al., 2021). Open-source dense encoders that are widely deployed today sit roughly in the 0.40–0.55 range on BEIR-5 averages — BGE-large and E5-large in this table are typical of that band. The current frontier (closed-source 1.5K+ dimension encoders, much larger architectures, or dense+rerank pipelines) reaches 0.55–0.60. rlat ships at 0.5144 with a 768d Apache-2.0 encoder and no proprietary index — competitive with the open-source frontier and strictly better on this measurement than the most-deployed alternatives a user would otherwise reach for.

Per-corpus floor on the base recipe (gte-modernbert-base 768d):

Result artifact: v2_floor_gte_mb_base_768d.json

The same locked stack generalises beyond prose corpora: on the LongMemEval 500-instance conversational-transcript split it scores MRR 0.936 — retrieval recall, not task accuracy (longmemeval_v2_retrieval.json).

Build & query speed

The question this benchmark answers: how does rlat compare on the operational metrics — how long does it take to build a knowledge model from scratch, how fast is a query against it, and how much disk does it use — versus the closest open-source alternatives a developer would otherwise reach for? Side-by-side at N=1,000 passages on a fixed deterministic synthetic corpus (Windows 11 + Intel CPU, no CUDA):

rlat wins on the user-facing metrics. Warm query p50 17.15 ms vs Chroma's 145.26 ms — 8.5× faster. On-disk size 2.72 MB vs Chroma's 8.59 MB — 3.2× smaller (rlat ships verified-retrieval metadata in about one third the bytes Chroma needs for its sqlite + WAL).

What 17 ms p50 means in context. Sub-20 ms warm-query latency on a CPU keeps interactive prompts responsive — well under the 100 ms threshold above which a user perceives lag. Hosted vector databases (Pinecone, Weaviate, Chroma cloud) run at similar single-query times once the data is in RAM, but add a network round-trip on top — a typical hosted end-to-end is 50–200 ms. rlat at 17 ms p50 is competitive with hosted vector DBs for the in-machine portion and saves the network hop entirely.

The OpenVINO runtime is doing the work — Intel CPUs auto-select it via rlat install-encoder. PyTorch is about 2× slower per query on the same machine; the OpenVINO win is the headline reason to install the Intel stack on developer workstations.

Where rlat doesn't win:

• Build time. OpenVINO is about 2.6× slower than PyTorch on this CPU for one-shot build throughput. The runtime stack is optimised for query-time latency, not for encoding bulk passages once. Build is one-shot; queries are repeated millions of times. We publish the gap honestly. For a faster build at a small recipe-determinism cost, pass rlat build --runtime auto — it defaults to OpenVINO on Intel and ONNX elsewhere; PyTorch is the explicit fallback for non-Intel CPUs without ONNX.
• Chroma builds fastest (21.5 s — it uses the lightweight 384d all-MiniLM-L6-v2 ONNX bundled embedder, versus rlat's 768d gte-modernbert-base) but pays the gain back 8× in query latency.

Result artifact: build_query_speed.json

pip install rlat[bench]
python -m benchmarks.user_bench.build_query_speed.run \
  --output benchmarks/results/user_bench/build_query_speed.json

Session-start primer

The question this benchmark answers: at the moment a session starts, which of rlat's session-start affordances actually moves the needle — the code-base primer (the corpus overview written by rlat summary), the memory primer (clustered cross-session memory entries), both primers loaded together, per-turn rlat search, or none at all (cold)?

rlat ships two complementary primer surfaces:

• Code-base primer — an extractive overview of the corpus: the top-N passages closest to the corpus centroid (Landscape) plus a per-source-file passage-count table (Structure). About 3 KB, generated once per build.
• Memory primer — clustered entries from the cross-session memory store: consolidated decisions, experiment outcomes, prior-conversation conclusions. About 2 KB.

The setup

25 session-start scenarios on this codebase (resonance-lattice.rlat, 3,506 passages, 126 files), each scenario a 2-turn conversation (opening question + follow-up). Four tiers:

Five lanes, 25 scenarios × 2 turns × 5 lanes is 250 inference calls + 250 judge calls. Total benchmark cost: $2.31. Sonnet 4.6 as answer model + judge with the four-state relaxed rubric.

The numbers

Per-tier breakdown — the headline finding

The aggregate numbers hide the most important result. Every primer type has a coverage profile: it shines on the tier its content was designed for and degrades to roughly cold elsewhere.

The signal:

• The code-base primer wins orientation (3/5) but is blind on memory recall (0/5) and weak on deep facts (2/10). The Landscape + Structure surface is exactly the right shape for "what is this project, what's in it, where do I look" — and exactly the wrong shape for "what does the incremental delta-apply pipeline do on refresh".
• The memory primer wins memory recall (5/5) but is blind on orientation (0/5). It surfaces decisions, not layout.
• rlat search wins specific-factual (8/10) and cross-reference (2/5) because it retrieves the actual passages the answer needs. But it's blind on orientation (0/5) — a single search returns the 3–5 most relevant passages, not a mental model of the whole corpus.
• Both primers loaded is the broadest coverage profile: it picks up the orientation wins of the code primer (3/5) and the memory wins of the memory primer (5/5), and the LLM can stitch context across them (3/10 specific-factual vs 2/10 for either primer alone).

Token usage

Measured input/output tokens per Sonnet call, averaged across the 25 scenarios:

The primer-only sizes, measured as input tokens above the cold baseline of ~90:

• Code primer ≈ 1,708 tokens per call (~3 KB markdown). Carries Landscape + Structure + a slice of Evidence.
• Memory primer ≈ 746 tokens per call (~2 KB markdown). Clustered memory entries.
• Both primers concatenated ≈ 2,454 tokens per call.
• rlat search top-5 ≈ 704 tokens of passages per call — the same order of magnitude as the memory primer alone, but dynamic per turn rather than fixed at session start.

The honest take: primers are not free, but they're cheap relative to both the corpus dump (about 1,400× larger) and the accuracy gap to cold (0% → 48% turn-1 correct on both_primers). The memory primer in particular is exceptionally cheap (~750 tokens) for a 5/5 win on memory-recall scenarios — the highest accuracy-per-token efficiency of any lane on its target tier.

What this means in practice

If a session starts with a question whose answer isn't in either primer's coverage zone, primers don't help — they degrade to roughly cold (cold scores 0/25 because the model has no rlat context at all and no tools to fetch any). The honest framing: primers are not a substitute for retrieval. They're an orientation surface for the opening minute of a session.

The combined-stack reading: load both primers at session start (free, ~5 KB combined) and keep rlat search available as a tool call. The primers carry orientation + memory recall; per-turn search picks up the specific facts the primers can't fit. None of the lanes individually crosses 60% turn-1 correct on this 25-scenario set — which is the right honesty signal: session-start is hard, and single-shot retrieval is not deep-search. For the highest accuracy on a session-start question that turns out to need synthesis, rlat deep-search (the hallucination-bench lane that hit 92.2%) is the right tool — at the cost of one extra round of latency.

Honest caveats

• MVP scope. 25 scenarios — half the original plan's 50. Tier-level n=5 is small; the per-tier numbers should be read as directional, not precise.
• Full-context lane skipped. The Token-spend benchmark already established that a whole-corpus dump is 67× pricier than rlat skill-context constrain without changing the conclusion; it was not re-run here.
• Judge variance. Sonnet 4.6 grading itself; we apply the relaxed rubric. The strict-rubric numbers ship in the result JSON.
• One corpus. Numbers measured on resonance-lattice.rlat (this project's docs + src). The shape of the result should generalise; the exact magnitudes are corpus-specific.

Result artifact: primer_effectiveness.json

pip install rlat[bench]
rlat install-encoder
rlat build ./docs ./src -o resonance-lattice.rlat
rlat summary resonance-lattice.rlat -o .claude/resonance-context.md
export CLAUDE_API=sk-ant-...
python -m benchmarks.user_bench.primer_effectiveness.run \
  --km resonance-lattice.rlat \
  --primer .claude/resonance-context.md \
  --memory-primer .claude/memory-primer.md \
  --output benchmarks/results/user_bench/primer_effectiveness.json \
  --budget-usd 5

The world-knowledge evidence

v3's claim is that a knowledge model can carry validated knowledge about its own world — standing rules, environment facts, tried-and-falsified approaches — and that serving it measurably changes answers. The maintainer's rule for this section is no claim without a public receipt: every claim below links its full pre-registered design (committed before any result existed), its verdict, and the raw run artifacts. Failures are published with the same prominence as passes — the last subsection is one.

R1 — Standing constraints stop rule-violating answers

Claim: serving a user's handful of standing hard rules ("never preview features", "EU data residency only") eliminates answers that violate them, at zero collateral cost.

Design — paired within-item arms (blind / true constraint served / irrelevant-placebo served) on 24 Microsoft Fabric (constraint, question) pairs plus a 10-question collateral set for over-blocking; an item counts only if the blind answer actually violates. Pre-registered bars: served violations ≤ ⅓ of blind, placebo ≈ blind, collateral drop < 10pp.

Honest caveat: a post-merge review found the API judge's placebo reading is subset-sensitive — computed within that judge's own decisive subset, placebo lands at 7/10 vs a 100% blind reference, breaching the ±10pp form of the guard. The served-collapse and collateral bars are judge-robust; the placebo guard specifically is judge-sensitive. The claim stands on the pre-registered subscription primary, with that qualification on the record.

Receipts: design · verdict · items · run 1 · summary · API confirm · judge script

R1-X — The same effect in a garden and a law practice

Claim: the constraint effect is not a software artifact — the identical design works where the world is a home garden or a small NSW law practice, with no software anywhere.

Design — R1's arms plus a fourth (blind-2, a fresh resample) as a noise-aware placebo guard, on 12 items + 6 collateral questions per domain, both fictional worlds so nothing can leak into the blind arm. Bars evaluated per domain; the cross-domain claim requires a PASS in both.

Honest caveat: the placebo guard is judge-sensitive. Under the pre-registered subscription primary it passes cleanly (zero placebo flips in both domains); under the Haiku API judge it fails in both (garden 2 flips vs 0 blind-2, practice 2 vs 1). The constraint-specific effect — the claim — is decisive under both judges; whether an irrelevant served rule also nudges answers a little is judge-dependent and unresolved at this n.

Receipts: design · verdict · items · run 1 · partial (pre-resume) · API confirm · judge script

R2 — A falsification ledger stops dead-end re-recommendations

Claim: serving a project's tried-and-falsified results ("we tried X, it made things worse, here's the benchmark") stops an assistant from re-recommending dead ends — and the falsification verdict, not mere topical mention, is the active ingredient.

Design — four arms per item (blind / matched ledger atom / verdict-free topical mention of the same approach / irrelevant-ledger placebo) on 20 questions over 10 falsified approaches, plus 8 collateral questions answered with the full 10-atom ledger served. Bars: ledger ≤ ⅓ blind; topical − ledger ≥ 25pp; placebo within ±10pp of blind; collateral drop < 10pp.

The placebo breach was decomposed by a pre-registered follow-up (run 2b): a fresh blind resample flipped the same two items (q12, q18) with no note served at all — 2 blind-2 flips ≥ 2 placebo flips, so under the rule fixed before the data, the deviation is sampling noise and the headline stands as a PASS with the noise caveat recorded. An independent Haiku API judge re-scored all 64 transcripts and confirmed all four bars (ledger 0/7, gap 86pp, collateral 16/16). The 86pp gap between the verdict and a verdict-free mention is the largest contrast this program has measured.

Receipts: design (incl. run-1 invalidation) · verdict · run-1 items · run-2 items · run 1 (invalid) · run 2 · run 2b · API confirm · judge script

E2c — The capture gate: the extractor earns its wake-up

Claim: the passive extractor that captures world facts from what a user types meets precision, recall, privacy, and domain-neutrality bars on its real production path before it is allowed to run.

Design — 10 synthetic sessions of user-turn text (4 software, 3 garden, 3 legal) each carrying 1–2 ground-truth world facts and 2–4 traps (transient facts, discovered-not-stated facts, hypotheticals, quoted-assistant text, corpus facts, and person-facts — the privacy class). Grading is deterministic term-matching: zero judge noise. Bars: precision ≥ 0.83, recall ≥ 0.85, person-fact emissions = 0 (one leak fails the bench regardless of averages), every domain ≥ 0.75 precision.

Honest caveats: the three false positives are borderline-defensible captures, none personal (a hypothetical restated as present-tense, two corpus-stated facts relayed as operative constraints) — they pollute mildly, they don't leak. A post-run review found a grader ordering flaw that could in principle have hidden a compound-emission leak; the fixed grader re-scanned the committed run-1 emissions and still finds 0 leaks.

Receipts: design + verdict · sessions + ground truth · runner + grader · run 1

R4 — Auto-suppression: tested three ways, falsified

This is the honest-failure record. The claim under test — the loop can automatically retire its own wrong facts safely ("gets better with use") — was tested three ways and falsified each time. It is published here at the same prominence as the passes above, because that is what makes the passes worth believing.

Design — an instrumented baseline run persisted the full per-serve credit stream (n=31 items, 3 seeds, 8 rounds), then each candidate suppression rule was pre-registered before its replay or live run. Bars, fixed in advance: SAFE (zero true facts cut, every seed), EFFECTIVE (at least as many wrong facts cut as the prior best rule), all seeds individually.

The diagnosis, which is the durable finding: pooled per-fact credit is the wrong statistic — one true fact ("Windows PowerShell 5.1", 122 serves) pooled at a mean delta of −0.140, below two wrong facts, because the selector serves it heavily off-topic where even a true fact doesn't help. No threshold on that statistic can be simultaneously safe and effective; four rule families failed there for the same structural reason. And off-policy replay overestimates safety under selection feedback: live suppression changes the serving distribution, and a true fact was cut before it could accumulate its protecting on-item record — exactly why the live confirmation was pre-registered as mandatory.

What the cycle bought: the instrumented A/B independently replicated the loop's value-add (learning − no-learning = +0.074 mean, p_wilcoxon 0.031, paired) — the value is real; the safe self-cleaning rule is the unsolved part. The two committed per-serve streams (unbiased baseline + live r4c) make every future suppression rule a free, reproducible offline replay before it costs a dollar.

Receipts: design v1 · design v2 (incl. its verdict) · live-confirmation design + verdict · run-1 verdict · replay analyzer · replay run 1 · replay run 2 (v2) · instrumented stream · live confirm stream

Reproduce it yourself

git clone https://github.com/tenfingerseddy/resonance-lattice.git
cd resonance-lattice
pip install -e .[bench]
rlat install-encoder
rlat build ./docs ./src -o resonance-lattice.rlat

# Build & query speed (no API spend, deterministic)
python -m benchmarks.user_bench.build_query_speed.run

# Token usage
export CLAUDE_API=sk-ant-...
python -m benchmarks.user_bench.token_usage.run --budget-usd 16

# Hallucination (measured $8.93 total at full N=63 across all 11 lanes)
python -m benchmarks.user_bench.hallucination.run --budget-usd 12

Each run.py accepts --n-tasks to subset for pilots and --budget-usd to abort at a hard cost cap.

Where next