Jev Search reranking Agent memory Community case ↗ 2026-09-19

Agent memory reranking: Jev vs a free local cross-encoder

A community project measured Jev as a memory reranker: Recall@1 went 0.41 → 0.62 with rigorous statistics — and they publish the three graded tests where answers did NOT improve. What Jev actually bought was context compression: 2 memories ranked by Jev answered as well as 5.

Recall@1

0.41 → 0.62

+0.21 R@1

Cross-encoder baseline vs Jev reranker, private 300-query developer store

Free local cross-encoderRecall@1 0.41

Jev reranker (opt-in)Recall@1 0.62

300 queries

Query store

private developer store

20/20 seeds

Significance

2000-draw paired bootstrap; permutation null 0/200

$0.0004

Cost per recall

paid reranker on a free path

2 vs 5

Context needed

memories for the same answer rate

TL;DR

▸hippo-memory is a SQLite-backed memory layer for AI agents. Its recall is BM25-only (R@5 74% on LongMemEval, no embeddings) — the question was whether a Jev reranker on top makes answers better, or just ranks prettier.
▸Ranking: Recall@1 0.41 → 0.62. The margin held in 20 of 20 bootstrap seeds; a permutation null reached it in 0 of 200 runs. This is the most statistically careful Jev number we have seen published.
▸The honest part: three graded answer tests showed NO improvement over the free cross-encoder. The real win is context compression — 2 Jev-ranked memories answered as well as 5 cross-encoder-ranked, which means cheaper, faster downstream LLM calls.
▸The feature ships off by default ($0.0004 per recall adds a paid call to a free path), and the failed tests are published in the same doc as the wins.

The task

hippo-memory (756★ at writing) gives coding agents a persistent, decaying, provenance-tracked memory store that survives tool switches — same .hippo/ store whether you run Claude Code, Cursor or Codex. Recall is deliberately cheap: BM25 over stored memories, no embedding API.

The maintainer's 'Receipts' discipline is what makes this case worth curating: every measurement the project has ever published is indexed in docs/evals/, pre-registrations sit next to results, and one earlier claim was formally retracted (v0.11.0 informal magnitude, RETRACTED at v1.7.9, mechanism still shipped).

The Jev reranker question is exactly the System One shape: given a query and 20 candidate memories, rank the few that matter — a Score-style judgment per candidate, thresholds in code, no generated prose.

How each model was used

Cross-encoder baseline

·Free, local, offline — the default path costs nothing
·Reranks BM25 candidates and passes the top 5 memories into context
·Answer rate on graded tests is the project's quality bar

Jev reranker (opt-in)

·`hippo recall "<query>" --reranker jev` — Jev reranks the candidate set
·Top 2 memories passed into context instead of 5
·Versioned model ID logged per response; feature off by default
·~$0.0004 per recall at early-access pricing

Results

Metric	Free local cross-encoder	Jev reranker (opt-in)
Recall@1	0.41	0.62
Memories needed for same answer rate	5	2
Graded answer tests (3 tests)	cross-encoder level	no gain — published
Cost per recall	$0 (local)	$0.0004
Statistical check	—	20/20 seeds · null 0/200

What moved the needle

Ranking improved, answers did not — and both facts are published

On three graded answer tests, Jev-ranked context answered no better than cross-encoder-ranked context. The maintainer's conclusion: what Jev buys today is a shorter context, not a smarter answer.

Context compression is the real economics

If 2 ranked memories answer as well as 5, every downstream LLM call gets cheaper and faster — the reranker pays for itself in the generation step it feeds.

The rigor is the differentiator

A 2000-draw paired bootstrap with 20/20 seed stability and a permutation null is more statistical care than most vendor launch posts manage; it is the standard reader-submitted cases should aim at.

What didn't work & limits

✕No answer-rate gain: the reranker improves ranking and context efficiency, not answer quality, on the tests run so far.
✕Private 300-query developer store — not reproduced on a public benchmark; your mileage will vary with your memory corpus.
✕Adds a paid network call to a path that is free and offline by default; that is why it ships opt-in.
✕Jev is English-primary per TypeSafe's own guidance; multilingual memory corpora are untested here.

Reproduce it

Templates for the same pattern — copy, point them at your own state, and keep the thresholds in code.

Jev 91 words

Lead Qualification as Composite Questions

Official best practice: score each axis independently, weight them in code.

state = { lead: { company_size, signup_source, pages_visited_7d, trial_features_used } }

questions:
- icp_fit: Score("How well the company matches our ideal customer profile")
    ["Outside ICP", "Adjacent market", "Core ICP"]
- activation: Score("How engaged the account was during trial")
    ["Signed up only", "Explored lightly", "Used the core feature repeatedly"]
- handoff: Choice("Next action")
    sdr_call: "Worth a human call this week"
    nurture:  "Keep in the email sequence"
    other:    "Not enough signal yet"

Combine in code, not in the model: handoff = sdr_call is respected only when
icp_fit.score >= 1.5 AND handoff.confidence > 0.6.

Jev 65 words

Search Reranking on a Budget

Filter 20 search hits down to the 3 worth an LLM's attention.

state = { query: "the user's original question", candidate: { title, snippet } }

questions:
- relevant: Noul("The candidate page actually answers this query")
- credible: Noul("The source looks authoritative rather than spam")

Run once per candidate. Keep results with relevant > 0.6 AND credible > 0.5,
drop the rest before anything expensive sees them (output tokens are free;
input is ~$0.04 per million tokens).