Reranker for Agent Zero

Reranks the documents Agent Zero recalls from memory using a cross-encoder reranker, so the most relevant memories rise to the top before they enter the prompt.

Agent Zero's memory recall is a single vector-similarity (embedding) search — fast, but lossy: it compresses each memory into one vector, so the ranking within the top results is often approximate. A cross-encoder reranker reads the query and each candidate together and scores true relevance, which reorders the candidates much more accurately. This plugin wraps the memory similarity search: it pulls a larger candidate pool, reranks it against your reranker endpoint, and returns the best matches first.

It works with two kinds of reranker server, and auto-detects which one you've pointed it at:

• Standard rerank servers — Cohere, Jina, Infinity, Text-Embeddings-Inference (TEI), and vLLM's own /v1/rerank. Any cross-encoder reranker that takes a query + documents and returns relevance scores.
• vLLM-served Qwen3-Reranker via /score — this one needs a client-side judge template that vLLM doesn't apply for you; the plugin handles that (see How it works).

So you can point it at a hosted rerank API or your own local reranker; set Reranker API to auto (default) and it figures out which protocol to speak.

Support

If this plugin is useful to you, you can support the developer.

What is a reranker? (new to the idea? start here)

When Agent Zero needs to "remember" something, it doesn't read all of its memories — that would be far too slow. Instead it does a fast approximate search and then puts the few best matches into the prompt. A reranker improves the quality of those few matches. Here's the problem it solves.

Stage 1 — fast search (what A0 does today): embeddings. Every memory is turned, ahead of time, into a single list of numbers (a "vector" or embedding) that captures its rough meaning. Your query is turned into a vector too, and A0 keeps the memories whose vectors are closest to the query's. This is fast — it can scan thousands of memories instantly — but it's lossy: squeezing a whole passage into one vector throws away detail, and the vector has to be a decent summary of the memory in general, with no idea what you'll later ask. So the results are "in the right ballpark," but the order within them is often wrong — the genuinely best memory might come back ranked 6th, while something that merely shares vocabulary lands on top.

Stage 2 — careful re-scoring: the reranker. A reranker is a different kind of model (a cross-encoder). Instead of comparing two pre-made summaries, it reads the query and one memory together, as a pair, and judges "does this memory actually answer this query?" Because it sees both at once, it catches the precise relevance — negation, qualifiers, the right entity — that the embedding flattened away. The catch: it's slower, because it can't precompute anything; it has to actually read each pair. So you'd never run it over thousands of memories.

Putting them together (what this plugin does). Use each model for what it's good at:

your query
   │
   ▼  Stage 1: embedding search  →  grab the top ~20 candidate memories   (fast, "recall")
   │
   ▼  Stage 2: reranker scores each (query, memory) pair  →  re-sort       (accurate, "precision")
   │
   ▼  the best few go into A0's prompt

Stage 1's job is recall — cheaply make sure the right memory is somewhere in the top 20. Stage 2's job is precision — read those 20 properly and float the genuinely-relevant ones to the top, so only the best reach the limited prompt space. Running the reranker on ~20 candidates is cheap; running it on everything would not be.

A concrete example. Ask "What changed in the March 1 upgrade?" Plain embedding search might return a memory about an unrelated connectivity problem near the top just because it shares words like "Agent Zero" and "March." The reranker, reading query-and-memory together, scores that one low ("doesn't answer the question") and promotes the actual upgrade notes — so the agent answers from the right memory.

The honest trade-off. Reranking adds a real (but optional, and tunable) bit of latency per recall, since the reranker has to read each candidate. It helps most when the first-stage search is imperfect — which is exactly the case for A0's plain embedding recall. Shadow mode (below) lets you turn it on without changing anything, watch it work, and only commit once you're happy.

What it does

• Reranks memory recall — transparently wraps Memory.search_similarity_threshold; every recall benefits, no agent action needed.
• Shadow mode — query the reranker but leave recall order unchanged, so you can confirm your endpoint works (and compare) before letting it change anything.
• Test connection — a button in the settings panel probes your endpoint live, auto-detects which API it speaks, and tells you whether it reranks correctly (before you save).
• Reranker status (reranker-status skill) — same check from chat: ask "is the reranker working / test the reranker / which reranker API" and the agent reports.
• Fail-safe — if the reranker is unreachable or errors, recall silently falls back to the original order. Reranking can never break memory.

Setup

1. Have a reranker server — either a standard/hosted rerank API (Cohere, Jina, Infinity, TEI, or vLLM /v1/rerank), or a local vLLM Qwen3-Reranker over /score, e.g.:

   vllm serve <path-to>/Qwen3-Reranker-8B --served-model-name a0-reranker --port 8002 \
     --runner pooling --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"method":"from_2_way_softmax"}'
   

2. Install this plugin (Plugin Hub, GitHub URL, or upload the ZIP) and enable it in Plugins.
3. Configure the endpoint: Settings → Agent → Reranker — set Reranker endpoint to your server's base URL (e.g. http://YOUR_HOST:8002; the plugin appends the API path). Leave Reranker API on auto and it detects the protocol. Until an endpoint is set, the plugin stays inert.
4. Verify it works: click Test connection in the config panel — it probes your endpoint and confirms it reranks correctly. (Optional: start with Apply reranked order off to run in shadow first, then flip it on once you're happy.)
5. Click Save. Test checks the values shown in the form so you can verify before committing — it does not save them. Until you hit Save, the plugin stays inert (a green Test will remind you). Every Test click probes the endpoint live, so a fixed URL tests correctly immediately.

How it works

The plugin speaks one of two reranker-server protocols (set Reranker API, default auto):

• rerank — the standard cross-encoder rerank API (/v1/rerank or /rerank): it sends the query plus the candidate documents and reads a relevance score per document. This is what Cohere, Jina, Infinity, TEI, and vLLM's /v1/rerank all speak, and the server applies whatever prompt the model needs.
• qwen3_score (client-templated /score) — for a raw scoring endpoint that doesn't apply the model's own prompt, so the plugin supplies it. The common case is a vLLM-served Qwen3-Reranker: it scores relevance by answering a yes/no judge prompt (P(yes) over the [no, yes] tokens), but vLLM's /score and /v1/rerank feed the model raw query + document text without that judge template — so the model never reaches the yes/no decision token and the scores are meaningless. This mode renders the judge template client-side and posts it to /score, which makes it rank correctly.

Auto-detect can't simply ask "does /rerank respond?" — a vLLM Qwen3 endpoint does answer /v1/rerank, just with garbage scores (the missing-template bug). So auto runs a tiny correctness probe: it sends a known relevant/irrelevant pair and picks whichever API ranks the relevant document above the irrelevant one. The result is cached per endpoint.

Programmatic API (for plugin authors)

Out of the box this plugin reranks memory recall. If you're writing another plugin that has its own candidate set to rerank, don't re-implement the broken-/v1/rerank workaround — defer to this plugin as the single authoritative reranker. It exposes one callable:

# only when a0_reranker is installed; degrade gracefully when it isn't
from importlib.util import find_spec
if find_spec("usr.plugins.a0_reranker.helpers.reranker"):
    from usr.plugins.a0_reranker.helpers import reranker
    pairs = reranker.rerank(query, documents)   # documents: list[str]
    # -> [(orig_index, score), ...] sorted best-first, or None if unavailable
    if pairs is not None:
        order = [documents[i] for i, _ in pairs]

rerank(query, documents, *, config=None) returns [(orig_index, score), …] (best-first; orig_index is the position in your input list) or None on any failure / no endpoint configured. It uses the same instruction-tuned scoring as the memory path (the qwen3_score judge template + the auto correctness probe), reading this plugin's configured endpoint by default — so the user configures the reranker once and every consumer benefits. It is synchronous + blocking (call it from a thread in an async context); it never raises into your code (None = "reranker unavailable, keep your prior order").

Configuration

Set these in Settings → Agent → Reranker (or in default_config.yaml; per-project overrides supported). Quick reference:

What each setting does (and how it affects quality)

Behaviour

• Enabled (rerank_enabled) — the master on/off. Off means memory recall behaves exactly like stock A0 (plain embedding order) and the reranker is never called — zero added latency. Use it to pause reranking without uninstalling.
• Apply reranked order (rerank_apply) — on = live: the agent actually receives the reranked memories (this is the point of the plugin). off = shadow: the reranker is still queried but its result is discarded and recall order is left unchanged. Shadow is a verify/safety mode — note it still costs the same per-recall latency, so don't leave it on long-term; use it (or the Test connection button) to confirm your endpoint works, then turn Apply on.

Connection (these are correctness, not quality knobs — get them right and the reranker works)

• Reranker API (rerank_api) — which protocol your server speaks. auto probes and picks the right one and is almost always correct. Forcing the wrong mode breaks scoring (e.g. forcing rerank at a vLLM-Qwen3 endpoint returns garbage — that's the exact trap auto's correctness probe avoids). Only set it manually if auto-detection ever guesses wrong.
• Reranker endpoint (rerank_endpoint) — your server's base URL. Required; the plugin appends the API path (/v1/rerank, /rerank, or /score). Must be reachable from the A0 container.
• Model id (rerank_model) — the model name your server expects (your vLLM --served-model-name, or e.g. rerank-english-v3.0 for Cohere, jina-reranker-v2-... for Jina). Wrong id = an error or the wrong model. The model you choose is the single biggest quality lever — a stronger/larger reranker ranks better than a small one.

Quality & performance levers

• Candidate pool (rerank_candidates) — how many of the top embedding-search results get reranked. This is the main quality/latency dial. Larger = the reranker sees more candidates and can rescue a genuinely-relevant memory that the vector search ranked low (say position 15) — more thorough, but more (query, doc) pairs to score, so more latency per recall. Too small limits how much it can fix, and a very shallow pool can even slightly hurt (it shuffles the top few without reaching the ones worth rescuing). ~12–20 is a good range; the default is 20. (A0's recall asks for a small limit; the plugin always reranks at least this pool, then returns the top limit.)
• Instruct line (rerank_instruction) — Qwen3-Reranker is instruction-aware: this line tells it what kind of relevance to judge. It sits in the judge prompt above the query and document ("Judge whether the Document meets the requirements based on the Query and the Instruct provided… <Instruct>: <this line>"), so it frames the yes/no question the model answers for each memory. The default ("Given a web search query, retrieve relevant passages that answer the query") is Qwen3-Reranker's standard generic instruction and is a sensible fit for memory recall — leave it unless you want to tune relevance for a specific domain (a task-matched instruction, e.g. "Given the user's request, retrieve past memories and facts relevant to it", can nudge ranking quality). Two caveats:
  1. It is used only in qwen3_score mode (it is part of the Qwen3-Reranker judge template). In the standard rerank mode (Cohere / Jina / Infinity / TEI), it is ignored — those servers don't take a per-request instruction this way.
  2. It is Qwen3-specific, so it only matters if you're on qwen3_score (or auto resolved to it). On any other reranker, changing it has no effect.
• Max document characters (rerank_max_doc_chars) — how much of each memory's text is fed to the reranker. Higher = better judgments on long memories (the reranker reads more context; truncating too aggressively measurably hurts ranking quality), but more tokens per pair = more latency (and more GPU load). The default 8000 is effectively "full text" for typical memory chunks, which are short. Lower it only if you store very long memories and need to bound latency, accepting some quality loss.
• Timeout (rerank_timeout) — how long to wait for the reranker before giving up. On timeout (or any error) recall falls back to the original embedding order — reranking never blocks or breaks memory. Raise it if your reranker is slow or cold-starts (large model on modest hardware); lower it to fail fast. Too low a timeout on a slow reranker means frequent fallback, so you quietly lose the benefit without an error.

Uninstalling

Uninstall through the Plugins UI for a clean removal — the uninstall hook unwraps the memory search method, leaving no trace. The plugin installs nothing on the system and stores no data.

License

MIT — see LICENSE.

Citing

@software{a0_reranker,
  author  = {King0James0},
  title   = {Reranker for Agent Zero},
  year    = {2026},
  url     = {https://github.com/King0James0/a0-reranker},
  license = {MIT}
}