Ghost Judge plugin thumbnail

Ghost Judge

Silent LLM quality gate that evaluates your agent's work with a separate judge model. Set a /goal and the judge ensures completeness before presenting results. Overclock any model.

Author Kironkeys 0 stars Version 1.0.0 Updated
WorkflowToolsAgentsDevelopmentAutomation
GitHub Repository Discussion How to install

README

Ghost Judge

Your agent says "done." But is it?

Ghost Judge is a silent quality gate for Agent Zero. It uses a separate LLM to evaluate your agent's work after every response — and if the work isn't complete, your agent keeps refining automatically. No manual re-prompting. No babysitting.

Think of it as overclocking your LLM. A $2/M model with Ghost Judge produces output that rivals a $15/M model — because the judge forces thoroughness, evidence-backing, and honest gap reporting that cheaper models skip on their own.

The capability was always there. The accountability wasn't.

What It Does

  1. You set a /goal — a standing objective with measurable criteria
  2. Your agent works toward it
  3. After each response, Ghost Judge silently sends the conversation to a separate judge model
  4. The judge evaluates: done or not done? If not done, it tells your agent exactly what's missing
  5. Your agent refines and tries again — automatically
  6. When the judge says done, the result is presented to you

You see one polished response. Behind the scenes, the judge may have sent your agent back 2-3 times to fill gaps, cross-reference sources, or fix lazy shortcuts.

No goal set = zero cost. The judge only fires when you activate it.

Why This Exists

Every LLM agent has the same problem: it declares "done" based on its own assessment. No second opinion. No verification. The agent that built the report is the same one grading it.

This leads to:

  • The 75% problem — the agent delivers most of the work but misses polish, edge cases, or completeness
  • Lazy delegation — "I dispatched a background task, I'll show you later" instead of presenting actual results
  • Single-source claims — presenting data from one source as verified fact
  • Identity merging — in OSINT work, combining data from different people into one profile
  • Missing cross-references — not checking if claims from source A contradict source B

Ghost Judge catches all of these by having a separate model — one that didn't do the work — evaluate whether the work is actually done.

The Turbo Analogy

A 1000hp engine with bald tires spins out. A 500hp engine with traction control puts every bit of power to the ground.

Ghost Judge is traction control for your LLM:

  • Cheap model + Ghost Judge = output quality that rivals expensive models
  • Expensive model + Ghost Judge = output quality that's untouchable
  • Any model without Ghost Judge = hoping the agent doesn't cut corners

The judge costs ~$0.03 per evaluation. The quality improvement is night and day.

Installation

Copy the plugin into your Agent Zero usr/plugins/ directory:

cp -r ghost-judge /path/to/agent-zero/usr/plugins/ghost_judge

Requirements:

  • Agent Zero v1.13+
  • An OpenRouter API key (set OPENROUTER_API_KEY in your environment)
  • The Commands plugin for /goal and /subgoal slash commands

No container restart needed — the extension auto-loads.

Configuration

default_config.yaml:

enabled: true
max_turns: 10                              # max refinement passes before auto-pause
judge_model: "x-ai/grok-4-3"              # any OpenRouter model works
judge_provider: "openrouter"
judge_reasoning_effort: "low"              # low/medium/high (for reasoning models)
judge_timeout: 30                          # seconds
judge_max_tokens: 4096
max_consecutive_parse_failures: 3          # auto-pause if judge can't return valid JSON
auto_detect_domain: true                   # classify task domain from tool usage
default_domain: "default"                  # fallback when no tools match

Model Recommendations

Model Cost Best For
Grok 4.3 (default) $1.25/M in Reasoning-based evaluation, catches subtle gaps
Claude Haiku 4.5 $1.00/M in Fast structured evaluation, strict JSON compliance
GPT-4.1 mini $0.40/M in Budget setups, still effective for basic quality checks
Gemini Flash $0.15/M in Ultra-budget, good for simple done/not-done checks

The judge model should be different from your main agent model. The whole point is independent evaluation.

Usage

Set a goal

/goal Build a complete research report on Square Roots with co-founders, funding history, and sources for every claim

Your agent starts working immediately. The judge evaluates after each response.

Add criteria mid-task

/subgoal cross-reference funding data from at least 3 sources
/subgoal include 2025-2026 news

The judge switches to an evidence-gated template — it must find concrete proof for each subgoal in the agent's output. No generic "all done" accepted.

Check status

/goal

Shows: current goal, turns used, last verdict, last reason.

Stop early

/goal clear

Domain-Aware Evaluation

Ghost Judge auto-detects what type of task your agent is doing based on which tools were called:

Tools Used Domain What the Judge Checks
sherlock, holehe, pdl_enrich OSINT Identity vectors, cross-referencing, source diversity, data conflicts
page_builder, render_ui Design Brand consistency, mobile layout, CTA, placeholder cleanup
deep_wiki, document_query Research Source diversity, evidence for claims, gap acknowledgment, synthesis
Anything else Default General completeness check

Each domain has its own evaluation criteria in prompts/judge.{domain}.md. You can add custom domains by creating new prompt files and adding tool mappings in helpers/domain_classifier.py.

OSINT Domain Criteria

The OSINT judge checks 7 intelligence vectors:

  • Identity (name, aliases, DOB)
  • Contact (phone, email)
  • Social media (profiles, cross-referenced)
  • Location (address, history)
  • Employment (current/recent)
  • Digital footprint (username reuse)
  • Public records

A dossier with fewer than 4 populated vectors is NOT DONE.

Design Domain Criteria

  • Brand consistency (colors, fonts, tone)
  • Section completeness (hero, CTA, content blocks)
  • Mobile responsiveness
  • Real assets used where available
  • No dead links or placeholder text

Research Domain Criteria

  • Multiple sources cited
  • Claims backed by evidence
  • Gaps explicitly acknowledged
  • Findings synthesized, not just listed
  • Actionable conclusions

How It Works Under the Hood

User: /goal research John Smith

Agent works → produces response
                    ↓
    ┌─────────────────────────────────┐
    │  Ghost Judge (monologue_end)    │
    │                                 │
    │  1. Read goal state (JSON file) │
    │  2. No goal? → exit (zero cost) │
    │  3. Classify domain from tools  │
    │  4. Build prompt (base + domain)│
    │  5. Send to judge model (Grok)  │
    │  6. Parse verdict JSON          │
    │                                 │
    │  Done? → mark complete, exit    │
    │  Not done? → inject feedback    │
    │         → agent refines         │
    │         → judge re-evaluates    │
    └─────────────────────────────────┘
                    ↓
User sees: polished, complete result

Key Design Decisions

  • Fail-open: If the judge API fails, your agent keeps working. A broken judge never blocks your agent.
  • Turn budget: Default 10 turns max. Prevents infinite refinement loops.
  • Parse failure tracking: If the judge model returns garbage 3 times in a row, it auto-pauses and tells you to check your model config.
  • Full conversation context: The judge sees the entire conversation, not just the last response. This catches cross-turn gaps like "you found this data 3 turns ago but didn't include it in your summary."
  • Subgoal evidence-gating: When subgoals exist, the judge demands specific evidence per criterion — file excerpts, data points, command results. No generic "all requirements met."

File Structure

ghost_judge/
├── plugin.yaml                     # Plugin metadata
├── default_config.yaml             # Judge model, turn budget, domain settings
├── __init__.py
├── commands/
│   ├── goal.command.yaml           # /goal slash command config
│   ├── goal.py                     # Goal set/status/clear/pause/resume
│   ├── subgoal.command.yaml        # /subgoal slash command config
│   └── subgoal.py                  # Subgoal add/remove/clear
├── extensions/
│   └── python/
│       └── _functions/
│           └── agent/AgentContext/communicate/end/
│               └── _45_quality_gate.py   # The core hook
├── helpers/
│   ├── __init__.py
│   ├── domain_classifier.py        # Tool names → domain mapping
│   ├── goal_state.py               # GoalState dataclass + GoalStore
│   ├── judge_client.py             # OpenRouter API call + JSON parsing
│   ├── prompt_builder.py           # Judge + continuation prompt builders
│   └── quality_gate.py             # Main judge logic
└── prompts/
    ├── judge.system.md             # Base judge system prompt
    ├── judge.default.md            # Generic evaluation criteria
    ├── judge.osint.md              # OSINT-specific criteria
    ├── judge.design.md             # Design-specific criteria
    └── judge.research.md           # Research-specific criteria

Adding Custom Domains

  1. Create a prompt file: prompts/judge.{your_domain}.md
  2. Add tool mappings in helpers/domain_classifier.py:
DOMAIN_TOOLS = {
    "osint": {"sherlock", "holehe", "pdl_enrich"},
    "design": {"page_builder", "render_ui"},
    "research": {"deep_wiki", "document_query"},
    "your_domain": {"your_tool_1", "your_tool_2"},  # add this
}
  1. The judge will auto-classify and use your domain prompt when those tools are called.

Real-World Test Results

Tested on live Agent Zero instances with GPT-5.4 as the main model and Grok 4.3 as the judge:

OSINT: Person Research

  • Without judge: Agent dispatched a background task and said "I'll show you later"
  • With judge: Judge caught the lazy response → agent produced a full cross-referenced dossier with 5 companies verified, 3 social profiles with URLs, source links, and confidence grades

Research: Company Deep Dive

  • Without judge: Agent delivered a report with single-source funding data
  • With judge: Judge caught weak sourcing → agent ran 6 additional targeted searches and cross-referenced from TechCrunch, AgFunderNews, VentureBeat, PitchBook, and Crunchbase

OSINT: Property Owner Resolution (Benchmark)

  • Without judge: Agent merged the property owner with an unrelated person (identity confusion)
  • With judge: Agent correctly resolved a trust entity, found the owner's trucking business at the same address, separated personal vs business phone numbers, and flagged identity merging risks

Cross-Surface Coherence

  • Without judge: Agent opened panels and declared "passed" without doing real work
  • With judge: Agent chained real data across Properties → Designs → Browser → Desktop → Chat, creating a document with evidence from every surface

Prior Art & Inspiration

Ghost Judge draws from two implementations:

  • Claude Code /goal — Anthropic's session-scoped stop hook that sends the full conversation to a small fast model after every turn. Ghost Judge adds domain-aware prompts and subgoal evidence-gating.
  • Hermes Agent /goal (Nous Research) — GoalManager with subgoals, parse failure tracking, and structured continuation prompts. Ghost Judge ports the subgoal evidence-gating and fail-safe patterns to Agent Zero's plugin architecture.

Community Testing — We Want Your Results

We built and tested Ghost Judge on cloud-hosted models (GPT-5.4, Grok 4.3). But the real question is: how far can this push local LLMs?

If you're running Agent Zero with local models (Llama, Qwen, Mistral, Gemma, DeepSeek, Phi) — we want to hear from you:

  • Does Ghost Judge make your 7B model perform like a 70B?
  • Which local model works best as the judge?
  • What domains benefit most from the quality gate?
  • How does reasoning effort affect local model judge accuracy?
  • Can a small local judge (3B) effectively gate a larger agent (70B)?

Open an issue or discussion with your test results. Include your agent model, judge model, task type, and before/after quality comparison. We'll compile community benchmarks and credit every contributor.

The thesis: a $0 local judge can turn a $0 local model into something that competes with $15/M cloud models. Help us prove it.

License

MIT — use it, fork it, improve it, ship it.

Credits

Built by kironkeyz. Architecture by Claude. Implementation by GPT. Judge model by Grok.

The whole point of this plugin is that no single model does its best work alone. Turns out that's true for building it too.