Agent Zero v2.11: Context Doctor, Vision Sidecars, & Faster Streaming

v2.11 adds Context Doctor, vision sidecars and a context usage indicator, makes streaming and WebUI synchronization faster, improves memory and embeddings, and defaults LiteLLM providers to Chat Completions.

Agent Zero
#releases

Agent Zero v2.11 changelog banner

🧩 New Plugins & Extension Points

  • See context use before it runs out — The Context Window Usage Indicator adds a composer ring and responsive breakdown with token counts, free space, provider pricing, cache-hit details, and independent mobile and desktop controls.
  • Repair malformed model output automatically — Context Doctor fixes tool-call JSON before dispatch, handles incomplete or thoughts-only responses, suppresses XML when needed, and preserves repaired output in WebUI logs.
  • Use vision sidecars per preset — Vision Sidecar Integration adds a strict sidecar slot with Main-first routing, one-call multi-image analysis, parallel-worker image resolution, and inline timeout and token controls.
  • Extend completed model results — A new result extension point lets extensions normalize or consume results before default history and tool processing. Repeat detection and empty-response handling move into configurable loop-control extensions.
  • Add responsive WebUI controls — Plugin slots are now available beside the model/profile strip and at the end of interface visibility controls.

⚡ Streaming & WebUI Performance

  • Streaming stays formatted — Live Responses now render safe Markdown and LaTeX throughout, while native response-tool output streams progressively through the canonical tool envelope.
  • Less work per token — Full-response parsing is coalesced behind a 128-character or 50-millisecond cadence without skipping per-chunk security checks or strict tool validation.
  • Stable live interactions — Message actions keep their hover, focus, and extension controls during streamed updates; native reasoning summaries become thoughts after streaming finishes.
  • Quicker real-time synchronization — Collection deltas, stable sidebar row identity, bounded log fields, deferred formatting, and capped process updates reduce WebUI churn. First user turns render optimistically and serialized state pushes keep Welcome messages visible.
  • Safer chat branches — Branched chats rebuild inherited Responses continuation, response-ID ownership, and cached context from trimmed local history.

🧠 Memory, Embeddings & Context

  • Gateway embedding IDs stay intact — OpenAI-compatible embedding calls retain provider-prefixed model IDs, matching chat-wrapper behavior.
  • Memory scores serialize reliably — NumPy relevance scalars become native floats at the source, while legacy conversion stays limited to the memory dashboard response.
  • Local models initialize once — SentenceTransformer models are cached by constructor options so preload and first-chat initialization share the same instance.
  • Context crosses fewer boundaries — Compacted summaries no longer look like orphaned assistant turns; settings snapshots, prompt JSON, parallel results, and job handles are scoped or serialized compactly to save work and tokens.

🎛️ Models & Interactive Browser

  • Provider accounting reaches the UI — Chat Completions and Responses preserve provider-reported token usage and cost, including terminal streaming chunks, without inventing absent accounting.
  • Chat Completions are now the default transport — OpenAI, Azure, GitHub Copilot, and GitHub Copilot OAuth explicitly use Chat Completions; Codex/ChatGPT and xAI Grok OAuth stay on Responses. New providers must opt in to Responses when needed.
  • Non-US browser keyboards work correctly — Configurable keyboard_layout and keyboard_variant settings apply XKB layouts to the private display and Xpra shadow server, including AltGraph and macOS Option input fallback behavior.

🛠️ Reliability Fixes

  • Small UI fixes add up — The mobile canvas rail can truly hide, onboarding closes before opening model presets, empty vision skip summaries stay hidden, and the ACP thumbnail uses its original artwork again.
  • Updates finish their work — Context Doctor installs its json_repair dependency after self-update without a rebuilt image, and Launcher README links now point to v1.6.

⚠️ Breaking Change

LiteLLM now defaults to Chat Completions. Providers that relied on the implicit Responses transport must explicitly select Responses; OpenAI, Azure, and GitHub Copilot remain unaffected because they are explicitly configured.