Agent Zero v1.17: Computer Use Remote, Visual Verification, & Platform Targeting
v1.17 turns Computer Use Remote into a fuller host desktop-control pipeline, with direct tool access, screenshot-backed verification, multimodal captures, platform-aware targeting, and clearer safety behavior when host permissions are missing.
computer_use_remote is callable in live sessions โ The model can now invoke the remote computer-use tool directly, while availability, trust mode, and re-arm enforcement remain runtime checks.
Host desktop control is separated from Xpra โ computer_use_remote is now the sole path for controlling the user's host desktop. linux-desktop targets only Agent Zero's internal Docker/Xpra desktop.
Host-screen requests route more accurately โ Host desktop queries rank ahead of the Xpra skill, while explicit Agent Zero Desktop requests continue to target the internal desktop.
๐๏ธ Visual Verification
Fresh screenshots are required after desktop actions โ State-changing desktop actions are considered unverified until a new screenshot visibly confirms the result.
Agents stop when screenshots are unavailable โ If a verification screenshot cannot be captured, the agent must stop instead of guessing or continuing blindly.
Screenshots return as multimodal results โ Computer-use captures now arrive as real vision messages, allowing the model to inspect the screen visually after each action.
Older captures are pruned โ Stale capture payloads are removed to keep long desktop-control sessions from growing context without bound.
๐งญ Platform Targeting
macOS Accessibility targeting โ A dedicated macOS skill supports Accessibility structural targeting through ax_snapshot and ax_action when the backend reports those capabilities.
Windows UIA targeting โ Windows gains UI Automation guidance for window management, selector passthrough, and click-last workflows.
Linux AT-SPI and Wayland targeting โ Linux snapshots include compact structural tree outlines so agents can select semantic targets more reliably.
Generic prompts stay backend-neutral โ Backend-specific action details live in platform skills, while the generic layer focuses on capability discovery and skill loading.
๐ Permissions & Safety
macOS approval denial handled cleanly โ COMPUTER_USE_APPROVAL_REQUIRED now maps to the existing re-arm-required stop flow, preventing repeated retries or screenshot fallbacks before permission is granted.
Window-hide guidance updated โ Ubuntu, GNOME, and Wayland sessions now prefer Super+H over Alt+F9.
Keystroke verification is clearer โ Guidance now reminds agents that a sent key chord proves only that the keys were sent, not that the requested window action succeeded.
๐ง Vision & Context
Codex OAuth proxy preserves image inputs โ Image content parts are converted to Responses API input_image parts instead of being flattened to text.
Multimodal regression coverage added โ Screenshot-bearing tool results now have coverage to protect the vision path.
Screenshot token estimates fixed โ Embedded base64 image data URLs are sanitized from prompt token estimates so screenshots no longer inflate context budgets.
We use essential cookies for login and saving your preferences. With your consent, we also
use analytics cookies (Google Analytics) to improve the website.
Learn more