Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents

---

Overview

We propose a set of inference-time self-improvement techniques that enable computer-use agents to detect and recover from failures during task execution — without any additional training. Concretely, our system wraps an existing vision-language model–based agent (e.g., OpenCUA-72B) with four groups of lightweight runtime modules, applied incrementally:

Module	Description
Visual Search	Before executing a click, the agent crops the target region and re-examines whether the coordinate is correct (coordinate verification). Optionally, the target point is visually annotated on the cropped image to aid the verifier (marking verification).
Repetition Detection	Sliding-window checks flag when the agent is issuing semantically or syntactically identical actions/code repeatedly, or when the screen state has not changed across several consecutive steps.
Terminal Execution	The agent is able to activate terminal via special hotkey to perform system-level operations.
Knowledge Support	The agent can invoke a computer.search tool to query an advanced LLM for external knowledge or command syntax. Application-specific keyboard shortcuts for LibreOffice Calc / Writer are also injected when relevant.

All experiments are conducted on OSWorld.

---

Installation

1. VM Setup

Follow the official OSWorld instructions to set up a VMware, VirtualBox, or Docker environment:

https://github.com/xlang-ai/OSWorld

2. Python Environment Setup

conda create -n osworld_gui python=3.10 -y
conda activate osworld_gui
pip install -r requirements.txt

---

Running Experiments

Quick Start

Fill in your API keys in run_opencua.sh, then run:

bash run_opencua.sh

API Keys

Open run_opencua.sh and replace the placeholder values:

export OPENCUA_URL=<% PUT-YOUR-OPENAI-URL-HERE %>      # Endpoint URL for the OpenCUA model
export OPENCUA_API_KEY=<% PUT-YOUR-OPENCUA-KEY-HERE %>  # API key for the OpenCUA model
export OPENAI_API_KEY=<% PUT-YOUR-OPENAI-KEY-HERE %>    # OpenAI key (used by the search tool)

Ablation: Disabling Individual Tricks

Each inference-time module can be independently disabled via command-line flags, which is useful for ablation studies. All tricks are enabled by default; uncomment the relevant flag in run_opencua.sh to turn one off:

Visual Search

Flag	What it disables
--disable_coordinate_verification	The agent will no longer crop the click target and ask the model to verify/adjust the coordinate before executing.
--disable_marking_verification	When coordinate verification is active, the target point is not visually marked on the cropped image shown to the verifier.

Repetition Detection

Flag	What it disables
--disable_auto_recovery	Disables action repetition, code repetition, and screen-change detection.
--disable_action_repetition	Disables the check that detects when the agent's action description is semantically identical across recent steps.
--disable_code_repetition	Disables the check that detects when the agent emits the exact same PyAutoGUI code multiple times in a row.
--disable_screen_change_detection	Disables detection of a stalled screen state (i.e., the pixel hash / accessibility-tree hash has not changed for the last 3 steps).

Terminal Execution

Flag	What it disables
--disable_terminal	Removes the terminal usage reminder from the agent's context.

Knowledge Support

Flag	What it disables
--disable_search_tool	Removes the computer.search tool from the agent's action space; the agent can no longer query external knowledge during execution.
--disable_libreoffice_hints	Disables knowledge of LibreOffice-specific keyboard shortcuts when the active application is Calc or Writer.