Web Skill Factory: evolving reusable, verified, code-native skills for web agents

.github/workflows/skills-tests.yml

@@ -0,0 +1,28 @@
+name: skills-tests
+on:
+  push:
+    paths: ["src/webwright/skill_factory/**", "src/webwright/tools/skill_use.py", "tests/skill_factory/**"]
+  pull_request:
+    paths: ["src/webwright/skill_factory/**", "src/webwright/tools/skill_use.py", "tests/skill_factory/**"]
+jobs:
+  unit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.12" }
+      - run: pip install httpx pyyaml jinja2 pydantic
+      - name: skills unit tests (LLM-free)
+        run: |
+          set -e
+          for t in tests/skill_factory/test_*.py; do
+            echo "== $t"; PYTHONPATH=src python "$t"
+          done
+      - name: wrapper usage check (F6)
+        run: |
+          set +e
+          bash src/webwright/skill_factory/examples/solve_with_library.sh > /dev/null 2>&1
+          [ $? -eq 1 ] && echo "usage-exit OK" || { echo "wrapper must exit 1 on missing args"; exit 1; }
+          bash -n src/webwright/skill_factory/examples/quickstart.sh || exit 1
+          bash src/webwright/skill_factory/examples/quickstart.sh badmode > /dev/null 2>&1
+          [ $? -eq 1 ] && echo "quickstart usage-exit OK" || { echo "quickstart must exit 1 on bad mode"; exit 1; }

README.md

@@ -165,6 +165,25 @@ python assets/task_showcase/app.py \
 
 ---
 
+## 🧠 Skill Library (reuse solved tasks across tasks)
+
+[`webwright.skills`](src/webwright/skills/) turns solved tasks into **reusable, executable code
+skills**, retrieves and judges them at solve time, gates what enters the library, and grows the
+library incrementally — a self-evolving *store → retrieve → use/adapt → gate → evolve* loop on top
+of Webwright's code-as-action solves. Plugs in with **no change to the agent loop**:
+
+- **Reuse** — the agent calls `python -m webwright.tools.skill_use --task "..." --library ...`
+  (like `self_reflection`/`image_qa`); it returns `{verdict: use|adapt|skip, source_path}`.
+- **Grow** — `python -m webwright.skills.update --manifest batch.json --library ./library`
+  distills a batch of gate-passed solves into a parameterized, primitive-decomposed skill.
+
+Validated end-to-end on a real public website (read-only GitHub): solve two repos from scratch →
+`update` builds a parameterized skill → a held-out repo is solved by reusing it (agent calls
+`skill_use`, verdict `use`, answer correct); a wrong solve is kept out by the gate; a second batch
+improves the existing skill in place. See [`src/webwright/skills/README.md`](src/webwright/skills/README.md).
+
+---
+
 ## 🚀 Quick Start
 
 ### Prerequisites

src/webwright/skill_factory/README.md

@@ -0,0 +1,129 @@
+# Web Skill Factory
+
+**Most agent skills are context the model refers to. Ours are programs.**
+
+Each solved task becomes runnable, parameterized code — verify it, run it without the
+model, and import it into the next task instead of re-exploring.
+
+Module: `webwright.skill_factory`, built on Webwright.
+
+## 🎥 Demo
+
+https://github.com/user-attachments/assets/d15b1f83-2c8d-4f2d-bbc5-be365c0bcf4e
+
+## ✨ Why
+
+- 🧩 **Import, don't just refer** — skills are executable code; a learned skill re-runs
+  standalone in ~30 s with **no model in the loop**
+- 🌱 **Self-evolving** — each batch of solves is gated, grouped by template, distilled;
+  wrong answers never enter, working skills never break
+- ✅ **Replay-verified** — a skill must reproduce its own training answers standalone
+  before it may land
+- 💸 **Cheap to adopt** — one tool + one CLI, zero agent-loop changes
+
+## 🗺️ How it works
+
+![data flow & interfaces](../../../assets/skill_factory_pipeline.png)
+
+```
+solve → gate → group by template → distill → replay-verify → library → next solve reuses
+```
+
+At solve time the agent asks the library once and gets `use` / `adapt` / `skip` —
+reuse never blocks solving.
+
+## 🚀 Quick Start
+
+```bash
+cd src/webwright/skill_factory/examples
+./quickstart.sh        # a learned skill drives a live site — no model, no API key needed
+```
+
+<details>
+<summary><b>More modes</b> — reuse with the agent, or rebuild the library yourself</summary>
+
+<br>
+
+```bash
+export OPENAI_API_KEY=...
+./quickstart.sh ask      # ask the library about a task it has never seen
+./quickstart.sh solve    # the agent REUSES the checked-in skill on a new route
+./quickstart.sh full     # the whole loop: 3 solves -> learn -> reuse (~30 min)
+```
+
+On a custom OpenAI-compatible gateway, also `export OPENAI_ENDPOINT=... OPENAI_MODEL=...`
+(for learn / skill_use; the endpoint is the FULL `.../responses` URL, not a base path) and
+`export MODEL_CFG=...` pointing at a copy of `examples/model_gateway.example.yaml` (for the
+agent — it reads a yaml, not these env vars).
+
+</details>
+
+Full tutorial — the loop spelled out, gateway setup, running skills without the agent:
+**[docs/quickstart.md](docs/quickstart.md)**
+
+## 📊 Results
+
+**Setting** — WebArena, 10 retrieve-type task templates across 3 self-hosted sites
+(shopping-admin, gitlab, map). Per template: 3 train solves build the library
+(gold-gated), 2 held-out instances measure reuse; every held-out task is solved both
+WITH the library and from scratch. Same model, same budget, 100 solves total.
+
+|                         | WITH library | from scratch |    Δ    |
+|-------------------------|--------------|--------------|---------|
+| held-out accuracy (20)  | **70%**      | 55%          | **+15 pp** |
+| held-out avg steps      | **14.7**     | 17.1         | −2.4    |
+| train accuracy (30)     | **86.7%**    | 76.7%        | +10 pp  |
+| train avg steps         | **13.7**     | 15.9         | −2.2    |
+
+- **4 held-out tasks rescued** (wrong → correct); net reuse-wins vs regressions **7 : 1**;
+  biggest win **33 → 10 steps**
+- retrieval stayed reliable as the library grew: all 20 held-out solves picked the right
+  skill, including two near-duplicate templates
+- mixed-template batches evolve safely: adds, refines and keeps with zero cross-contamination
+
+## 🧠 Design
+
+**Actions are already code.** Webwright solves by writing code — every solve leaves a
+working script behind, so skills are a byproduct, not an instrumentation layer. It
+complements `crafted_cli`: craft parameterizes one script by *anticipating* what might
+vary; the factory parameterizes across solves from the differences *actually observed*.
+
+**One solve isn't a skill yet.** A single script is correct but narrow. Aggregating
+verified solves of the same template turns observed differences into parameters and
+recurring patterns into primitives — different runs' strategies become fallbacks, and
+what lands is the best algorithm the solves discovered.
+
+**Parameters are evidence, primitives are the product.** Parameters aren't guessed up
+front — they are the differences actually observed between solves. The expensive core is
+factored into named primitives (`login()`, `open_grid()`, `extract_rows()`) with a thin
+task layer on top, so future tasks can reuse and compose the pieces even when the final
+step differs.
+
+**Nothing lands unproven.** The admission gate checks the *inputs*, but distillation
+itself can introduce bugs — and an agent reading a broken skill as source will quietly
+work around them, so agent-in-loop numbers alone can't certify a library. The proof is
+model-free execution: a skill must replay its own training taskspecs standalone and
+reproduce the answers before it lands, and every skill carries its grade —
+`executable` (proved) or `reference` (a prior the agent may still read).
+
+**Growth never breaks what works.** New templates add skills, new solves refine existing
+ones in place (regression-replayed against their stored training examples), untouched
+skills stay byte-identical.
+
+## 🔌 Plugs into Webwright
+
+Two touch points, no agent-loop changes:
+
+```bash
+python -m webwright.tools.skill_use --task "<task>" --library ./library   # reuse at solve time
+python -m webwright.skill_factory learn outputs/ --library ./library     # grow it afterwards
+```
+
+## 📚 Documentation
+
+| doc | what's in it |
+|---|---|
+| [docs/quickstart.md](docs/quickstart.md) | the complete tutorial: the flights loop, gateway knobs, standalone usage, measured costs |
+| [docs/manual.md](docs/manual.md) | manual mode: manifests field by field, gold gates, the batch pipeline |
+| [docs/reference.md](docs/reference.md) | verification & grades, every flag and env var, component map, backend |
+| [examples/README.md](examples/README.md) | the checked-in skill and the example inputs |