Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54
Open
DEM1TASSE wants to merge 51 commits into
Open
Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54DEM1TASSE wants to merge 51 commits into
DEM1TASSE wants to merge 51 commits into
Conversation
…tool
A built-in submodule turning solved tasks into reusable, executable code skills:
- skills/{library,retrieve,decide,gate,update,llm}: store / retrieve (relevance) /
decide (use·adapt·skip utility) / admission gate (gold|self_verify|none) /
evolve (incremental growth on existing library) — backend-agnostic via configure_llm
over webwright's own Model abstraction (no hardcoded gateway/key/path)
- tools/skill_use.py: solve-time tool (agent invokes like self_reflection/image_qa) ->
retrieve+decide -> JSON recommendation (use/adapt/skip + source path)
- python -m webwright.skills.update --manifest batch.json --library ./lib : batch growth
- tests/skills: 5 unit tests pass against the migrated module (logic == original)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…skill_use CLI - skills/prompt.with_skill_hint: prepend skill-library usage hint to task prompt (non-invasive; webwright merges system_template by replacement, so prompt-level is the clean way) - config/skill_mode.yaml: optional overlay doc + step budget for skill-reuse runs - llm._model(): bare CLI (python -m webwright.tools.skill_use) builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env -> same backend as agent, no hardcoded gateway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: what the module is, the two plug points (skill_use tool + update CLI), components table, gate semantics, backend config, results summary - llm._model(): bare CLI builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: Skill Library section (what it is, reuse via skill_use tool, grow via update CLI, end-to-end validation summary) - tests/skills: 5 unit tests for library/gate/update/evolve/retrieve+decide Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Remove _grow / update() / _UPDATERS dispatch — evolve() is the single entry now; drop the test_update test that exercised the removed grow path. Keep retrieve/llm fallbacks (useful). Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…val) Three bugs hit when update.refine emits a large skill on a slow gateway: - llm() ignored max_tokens -> model default ~4000 truncated the refined skill mid-code - llm() had no timeout override -> model default 120s ReadTimeout'd on the ~16k-token refine (now request_timeout_seconds defaults 600, env SKILL_MODEL_TIMEOUT) - _extract_code returned raw text (with ```python fence) when the closing fence was missing (truncated) -> skill failed to compile; now strips the opening fence anyway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…lve-time reuse, direct skill run) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
@microsoft-github-policy-service agree company="Microsoft" |
…te+manifest -> update -> reuse); fix output_schema examples to gate's {type} form
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bArena numbers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- traces_from_manifest: 'admit' is now REQUIRED per run — a missing gate verdict raises instead of silently defaulting to admitted (was the main pollution risk) - _slug: templates longer than 48 chars get a short content-hash suffix so two templates sharing a long prefix can no longer overwrite each other's skill - skill_use.recommend: the decision's skill_id must be one of the RETRIEVED candidates; anything else (LLM hallucination, even an existing library id) downgrades to skip - tests for all three Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… is truthy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tive paths, missing answer file) Independent cleanroom reproduction (fresh clone + venv, README-only, public GitHub tasks) surfaced usability failures; mechanism itself reproduced end-to-end in 25 min. - with_skill_hint resolves the library path to ABSOLUTE (F2): the hint's command runs in the agent's workspace, where a relative ./library silently resolved to a nonexistent dir -> empty library -> every lookup skipped, no error, answer still right - skill_use.recommend: a missing/empty library now answers skip with an explicit 'warning: library empty at <abspath>' BEFORE Library() can mkdir the bogus path (F3) - README: step 1 now tells the agent to write agent_response.json (stock webwright does not produce it; the gate/manifest flow assumed it) with a copyable ANSWER_SPEC (F1); absolute-path + --library-beats-env notes (F2/F4); custom endpoint tip (F5) - skills/__init__ no longer eagerly imports update -> no more runpy RuntimeWarning on 'python -m webwright.skills.update' (F6); import evolve/Trace from the submodule - tests: hint abspath, empty/missing-library warning (incl. no-mkdir side effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, filled-in inputs - example_library/: the commit-counting skill verbatim as evolve wrote it (runnable standalone via taskspec, no LLM in the loop; functionally verified against a local repo) - README: what a skill looks like (catalog card + the distilled git-log algorithm), measured held-out numbers (33->10 steps; wrong->correct rescues; honest note that reuse costs more than it saves on cheap tasks), three try-it paths - honest coverage-boundary demo: an unseen period shape raises cleanly; on the real held-out run the agent read the source and adapted around it - tasks/batch/taskspec example JSONs matching the how-to-use steps - links from the module README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, safe growth, measured results) + data-flow/interfaces diagram
…no manifest) python -m webwright.skills learn <runs_dir> --library ./library - auto: reads task.json/agent_response.json per run, gates (gold via --golds, else self_verify), groups tasks into templates + extracts params with one LLM call per chunk (default 25; existing templates passed in so chunks refine instead of duplicating), infers output_schema from the answer shape, site from start_url - idempotent: processed runs remembered in library/.learned.json; --dry-run plan mode - README: Quickstart (two commands) + use cases up top; old walkthrough demoted to 'Manual mode'; examples/solve_with_library.sh wrapper (hint + answer instruction) - unit tests for the LLM-free plumbing
…ks, leaks) External-user test of the friendly path surfaced that a trivially-easy config mistake (gateway key + unset OPENAI_ENDPOINT) silently disabled ALL reuse. Fixes: - skill_use: a hard error still degrades to skip (never block solving) but now says LOUDLY it is a LOOKUP FAILURE, not a no-match — error field in the JSON, hint about OPENAI_ENDPOINT/SKILL_MODEL_ENDPOINT, and a stderr line (F1+F2) - README Quickstart: gateway users must export OPENAI_ENDPOINT/OPENAI_MODEL for BOTH steps, stated where step-1 users actually look (F2) - learn: grouping-LLM failure now exits with a one-line actionable message instead of a 40-line traceback (F3); skipped-for-missing-answer runs get a visible summary with the correct pointer (the old message named a command that does not exist) (F4) - learn: strips the answer-output instruction from task text so it cannot leak into templates/skill_ids (F7) - solve_with_library.sh: usage check instead of passing empty args into the CLI (F6)
…ers get them too)
…ssion tests - README: "Only verified solves get in" -> "Validation-gated, exactly as strong as the gate you give it" — states plainly that the default self_verify checks shape only and that the WebArena numbers used the gold gate; learn prints the same warning at run time when no --golds is given - examples/learned_library/: a skill produced by "skills learn" from 3 real GitHub solves — n_solves=3, owner/repo lifted to parameters, two extraction strategies as fallbacks; verified standalone on an unseen repo (numpy/numpy -> v2.5.1, no model); test_learned_example.py locks n_solves>=3 + lifted params + no leak - regression tests for the interface-test findings: F1 (skill_use surfaces hard errors as ERROR, not quiet skip), F3 (learn exits with an actionable message)
There was a problem hiding this comment.
Pull request overview
This PR introduces a new webwright.skills subsystem that turns previously solved tasks into reusable, executable “skills”, enabling solve-time reuse (via a CLI tool) and offline library growth (via learn/update pipelines) while keeping the main agent loop unchanged.
Changes:
- Adds a disk-backed skill library (
Skill/Library) plus retrieve/decide/gate/evolve/learn modules to store, select, admit, and incrementally refine skills. - Adds
webwright.tools.skill_useas a solve-time CLI that recommendsuse|adapt|skipand provides the source path for reuse. - Adds docs/config/examples and new tests to validate deterministic plumbing and example artifacts.
Reviewed changes
Copilot reviewed 29 out of 31 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/skills/test_retrieve_decide.py | Adds deterministic tests for retrieve/decide + skill_use/prompt behavior (currently not pytest-discoverable). |
| tests/skills/test_library.py | Adds tests for disk persistence of Library (currently not pytest-discoverable). |
| tests/skills/test_learned_example.py | Adds a check that the checked-in learned example is aggregated/parameterized (currently not pytest-discoverable). |
| tests/skills/test_learn.py | Adds tests for learn plumbing + regression handling (currently not pytest-discoverable). |
| tests/skills/test_gate.py | Adds tests for gate admission logic (currently not pytest-discoverable). |
| tests/skills/test_evolve.py | Adds tests for evolve behavior and slug collision avoidance (currently not pytest-discoverable). |
| src/webwright/tools/skill_use.py | Introduces solve-time library recommendation tool with guardrails against missing/empty libraries and hallucinated skill IDs. |
| src/webwright/skills/update.py | Implements incremental library evolution and refinement prompt construction + manifest ingestion. |
| src/webwright/skills/retrieve.py | Implements LLM-based retrieval plus a simple deterministic keyword-overlap fallback. |
| src/webwright/skills/decide.py | Implements LLM-based use/adapt/skip decision over retrieved candidates. |
| src/webwright/skills/gate.py | Implements admission gate (gold/self_verify/none/auto) to prevent wrong solves from entering the library. |
| src/webwright/skills/learn.py | Adds “friendly” pipeline to learn skills from run folders with gating, chunked grouping, and an idempotent ledger. |
| src/webwright/skills/library.py | Adds on-disk skill storage (<id>/skill.py + meta.json) and simple list/get/add APIs. |
| src/webwright/skills/llm.py | Adds backend-agnostic LLM helper using Webwright’s Model abstraction. |
| src/webwright/skills/prompt.py | Adds with_skill_hint() helper that prepends a bash command hint to consult the skill library. |
| src/webwright/skills/init.py | Exposes the public webwright.skills API surface for consumers. |
| src/webwright/skills/main.py | Adds `python -m webwright.skills <learn |
| src/webwright/skills/README.md | Adds comprehensive module documentation, usage patterns, and rationale. |
| src/webwright/skills/pipeline_diagram.svg | Adds diagram documenting data flow and interfaces for the skills pipeline. |
| src/webwright/config/skill_mode.yaml | Adds optional config overlay to increase step budget for skill reuse runs. |
| src/webwright/skills/examples/README.md | Adds examples overview and how-to for running skills/tools and batch pipeline. |
| src/webwright/skills/examples/solve_with_library.sh | Adds helper script to prepend hint + answer spec and run Webwright with a library. |
| src/webwright/skills/examples/taskspec.example.json | Adds example taskspec input for running a skill standalone. |
| src/webwright/skills/examples/tasks.example.json | Adds example batch task list input with params/golds. |
| src/webwright/skills/examples/batch.example.json | Adds example manifest for update (admit/params/schema/etc). |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/skill.py | Adds a runnable example skill produced by the pipeline. |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/meta.json | Adds metadata for the example skill. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/skill.py | Adds a checked-in “learned” skill example aggregated from multiple solves. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/meta.json | Adds metadata for the learned skill example. |
| README.md | Adds top-level README section linking to the new skill-library capability and docs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| f"answer={json.dumps(tr.answer, ensure_ascii=False)[:120]})\n```python\n{tr.code}\n```" | ||
| ) | ||
| sys_prompt = _REFINE_SYS + (_REFINE_INCREMENTAL if existing else "") | ||
| code = _extract_code(llm(sys_prompt, "\n\n".join(blocks), max_tokens=16000, timeout=400)) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — removed the kwarg (it was indeed swallowed by llm()'s **_). Timeout is governed by the model's request_timeout_seconds (600s default in _model()).
Comment on lines
+175
to
+179
| if answer is None and (d / "agent_response.json").exists(): | ||
| try: | ||
| answer = json.load(open(d / "agent_response.json")).get("retrieved_data") | ||
| except Exception: | ||
| pass |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+198
to
+199
| manifest = json.load(open(a.manifest, encoding="utf-8")) | ||
| traces = traces_from_manifest(manifest) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+59
to
+64
| def llm_json(system: str, user: str, **kw: Any) -> dict: | ||
| """Call + parse the first {...} JSON object out of the reply.""" | ||
| txt = llm(system, user, **kw) | ||
| match = re.search(r"\{.*\}", txt, re.S) | ||
| if not match: | ||
| return {} |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — llm_json now walks the reply with json.JSONDecoder().raw_decode and returns the first valid JSON object, skipping non-JSON brace snippets; matches the docstring.
Comment on lines
+48
to
+56
| meta = json.loads((d / "meta.json").read_text()) | ||
| code = (d / "skill.py").read_text() if (d / "skill.py").exists() else "" | ||
| return Skill(skill_id=skill_id, code=code, meta=meta) | ||
|
|
||
| def add(self, skill: Skill) -> None: | ||
| d = self._dir(skill.skill_id) | ||
| d.mkdir(parents=True, exist_ok=True) | ||
| (d / "skill.py").write_text(skill.code) | ||
| (d / "meta.json").write_text(json.dumps(skill.meta, ensure_ascii=False, indent=2)) |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — explicit UTF-8 on all skill/meta reads and writes in library.py (and the same for learn.py's ledger/run files).
Comment on lines
+34
to
+35
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+86
to
+87
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+26
to
+27
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+69
to
+70
| if __name__ == "__main__": | ||
| run() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+53
to
+55
| if __name__ == "__main__": | ||
| run() | ||
| run_regressions() |
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all) that runs both run() and run_regressions().
The README table (held-out 70% vs 55%, 14.7 vs 17.1 steps; train 26/30 vs
23/30) previously existed only as prose. Now:
- evals/webarena/results/: sanitized per-task records of the exact run behind
the table — task id, answer, gold score, steps, skill verdict; one command
("reproduce.py table --results results") re-derives the table, no setup
- evals/webarena/reproduce.py: self-contained driver that re-runs the whole
experiment (train -> gold-gated update -> held-out with/base -> table)
against your own WebArena deployment via microsoft/webarena-verified;
resumable, parallelizable per template
- run_all.sh + model.eval.yaml (agent-model overrides; gateway pointer)
- tests/skills/test_eval_snapshot.py: CI-locks the records to the published
numbers and enforces snapshot sanitization (no local paths/hosts/keys)
- skills README + examples README now link the records instead of asking for
trust; CI also triggers on evals/webarena/**
- prompt.py: shell-quote task and library in the skill_use hint (shlex.quote) —
$VAR / $(...) / backticks expanded in bash even inside the old double quotes;
regression test added
- llm.py: llm_json now scans for the FIRST valid JSON object (raw_decode) as
documented, instead of a greedy first-{ to last-} regex that could span
unrelated braces
- update.py: drop the misleading llm(..., timeout=400) kwarg (silently
swallowed; the model's request_timeout_seconds already governs); read JSON
files via read_text instead of unclosed open()
- library.py / learn.py: explicit UTF-8 on every skill/meta/ledger read+write
(locale-independent on Windows)
- tests: pytest-discoverable test_all() entry points in all 7 files (CI keeps
running them as scripts too)
adamlu123
reviewed
Jul 9, 2026
Author
There was a problem hiding this comment.
It's the guided tour of the examples/ directory (the module README links here for every "see examples"): two real, checked-in skill libraries — learned_library/ is the Quickstart loop's actual output (3 GitHub solves -> learn -> owner/repo lifted to parameters, runs standalone on unseen repos with no model), example_library/ is verbatim update.evolve output from the WebArena eval — plus the solve wrapper and filled-in copies of every input file the manual pipeline asks you to write. 8ea59be makes this explicit in the file's opening paragraph and adds the learned_library provenance section.
Contributor
There was a problem hiding this comment.
this file looks redundant, can we remove?
Author
There was a problem hiding this comment.
Agreed — removed in 8ea59be. Nothing referenced it: the skill hint is prompt-level (with_skill_hint), and no documented path needed the step_limit bump.
| # 2. turn everything you've solved into skills — no manifest, no fields to learn | ||
| python -m webwright.skills learn outputs/ --library ./library | ||
| ``` | ||
|
|
Contributor
There was a problem hiding this comment.
@DEM1TASSE It is better to add the complete example in the quick start session.
It needs to additionally include how to use the skill library.
Author
There was a problem hiding this comment.
Done in 8ea59be — the Quickstart is now the complete loop on a copy-pasteable example (public GitHub): solve 3 instances -> learn -> an unseen instance reuses the skill (with the expected skill_decision.json shown), plus how to use the library without the agent (querying skill_use directly, and running the learned skill standalone with no model — verified pandas-dev/pandas -> v3.0.4). The same loop's output is checked in at examples/learned_library/.
…larify examples/ - README Quickstart is now the complete loop on a runnable example (public GitHub): solve 3 instances -> learn -> an UNSEEN instance reuses the skill, plus using the library without the agent (skill_use query, running the learned skill standalone, verified: pandas-dev/pandas -> v3.0.4 with a params-only taskspec) - remove config/skill_mode.yaml: nothing referenced it (the skill hint is prompt-level via with_skill_hint; the step_limit bump was never needed by any documented path) - examples/README: state the directory's purpose up front, and document learned_library's provenance (the Quickstart loop's checked-in result) with the unseen-repo runs and the CI test that locks it
The GitHub release example couldn't show what a skill carries: a frontier model already knows GitHub cold, and the answer is one search away. Google Flights is the task family Webwright's main README uses, and it has the two properties the Quickstart needs: the answer is live (unrecallable), and the UI is fiddly enough that from-scratch solves cost 13-26 steps. - Quickstart rewritten around the flights loop: 3 routes solved from scratch -> learn lifts FIVE parameters (origin/destination city+code, date) -> an unseen route reuses the skill (verdict=use) -> the same skill runs standalone in ~30 s with no model (the cron fare-watcher payoff) - honest verification note for live data: no gold exists, gate=self_verify (the run-time warning says so); what is verified is consistency — the standalone skill and a fresh agent solve of an unseen route returned the same answer minutes apart - examples/learned_library gains the flights skill next to the GitHub release skill (two task families, two sites, one library); provenance documented in examples/README; test_learned_example locks every skill in the directory
Same unseen route (SEA->DEN), minutes apart: from scratch 17 steps, with the library 15 steps (verdict=use), standalone skill ~30 s with no model — all three returned the identical answer. Live-data verification stated as what it is: consistency across independent paths, not a gold gate.
- Quickstart now shows the full ledger for the same unseen route: from scratch 17 steps / ~378k tokens / 8.9 min, with the library 15 steps / ~407k tokens / 5.2 min, standalone 0 tokens / ~30 s — same answer all three ways. Read honestly: single-reuse saves wall time (not steps or tokens) on a site the model knows; the economics are in repeats running with no model at all - examples/learned_library keeps only the flights skill (the Quickstart's own artifact); the GitHub release skill and its doc sections are removed
…gnal) A run's agent_response.json already says whether the agent itself believed the answer (SUCCESS vs NOT_FOUND_ERROR); learn was throwing that signal away. The self_verify gate now rejects runs whose own agent reported failure — no extra model call. Still self-grading: a wrong answer the agent BELIEVED passes; the webwright self_reflection label stays unused as a gate because completed runs always carry a passing label (it is a solve-completion condition). Docs and run-time warning updated to the new honest wording; gold path unchanged.
Synthetic run dir with a well-formed answer but status=NOT_FOUND_ERROR: learn must reject it at the gate (0/1 admitted) and never reach the LLM — offline.
examples/quickstart.sh bakes in all routes/dates/taskspecs so nothing needs writing: default mode runs the checked-in flights skill standalone (no model, no API key; date auto-set to +30 days), 'ask' queries the library, 'solve' has the agent reuse the checked-in skill on a new route, 'full' rebuilds the library end to end. Also: the checked-in skill dir is stripped to skill.py + meta.json (stray run artifacts removed), the demo runs in a temp workdir via WORKSPACE_DIR so it can never pollute the library, README leads with the one-command path, CI checks the script's syntax and usage exit.
Same records, one file: reproduce.py table accepts a directory or the merged json, the CI snapshot lock reads the merged file, README commands updated. Auditability unchanged — every per-task record is still there.
results.json + reproduce driver + snapshot-lock test removed to keep the PR lean; README references now say the records live in the companion research repo and can be shipped on request.
…step docs Two findings from the final external regression on c8b27c5: - the checked-in flights skill wrote screenshots/log to Path(__file__).parent, so every quickstart demo dirtied the shared library. Artifacts (answer, log, screenshots) now all go under WORKSPACE_DIR (default: cwd); the refine prompt bakes the same rule into every future generated skill, and test_learned_example rejects any artifact path anchored to __file__ - "gateway users: just export env vars" was wrong for the solve steps: the agent's model reads its -c yaml, not OPENAI_ENDPOINT. README and quickstart.sh now say both knobs plainly; quickstart.sh takes MODEL_CFG to point solve/full at a gateway model config
skills write artifacts to their cwd (by design, WORKSPACE_DIR default) — so every documented standalone invocation now cds into mktemp -d first instead of running inside examples/ or the module dir. quickstart.sh already did this; the manual Try-it flows now match.
…ting Two gaps a reader caught: the manifest box never said the solve's CODE (dir/final_script.py) is what becomes Trace.code — evolve's most important input was invisible; and the exact-template bucketing (one bucket, one skill) was a side note. Both are now explicit lines in the UPDATE lane.
Closes the output-side gap: the gate checked evolve's INPUT (each solve's answer) but the synthesized skill was never executed before landing — a bug introduced during distillation reached the library unchecked (the exact class the werkzeug [] failure demonstrated). _refine now replays the candidate on each source trace's own taskspec (no model, WORKSPACE_DIR-isolated, 240s cap). Exact answer match passes; in the default "shape" mode a non-empty, schema-shaped answer also passes (live sites drift between solve and replay — prices, listings) while crashes, timeouts and empty/misshapen output are caught. On failure the LLM gets one repair attempt with the concrete replay failures; if that also fails the skill is NOT written (changelog gains "rejected"). learn defaults to shape; the update CLI defaults to off (benchmark sites may need credentials) with --verify to opt in. Also fixes a latent NameError: update.main() used Path without a module-level import since the context-manager cleanup; CLI smoke test added. test_evolve now exercises the real repair loop with only the LLM faked: wrong->repaired lands, wrong-twice rejected, drift tolerated in shape mode, crash caught.
…tials Real-data efficacy run of the new verify gate (7 templates whose old skills failed the standalone audit, re-evolved with --verify strict against live sites): 0 bad skills landed (old evolve: 7/7 landed broken), the repair round saved 2 (a relative-URL crash and an empty-extraction bug), and one rejection was a comparator false negative — the skill returned [5] where the solve had recorded ["5"]. Scalars are now compared type-normalized (5 == "5"), the same normalization WebArena's own evaluator applies; that template now lands first-pass. traces_from_manifest passes credentials through so replay can log in on auth'd sites.
learn was ledger-marking a template's runs even when evolve rejected the distilled skill, permanently burning those solves. Rejection now skips the ledger write and says so; regression locks it.
learn's replay gate defaults to strict: a distilled skill must run standalone on its own training taskspecs and reproduce the recorded answers, or it does not enter the library (one repair attempt, then rejected with the runs kept retryable). shape stays as the documented escape hatch for task families whose answers are live data — the quickstart's flights flow passes --verify shape explicitly with the reason inline, and strict rejections print a hint pointing at it. README states the bar as the bar.
- verification now yields a GRADE: executable (replayed its training taskspecs standalone, answers reproduced) vs reference (failed the bar; lands only via --on-fail reference as a labeled prior for the agent, never overwriting an existing skill). --verify-rounds N caps build attempts (default 2) - incremental refine is regression-checked: admitted solves' taskspecs+answers persist in the skill's replays.json (credentials never stored; borrowed from the incoming batch at replay time); a refine must pass old + new instances, and a VERIFIED skill with no stored examples refuses refinement outright — its old coverage could not be proven intact - README documents the grade trade-offs (executable: pricier to build, runs with plain playwright and no model; reference: readable prior) and the honest footnote that the WebArena numbers came from an effectively all-reference library (3/10 replayed clean) — evidence for the reference grade's value, with the flights quickstart evidencing the executable grade
Reviewer feedback: 'X.skills' reads as a pack of ready-made skills; this is a pipeline that BUILDS them. New name: Webwright Skill Lab — a self-evolving factory for reusable, verified, code-native web-agent skills. Module dir, tests dir, every import/path/命令 in docs, CI globs, the wrapper and quickstart, and the pipeline diagram title all move; webwright.tools.skill_use keeps its name (it is the consume-side tool).
DEM1TASSE
changed the title
Subtitle wording settles as 'evolving reusable, verified, code-native skills for web agents'. Directory, tests dir, imports, docs, CI globs and the pipeline-diagram title all follow; webwright.tools.skill_use unchanged.
DEM1TASSE
changed the title
Layout feedback: the examples dir had two libraries, stray run artifacts (screenshots + log had crept back into the flights skill dir), and a commit-count taskspec sample — hard to see what's what. Now: one library (the Quickstart's flights skill, skill.py + meta.json only, locked by a test that rejects any extra file), the two manual-mode input samples, the two scripts, and a short README. The pipeline diagram moves to assets/ (skill_factory_pipeline.png) matching how webwright keeps its media; the module stays a webwright subpackage — that is the repo's convention for every capability, and webwright.tools.skill_use imports it.
Overview is now pitch + 2-min demo + diagram + five highlight bullets + a one-command entry point. The moved-down material keeps its content: the 'why webwright / one solve isn't a skill / safe growth' prose becomes a 'How it works' section after the Quickstart, and the gate-honesty text, grade table and the WebArena-was-reference footnote merge into a single 'Verification and grades' section near the end. Demo video ships as assets/skill_factory_demo.mp4.
Updated README.md to improve clarity and formatting.
The README was five documents welded together and taught the same
solve->learn->reuse loop twice. Now it matches Webwright's own README
style — emoji sections, short bullets, whitespace, one quickstart command
(extra modes behind a <details> fold) — and stays at the landing layer:
pitch, demo, why, how-it-works, cost table, results, docs links. The
manual, verification/grades discussion, design rationale, evaluation
detail and all the honest footnotes move verbatim into module-local
docs/{quickstart,manual,verification,architecture,evaluation}.md.
Per review: the complete WebArena table + eval setting and the design rationale belong on the landing page; docs/ consolidates to quickstart, manual, and a single reference (verification & grades, every flag and env var, component map, backend). architecture.md and evaluation.md are gone.
…play-verify The measured-cost table is one example, not a landing claim — it lives in docs/quickstart.md (already there). Design grows two beats pulled from the project's discussions: parameters-are-evidence / primitives-are-the-product, and nothing-lands-unproven (why input gating isn't enough, why the proof is model-free replay, and what the executable/reference grades mean). The pipeline diagram's UPDATE lane now shows the replay-verify stage between evolve and the library write-back, including the regression replay on incremental refines.
Repro notes found three fixable traps. (1) OPENAI_ENDPOINT must be the FULL request URL — every error hint and doc now says so with an example, because a bare '.../api' base fails against the model class. (2) The agent in solve/full reads a yaml, not env vars: quickstart.sh now prints a loud warning when OPENAI_ENDPOINT is set but MODEL_CFG is not, and examples/model_gateway.example.yaml ships as a fill-in template (placeholders only). (3) docs note that solves are 5-30 min tasks — run them in the background under tooling that enforces command timeouts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Web Skill Factory (
webwright.skill_factory) — a self-evolving skill factory (MVP): turn solved tasks into reusable,executable code skills, retrieve + judge them at solve time, gate what enters the library, and grow
the library incrementally. A self-evolving loop on top of Webwright's code-as-action solves:
This is the reuse + accumulation layer on top of Webwright's code-as-action solves: it consumes
the
final_script.pyevery solve already produces (plain or crafted mode — both work), accumulatesskills across tasks, judges when a prior skill applies, and improves skills as more solves arrive —
with a gate so wrong solves don't pollute the library. It complements
crafted_cli: wherecrafted_cliparameterizes a single task's script by anticipating what might vary,update.refineparameterizes across multiple verified solves — the differences actually observed between
instances become the parameters.
Modular composition (~810 lines of core code)
Ten small, single-responsibility modules — each with a stable interface and a swappable
implementation:
skill_factory/library.pySkill+Library, skills on disk (skill.py+meta.json)skill_factory/retrieve.pyskill_factory/decide.pyskill_factory/gate.pyskill_factory/update.pyrefineparameterizes + decomposes into primitivesskill_factory/llm.pyModel(no endpoint/key hardcoded)skill_factory/prompt.pywith_skill_hint)skill_factory/learn.pylearn <runs_dir>: auto-group runs into templates, gate, evolve; no manifest to writeskill_factory/__main__.pypython -m webwright.skill_factory <learn|update>dispatchertools/skill_use.pyHow it plugs in (no change to the agent loop or default config)
skill_usetool, invoked from bash likeself_reflection/image_qa:{verdict: use|adapt|skip, skill_id, source_path, how_to_reuse}.updateCLI distills a batch of gate-passed solves into aparameterized, primitive-decomposed skill:
learngroups a folder of finished runs intotemplates (one LLM call per chunk), gates them, and evolves the library — idempotent,
--dry-run:examples/learned_library/checks in the skill this produced from 3 real Google Flightssolves — five parameters lifted (origin/destination city+code, date), verified on an unseen
route three independent ways (from scratch / reuse / standalone, same answer) — with a CI
test locking it.
Validation
WebArena: 10 templates × 3 domains — reuse lifts accuracy +15pp and saves steps on held-out tasks
10 retrieve-type task templates across shopping_admin / gitlab / map. Per template: 3 train
tasks build the library (solved from scratch; only gold-verified solves are admitted), 2 held-out
tasks (unseen instances of the template — different parameter values) measure reuse. Every task is
solved both WITH the library and from scratch (BASE) — 100 solves total.
Per-task records and a reproduction driver are kept in the companion research repo and can be
shipped here on request.
Highlights:
library; net reuse-wins 7 vs 1 regression across the 20 held-out tasks.
33 steps (scratch) to 10 (reuse); a map routing task from 29 to 16.
here. (The gate is exactly as strong as its verifier — the default
self_verifyis a shapecheck only; see the README's "validation-gated" section.)
update.refinelifts per-instance differences into parametersand bakes the aggregation logic (top-n ranking, commit counting, route-time extraction) into
primitives, so unseen instances of the template solve by a direct
useof the skill.skill from the shared library (grown to 10 skills over the run), including telling apart two
near-duplicate gitlab commit-counting skills (by-date vs by-period).
evolvebatches produce 4 independent skills — new templates get added, existing skills arerefined in place (working functions kept), skills with no new traces stay byte-identical, and
zero cross-contamination between skills; held-out reuse against the mixed-built library matches
the per-template-built one.
Real website (public GitHub, read-only): the full loop end-to-end
Solve two repos from scratch ->
updatebuilds a parameterized skill -> a held-out repo is solved byreusing it (the agent calls
skill_use, verdictuse, answer correct). Reuse pays off most onmulti-step tasks where saved exploration outweighs the lookup overhead (see the WebArena numbers);
on short single-page lookups it is roughly break-even.
7 unit-test files under
tests/skill_factory/(library / gate / evolve / retrieve+decide / learn /learned-example lock / eval-snapshot lock) run in CI on every push touching the module
(
.github/workflows/skills-tests.yml).Status: a deliberately simplistic MVP
Most steps are a single LLM call (retrieve = one catalog prompt, decide = one prompt, refine =
one batched prompt) — chosen for clarity, not yet for scale/accuracy. The point is the modular
shape: each stage has a stable interface, so swapping in something stronger (embedding retrieval,
a learned ranker, WebJudge / cross-source consistency for the real-website gate) is a localized
change that does not touch the others or the agent loop.
Scope
Purely additive (zero deletions), confined to
src/webwright/skill_factory/(module + examples,including a checked-in learned skill),
src/webwright/tools/skill_use.py,tests/skill_factory/and one CI workflow. The actual implementation is ~670 lines of logic(non-blank, non-comment, across the skills module + the
skill_usetool); the rest is tests,examples, eval records, and docs. No edits to the agent loop, models, or existing configs.
Module README:
src/webwright/skill_factory/README.md.