diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json new file mode 100644 index 00000000..46a9b5f0 --- /dev/null +++ b/.claude-plugin/marketplace.json @@ -0,0 +1,15 @@ +{ + "$schema": "https://anthropic.com/claude-code/marketplace.schema.json", + "name": "diffusers-workflow", + "description": "Skills for driving a diffusers-workflow server from Claude Code: one composition skill per model family.", + "owner": { + "name": "Don Kackman" + }, + "plugins": [ + { + "name": "dw", + "source": "./plugins/dw", + "description": "Compose MiniMax H3 video, MiniMax Music 3 and LTX-2.5 workflows over a dw MCP server: which template fits which shape, the hard rules, cost, and how to judge the output. Prompt format comes from the vendors' own guides." + } + ] +} diff --git a/CLAUDE.md b/CLAUDE.md index e98cf2d2..78af1f0a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,6 +45,16 @@ See docs/SERVER.md, `dw/server/CLAUDE.md` and `ui/CLAUDE.md`. The stdio MCP server lives in `dw_mcp/` — see `dw_mcp/CLAUDE.md` and docs/MCP.md. +### Claude Code plugin + +`.claude-plugin/marketplace.json` publishes the `dw` plugin in `plugins/dw/`: one +composition skill per model family (`minimax-h3`, `minimax-music3`, `ltx-2.5`) that +chooses a template for a request's shape and states the family's hard rules. Model +knowledge lives there and in the catalog, never in engine code; every number a skill +states is pinned to a diffusers symbol by `tests/test_plugin_skills.py`. `plugin.json`'s +version is the engine's, bumped by `scripts/release.sh`. Adding or re-auditing a family +is `.claude/skills/model-family-onboarding/`. + ### REPL Architecture The REPL (`dw/repl.py`) uses a **persistent worker subprocess** (`dw/worker.py`) to keep GPU models cached between runs. Communication is via `multiprocessing.Queue`. Worker management is in `dw/repl_worker.py`, command handlers in `dw/repl_commands.py`. @@ -103,7 +113,7 @@ read an inferred workspace back as one the user named - `get_prompt_dir` yields to its older discovery (`./prompts`, then the walk up from the workflow file) for an inferred workspace but not for an explicit one. `--workflow-dir`, `--output-dir` and `--prompt-dir` each still override one folder. See -docs/WORKSPACES.md, and docs/proposals/workspaces.md for the later stages +docs/WORKSPACES.md, and docs/proposals/server-workspaces.md for the later stages (workflow search path, run directories, `asset:`/`output:` references). ### Type System diff --git a/README.md b/README.md index 25558ff6..26f9fd99 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,15 @@ claude mcp add --transport http dw http://gpu-box:8765/mcp \ --header "Authorization: Bearer $DW_API_TOKEN" ``` +The [dw plugin](plugins/dw/README.md) adds one skill per model family - what +to run for a given shape, the rules that bite, what it costs - and points at +the vendors' own prompt guides rather than restating them: + +``` +/plugin marketplace add dkackman/diffusers-workflow +/plugin install dw@diffusers-workflow +``` + You don't have to compose that command by hand — the server's own **Server** page builds it from the address you pick, alongside the directories it resolved and the workspaces it holds: diff --git a/docs/MCP.md b/docs/MCP.md index 80413c39..01449cad 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -46,6 +46,14 @@ call; that probe is fatal for a remote URL and only a warning for a loopback one (where it usually means `dw.serve` is not up yet). [REMOTE.md](REMOTE.md) covers the remote setup end to end. +Claude Code users can add the composition skills as well: +`/plugin marketplace add dkackman/diffusers-workflow` then +`/plugin install dw@diffusers-workflow`. The plugin ships one skill per model +family (MiniMax H3, MiniMax Music 3, LTX-2.5) that picks a template for a +request's shape and states the family's rules - see +[plugins/dw/README.md](../plugins/dw/README.md). It is optional; every tool +below works without it. + ### Use the absolute path to `dw-mcp` **This is the one setup detail that reliably goes wrong.** If you installed @@ -224,7 +232,7 @@ when no single workflow covers it. | --- | --- | --- | | `get_output_image(name, max_dimension=768)` | `name`, `max_dimension` | Look at a generated image, downscaled to `max_dimension` on its longest side. Returns the image plus a text part reporting `original_size`, `returned_size` and `bytes`, so a downscale is never silent | | `get_output_text(name, max_characters=20000)` | `name`, `max_characters` | Read a text output — a prompt enhancement, or any step whose result is `text/plain` or JSON. Reports the file's real length and whether it was truncated | -| `download_output(name, destination=None, overwrite=False)` | `name`, `destination`, `overwrite` | Save one output file to local disk, of any content type. `destination` may be a full path, a directory, or omitted to save under the output's own name in the current working directory; `~` expands and missing parent directories are created. `overwrite=True` is required to replace a file already at the resolved path. Returns nothing to the conversation but where the file landed — unlike the other media tools, the point is a file on disk, not a payload in context. Writes on the machine running the MCP server - over `dw.serve --mcp` that is the GPU box | +| `download_output(name, destination=None, overwrite=False)` | `name`, `destination`, `overwrite` | Save one output file to local disk, of any content type. `destination` may be a full path, a directory, or omitted to save under the output's own name in the current working directory; `~` expands and missing parent directories are created. `overwrite=True` is required to replace a file already at the resolved path. Returns nothing to the conversation but where the file landed — unlike the other media tools, the point is a file on disk, not a payload in context. Writes on the machine running the MCP server - over `dw.serve --mcp` that is the GPU box. A write that fails there (a path that exists only on the client, for instance) comes back as an error naming the server-side write and the client-side alternatives, not as an anonymous tool failure | | `delete_output(name)` | `name` | Permanently remove one generated file from the output directory | ### Authoring, assets and workspaces diff --git a/docs/QUANTIZATION.md b/docs/QUANTIZATION.md index 00787bb4..0a50812d 100644 --- a/docs/QUANTIZATION.md +++ b/docs/QUANTIZATION.md @@ -192,7 +192,7 @@ through `from_pretrained_arguments`, so quantization is keyed by component name "configuration": { "config_type": "TorchAoConfig" }, "arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" } }, - "text_encoder": { + "language_model": { "configuration": { "config_type": "transformers.TorchAoConfig" }, "arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" } } diff --git a/docs/RECIPES_24GB.md b/docs/RECIPES_24GB.md index 2028a495..cdb4df1f 100644 --- a/docs/RECIPES_24GB.md +++ b/docs/RECIPES_24GB.md @@ -102,6 +102,20 @@ workflows below fit; without it, the load is an OOM kill, not a slowdown. **Examples:** [reference-to-video.json](../workflows/templates/minimax/reference-to-video.json), [chain-matched-to-audio.json](../workflows/templates/minimax/chain-matched-to-audio.json), [image-to-video.json](../workflows/templates/minimax/image-to-video.json), [dialogue-short.json](../workflows/templates/minimax/dialogue-short.json) (five ref2va shots + two Z-Image portraits in ~35 minutes end to end) +## MiniMax-Music3 + +Music3 runs at about 22GiB in bfloat16 under the templates' `components_manager` +auto CPU offload, which keeps only the running component resident; no quantization +is needed on a 24GB card. The language model is the part worth offloading harder: +a leaf-level `group_offload` of `language_model` brings it to about 8GiB (the model +card's low-VRAM recipe). Two things the examples carry: `release_pipeline` on the +music step in any workflow that loads H3 afterwards, since host RAM is the binding +constraint (see Multi-model workflows below), and the run's time follows the length +the model actually sings, not `audio_duration`, which is a ceiling of at most 9000 +frames at 25 frames per second (360 seconds). Output is 44.1 kHz stereo. + +**Examples:** [music.json](../workflows/templates/minimax/music.json), [music-video.json](../workflows/templates/minimax/music-video.json) + ## LTX-2.5 (22B, video + audio) A standard pipeline, but placed per component rather than with a pipeline-level diff --git a/docs/RELEASING.md b/docs/RELEASING.md index d0075cca..71ef4b83 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -8,20 +8,23 @@ scripts/release.sh 0.38.0-alpha.1 "UI front end" # optional tag message ``` The script bumps `pyproject.toml` (the single source of the version — -`dw.__version__` reads it at runtime), commits just that file, pushes +`dw.__version__` reads it at runtime) and sets the same version in +`plugins/dw/.claude-plugin/plugin.json`, so an installed plugin names the +engine it was written against; it commits just those two files, pushes master, tags the bump commit `v0.38.0`, and pushes the tag. It refuses a malformed version, a branch other than master, an existing tag, or a dirty index (unstaged changes elsewhere are fine — the release commit -is path-limited to pyproject.toml). +is path-limited to those two files). By hand, the equivalent is: ```bash # 1. Bump the version in pyproject.toml: # version = "0.38.0" -git commit -m "release 0.38.0" -- pyproject.toml +# 2. Set the same version in plugins/dw/.claude-plugin/plugin.json +git commit -m "release 0.38.0" -- pyproject.toml plugins/dw/.claude-plugin/plugin.json -# 2. Tag the bump commit and push +# 3. Tag the bump commit and push git tag -a v0.38.0 -m "release 0.38.0" git push origin master v0.38.0 ``` diff --git a/docs/TASKS.md b/docs/TASKS.md index 70ccd507..49930833 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -389,7 +389,7 @@ returns frames without it, and this puts it back: | Argument | Required | Description | | -------- | -------- | ----------- | | `video` | Yes | The frames - a frame list, a frame array or tensor, or an audio+video pair whose own soundtrack is replaced | -| `audio` | Yes | The soundtrack - a waveform, or the earlier step whose video carried one, which brings its sample rate along | +| `audio` | Yes | The soundtrack - a waveform, the earlier step whose video carried one, or the path or URL of an audio or video file; the last two bring their sample rate along | | `sample_rate` | No | Sample rate of the waveform. Required unless `audio` carries one; given here it wins | **Example:** [assemble-and-score.json](../workflows/templates/assemble-and-score.json) @@ -419,7 +419,7 @@ a length still passes the whole track along: | Argument | Required | Description | | -------- | -------- | ----------- | -| `audio` | Yes | Path or URL of an audio file, a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | | `start_seconds` / `duration_seconds` | One pair | The slice in seconds; either may be omitted | | `start_frame` / `num_frames` / `fps` | One pair | The slice in video frames; `fps` is required, start and count may be omitted | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | @@ -465,7 +465,7 @@ ending. The curve is the equal-power cosine the seam joins use: | Argument | Required | Description | | -------- | -------- | ----------- | -| `audio` | Yes | Path or URL of an audio file, a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | | `fade_in_ms` | No | Length of the fade in, from the head of the track (default: 0) | | `fade_out_ms` | No | Length of the fade out, to the tail of the track (default: 0) | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | @@ -494,7 +494,7 @@ changes, so the dynamics survive: | Argument | Required | Description | | -------- | -------- | ----------- | -| `audio` | Yes | Path or URL of an audio file, a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | | `peak_dbfs` | No | The level the loudest sample is moved to, in dB below full scale (default: -1.0). 0 is full scale | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | @@ -524,7 +524,7 @@ at every cut: | Argument | Required | Description | | -------- | -------- | ----------- | -| `audios` | Yes | The tracks to layer - waveforms, audio file paths, or videos generated with a soundtrack | +| `audios` | Yes | The tracks to layer - waveforms, audio or video file paths, or videos generated with a soundtrack | | `gains` | No | One plain multiplier per track, in the same order - not decibels. Defaults to unity on every track | | `sample_rate` | With a raw waveform | Sample rate of the waveforms. Required unless every track brings its own; given here it wins | @@ -556,7 +556,7 @@ supplied recording once, up front, feeds it what it already wants: | Argument | Required | Description | | -------- | -------- | ----------- | -| `audio` | Yes | Path or URL of an audio file, a video generated with a soundtrack (which brings its sample rate along), or a waveform | +| `audio` | Yes | Path or URL of an audio or video file, a video generated with a soundtrack (which brings its sample rate along), or a waveform | | `target_sample_rate` | Yes | The rate to convert to | | `sample_rate` | With a waveform | Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry | diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md index 245ae3d4..43aa7176 100644 --- a/docs/WORKFLOW_GUIDE.md +++ b/docs/WORKFLOW_GUIDE.md @@ -557,7 +557,7 @@ once, not twice. - **Ignored when the component's device is the CPU**, where there is nothing to move it off of. -On a 24GB card, `MiniMaxH3Ref2VA.json` peaks at 18.9GiB of reserved VRAM with on-demand +On a 24GB card, `templates/minimax/reference-to-video.json` peaks at 18.9GiB of reserved VRAM with on-demand VAEs against 23.2GiB resident, and the tighter resident fit costs 40 allocator retries - cache flushes forced by a failed allocation - where the on-demand run has none. The headroom is also what lets the chained variant run: its later segments carry an extra @@ -815,7 +815,7 @@ and `load_components` pulls the weights: "modules_to_not_convert": ["proj_in", "proj_out"] } }, - "text_encoder": { + "language_model": { "configuration": { "config_type": "transformers.TorchAoConfig" }, "arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" } } @@ -1172,7 +1172,9 @@ directory being copied. `latest` only selects a run where run directories are; a workflow or file that happens to be called `latest` is still named as itself. Like `asset:`, a reference resolves to a path and then whatever loads paths loads it, so -it works under `image`, `video`, a `from_file`, or a list of them. It resolves against +it works under `image`, `video`, a `from_file`, or a list of them. The audio tasks take +a video file's path too and use the soundtrack muxed into it, which is how a finished +cut is scored in a later run without re-cutting it. It resolves against the output directory the run was told to write to, and cannot leave it: `..`, an absolute path, and a symlink pointing out are all refused. diff --git a/docs/WORKSPACES.md b/docs/WORKSPACES.md index 5eaa8bc3..079092a9 100644 --- a/docs/WORKSPACES.md +++ b/docs/WORKSPACES.md @@ -263,7 +263,7 @@ it. Use them to keep work apart, not to keep it private. ## Where this is going Workspaces were the first stage of the design in -[proposals/workspaces.md](proposals/workspaces.md). The resolver, the workflow +[proposals/server-workspaces.md](proposals/server-workspaces.md). The resolver, the workflow search path with writes confined to the writable root, run directories with an on-disk manifest, `asset:` and `output:` references, and server-side named workspaces are all implemented. What remains from the proposal is an MCP diff --git a/docs/img/workflow-h3-with-mixed-audio.png b/docs/img/workflow-h3-with-mixed-audio.png new file mode 100644 index 00000000..705a46d2 Binary files /dev/null and b/docs/img/workflow-h3-with-mixed-audio.png differ diff --git a/docs/proposals/agent-catalog-legibility.md b/docs/proposals/agent-catalog-legibility.md index 17014b09..92fa8ae4 100644 --- a/docs/proposals/agent-catalog-legibility.md +++ b/docs/proposals/agent-catalog-legibility.md @@ -3,7 +3,7 @@ Status: Parts 1–2 designed 2026-09-06 — see the [design spec](../superpowers/specs/2026-09-06-agent-catalog-legibility-design.md) and the **Ledger** at the end of this document, which records what was -actually done against each proposal. Parts 3–4 remain open. Synthesizes and replaces two prior proposals, +actually done against each proposal. Part 3 is carried as a constraint and Part 4 shipped through both channels (see the Ledger and the plugin drill at the end); no part remains open. Synthesizes and replaces two prior proposals, `catalog-shape-index.md` and `mcp-discovery-data.md` (both fully folded into this document and deleted — see git history for the original framing), plus the discovery-facing conclusion of @@ -372,7 +372,7 @@ the moment an agent is actually driving H3. Loading it by default, or even indexing it by default, works against the economy Part 1 fought to establish. -## Part 4: packaging model-specific knowledge — an open design question +## Part 4: packaging model-specific knowledge — resolved 2026-09-07 in favour of Channel B Given Part 3's constraint (data, not code) and its budget concern (don't bloat the standing MCP index), there appear to be two non-exclusive @@ -411,6 +411,18 @@ authoring knowledge independently. Whether that's practical, and whether it should ship as a plugin bundled with `dw_mcp` or live separately, is exactly the kind of specific-mechanism question this proposal defers. +**Resolved (2026-09-07, shipped 2026-09-08).** Both channels are in use, +asymmetrically: dw-generic authoring knowledge went to the guides (Channel +A, the `WORKFLOW_GUIDE.md` authoring section), and model-family knowledge +ships as the dw plugin's per-family skills (Channel B: `plugins/dw/skills/` +for MiniMax H3, MiniMax Music 3 and LTX-2.5), while `dw/server/guides.py` +deliberately indexes no model family. The two-copies risk named above is +answered by keeping the skills thin - they defer prompt format to the +vendors' own text and quote catalog names rather than catalog content - and +by `tests/test_plugin_skills.py`, which pins every number a skill states to +the diffusers module that enforces it. The paragraphs above stay as the +design record; the "Plugin drill" section below is the acceptance test. + ## Principle: format-knowledge belongs in guides and templates, not in new engine code The catalog and guides answer "which existing workflow fits", and Part 2's @@ -505,7 +517,7 @@ starts from the record rather than the intent. Updated as work lands. | 9 composition rules | done (plan 2, task 6) | spec §2.3 | the cartesian rule and the one-step-per-pair form are stated in the authoring section, the general case of `scripted-dialogue-and-tts.md`'s reasoning | | 10 save-time metadata | done (task 6) | spec §1.6 | derived and returned on save; empty summary warns, never rejects | | Part 3 constraint | carried | spec "Principle" | derivation reads structure, never model family | -| Part 4 packaging | first item through Channel A (2026-09-07) | `WORKFLOW_GUIDE.md` authoring section | the spoon-set drill (cold session, PR #48 merged) planned a shared seed for "identical but for colour", which draws a different object per prompt; `### Keeping a set consistent` now says which of style, object or character consistency wants prompts, an edit pass or a reference, and `templates/consistent-set.json` (declared `image-set`: derivation says `image-edit`) is the generate-then-edit shape on FLUX.1 Kontext, whose one pipeline both draws and edits; measured on lem 2026-09-07 at 11.5 warm minutes (2.2 for the base, 3.1 per edit, sequential offload) and the four mugs came out identical but for colour. A FLUX.2 draft found that the HF remote text encoder every FLUX.2 entry names now answers with an HTML page (broken on lem since 2026-08-26); `remote.py` now says so instead of raising an unpickling error, and both FLUX.2 entries load the repo's own 4-bit text encoder under model offload instead. The FLUX.2 text encoder is declared as its own component pinned to the CPU, because BitsAndBytes materializes on the accelerator and the two 4-bit models do not fit at load; the `flash_hub` backend went too, since it needs the `kernels` package no install has. models/flux2-dev measured at 3.3 warm minutes on lem. The Krea edit template moved to FLUX.1 Kontext for the same reason, and a catalog test now refuses `trust_remote_code` and `custom_pipeline` in any bundled entry or packaged builtin, so nothing dw ships asks an operator to lower `--trust-workflows`. The Florence-2 and Phi-3.5 builtins went with it: the `text_generation` task had replaced both, and `describe-and-regenerate` now composes it twice. `### Remote code is refused by default` came out of the same run: the first draft used the Krea edit template's `custom_pipeline`, which a server without `--trust-workflows` refuses at load. Still open: `templates/minimax/README.md` and `ltx2/README.md` are model knowledge as data, unindexed — a third channel. Catalog repair task 1: the latent handoff a two-stage flow needs is proven by test (tests/test_result.py::TestLatentHandoff), no engine change. Catalog repair task 2: two-stage.json is the three-move flow (8 sigmas at 768x448, 2x latent upsample, renoise + 3 stage-two sigmas at 1536x896, audio latents carried); latents pass by name; a test holds noise_scale to STAGE_2_DISTILLED_SIGMA_VALUES[0]. Cost and the sharpness comparison await the lem run (task 7). Catalog repair task 3: the six prompts/ltx2 captions are rewritten to the trained format (one paragraph, 150-220 words, shot type/camera motion/viewpoint in prose, sound interleaved; the two I2V ones describe only what changes), intended_model ltx-2.5, four summaries say LTX-2.5; tests/test_ltx_prompt_library.py holds the shape. Catalog repair task 4: h3_context_ir names its two source guides, writes N/A for silent audio fields, numbers