Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "diffusers-workflow",
"description": "Skills for driving a diffusers-workflow server from Claude Code: one composition skill per model family.",
"owner": {
"name": "Don Kackman"
},
"plugins": [
{
"name": "dw",
"source": "./plugins/dw",
"description": "Compose MiniMax H3 video, MiniMax Music 3 and LTX-2.5 workflows over a dw MCP server: which template fits which shape, the hard rules, cost, and how to judge the output. Prompt format comes from the vendors' own guides."
}
]
}
12 changes: 11 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,16 @@ See docs/SERVER.md, `dw/server/CLAUDE.md` and `ui/CLAUDE.md`.

The stdio MCP server lives in `dw_mcp/` — see `dw_mcp/CLAUDE.md` and docs/MCP.md.

### Claude Code plugin

`.claude-plugin/marketplace.json` publishes the `dw` plugin in `plugins/dw/`: one
composition skill per model family (`minimax-h3`, `minimax-music3`, `ltx-2.5`) that
chooses a template for a request's shape and states the family's hard rules. Model
knowledge lives there and in the catalog, never in engine code; every number a skill
states is pinned to a diffusers symbol by `tests/test_plugin_skills.py`. `plugin.json`'s
version is the engine's, bumped by `scripts/release.sh`. Adding or re-auditing a family
is `.claude/skills/model-family-onboarding/`.

### REPL Architecture

The REPL (`dw/repl.py`) uses a **persistent worker subprocess** (`dw/worker.py`) to keep GPU models cached between runs. Communication is via `multiprocessing.Queue`. Worker management is in `dw/repl_worker.py`, command handlers in `dw/repl_commands.py`.
Expand Down Expand Up @@ -103,7 +113,7 @@ read an inferred workspace back as one the user named - `get_prompt_dir` yields
to its older discovery (`./prompts`, then the walk up from the workflow file)
for an inferred workspace but not for an explicit one. `--workflow-dir`,
`--output-dir` and `--prompt-dir` each still override one folder. See
docs/WORKSPACES.md, and docs/proposals/workspaces.md for the later stages
docs/WORKSPACES.md, and docs/proposals/server-workspaces.md for the later stages
(workflow search path, run directories, `asset:`/`output:` references).

### Type System
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,15 @@ claude mcp add --transport http dw http://gpu-box:8765/mcp \
--header "Authorization: Bearer $DW_API_TOKEN"
```

The [dw plugin](plugins/dw/README.md) adds one skill per model family - what
to run for a given shape, the rules that bite, what it costs - and points at
the vendors' own prompt guides rather than restating them:

```
/plugin marketplace add dkackman/diffusers-workflow
/plugin install dw@diffusers-workflow
```

You don't have to compose that command by hand — the server's own **Server**
page builds it from the address you pick, alongside the directories it
resolved and the workspaces it holds:
Expand Down
10 changes: 9 additions & 1 deletion docs/MCP.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ call; that probe is fatal for a remote URL and only a warning for a
loopback one (where it usually means `dw.serve` is not up yet).
[REMOTE.md](REMOTE.md) covers the remote setup end to end.

Claude Code users can add the composition skills as well:
`/plugin marketplace add dkackman/diffusers-workflow` then
`/plugin install dw@diffusers-workflow`. The plugin ships one skill per model
family (MiniMax H3, MiniMax Music 3, LTX-2.5) that picks a template for a
request's shape and states the family's rules - see
[plugins/dw/README.md](../plugins/dw/README.md). It is optional; every tool
below works without it.

### Use the absolute path to `dw-mcp`

**This is the one setup detail that reliably goes wrong.** If you installed
Expand Down Expand Up @@ -224,7 +232,7 @@ when no single workflow covers it.
| --- | --- | --- |
| `get_output_image(name, max_dimension=768)` | `name`, `max_dimension` | Look at a generated image, downscaled to `max_dimension` on its longest side. Returns the image plus a text part reporting `original_size`, `returned_size` and `bytes`, so a downscale is never silent |
| `get_output_text(name, max_characters=20000)` | `name`, `max_characters` | Read a text output — a prompt enhancement, or any step whose result is `text/plain` or JSON. Reports the file's real length and whether it was truncated |
| `download_output(name, destination=None, overwrite=False)` | `name`, `destination`, `overwrite` | Save one output file to local disk, of any content type. `destination` may be a full path, a directory, or omitted to save under the output's own name in the current working directory; `~` expands and missing parent directories are created. `overwrite=True` is required to replace a file already at the resolved path. Returns nothing to the conversation but where the file landed — unlike the other media tools, the point is a file on disk, not a payload in context. Writes on the machine running the MCP server - over `dw.serve --mcp` that is the GPU box |
| `download_output(name, destination=None, overwrite=False)` | `name`, `destination`, `overwrite` | Save one output file to local disk, of any content type. `destination` may be a full path, a directory, or omitted to save under the output's own name in the current working directory; `~` expands and missing parent directories are created. `overwrite=True` is required to replace a file already at the resolved path. Returns nothing to the conversation but where the file landed — unlike the other media tools, the point is a file on disk, not a payload in context. Writes on the machine running the MCP server - over `dw.serve --mcp` that is the GPU box. A write that fails there (a path that exists only on the client, for instance) comes back as an error naming the server-side write and the client-side alternatives, not as an anonymous tool failure |
| `delete_output(name)` | `name` | Permanently remove one generated file from the output directory |

### Authoring, assets and workspaces
Expand Down
2 changes: 1 addition & 1 deletion docs/QUANTIZATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ through `from_pretrained_arguments`, so quantization is keyed by component name
"configuration": { "config_type": "TorchAoConfig" },
"arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" }
},
"text_encoder": {
"language_model": {
"configuration": { "config_type": "transformers.TorchAoConfig" },
"arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" }
}
Expand Down
14 changes: 14 additions & 0 deletions docs/RECIPES_24GB.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,20 @@ workflows below fit; without it, the load is an OOM kill, not a slowdown.

**Examples:** [reference-to-video.json](../workflows/templates/minimax/reference-to-video.json), [chain-matched-to-audio.json](../workflows/templates/minimax/chain-matched-to-audio.json), [image-to-video.json](../workflows/templates/minimax/image-to-video.json), [dialogue-short.json](../workflows/templates/minimax/dialogue-short.json) (five ref2va shots + two Z-Image portraits in ~35 minutes end to end)

## MiniMax-Music3

Music3 runs at about 22GiB in bfloat16 under the templates' `components_manager`
auto CPU offload, which keeps only the running component resident; no quantization
is needed on a 24GB card. The language model is the part worth offloading harder:
a leaf-level `group_offload` of `language_model` brings it to about 8GiB (the model
card's low-VRAM recipe). Two things the examples carry: `release_pipeline` on the
music step in any workflow that loads H3 afterwards, since host RAM is the binding
constraint (see Multi-model workflows below), and the run's time follows the length
the model actually sings, not `audio_duration`, which is a ceiling of at most 9000
frames at 25 frames per second (360 seconds). Output is 44.1 kHz stereo.

**Examples:** [music.json](../workflows/templates/minimax/music.json), [music-video.json](../workflows/templates/minimax/music-video.json)

## LTX-2.5 (22B, video + audio)

A standard pipeline, but placed per component rather than with a pipeline-level
Expand Down
11 changes: 7 additions & 4 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,23 @@ scripts/release.sh 0.38.0-alpha.1 "UI front end" # optional tag message
```

The script bumps `pyproject.toml` (the single source of the version —
`dw.__version__` reads it at runtime), commits just that file, pushes
`dw.__version__` reads it at runtime) and sets the same version in
`plugins/dw/.claude-plugin/plugin.json`, so an installed plugin names the
engine it was written against; it commits just those two files, pushes
master, tags the bump commit `v0.38.0`, and pushes the tag. It refuses
a malformed version, a branch other than master, an existing tag, or a
dirty index (unstaged changes elsewhere are fine — the release commit
is path-limited to pyproject.toml).
is path-limited to those two files).

By hand, the equivalent is:

```bash
# 1. Bump the version in pyproject.toml:
# version = "0.38.0"
git commit -m "release 0.38.0" -- pyproject.toml
# 2. Set the same version in plugins/dw/.claude-plugin/plugin.json
git commit -m "release 0.38.0" -- pyproject.toml plugins/dw/.claude-plugin/plugin.json

# 2. Tag the bump commit and push
# 3. Tag the bump commit and push
git tag -a v0.38.0 -m "release 0.38.0"
git push origin master v0.38.0
```
Expand Down
12 changes: 6 additions & 6 deletions docs/TASKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,7 @@ returns frames without it, and this puts it back:
| Argument | Required | Description |
| -------- | -------- | ----------- |
| `video` | Yes | The frames - a frame list, a frame array or tensor, or an audio+video pair whose own soundtrack is replaced |
| `audio` | Yes | The soundtrack - a waveform, or the earlier step whose video carried one, which brings its sample rate along |
| `audio` | Yes | The soundtrack - a waveform, the earlier step whose video carried one, or the path or URL of an audio or video file; the last two bring their sample rate along |
| `sample_rate` | No | Sample rate of the waveform. Required unless `audio` carries one; given here it wins |

**Example:** [assemble-and-score.json](../workflows/templates/assemble-and-score.json)
Expand Down Expand Up @@ -419,7 +419,7 @@ a length still passes the whole track along:

| Argument | Required | Description |
| -------- | -------- | ----------- |
| `audio` | Yes | Path or URL of an audio file, a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) |
| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) |
| `start_seconds` / `duration_seconds` | One pair | The slice in seconds; either may be omitted |
| `start_frame` / `num_frames` / `fps` | One pair | The slice in video frames; `fps` is required, start and count may be omitted |
| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) |
Expand Down Expand Up @@ -465,7 +465,7 @@ ending. The curve is the equal-power cosine the seam joins use:

| Argument | Required | Description |
| -------- | -------- | ----------- |
| `audio` | Yes | Path or URL of an audio file, a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) |
| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) |
| `fade_in_ms` | No | Length of the fade in, from the head of the track (default: 0) |
| `fade_out_ms` | No | Length of the fade out, to the tail of the track (default: 0) |
| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) |
Expand Down Expand Up @@ -494,7 +494,7 @@ changes, so the dynamics survive:

| Argument | Required | Description |
| -------- | -------- | ----------- |
| `audio` | Yes | Path or URL of an audio file, a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) |
| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) |
| `peak_dbfs` | No | The level the loudest sample is moved to, in dB below full scale (default: -1.0). 0 is full scale |
| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) |

Expand Down Expand Up @@ -524,7 +524,7 @@ at every cut:

| Argument | Required | Description |
| -------- | -------- | ----------- |
| `audios` | Yes | The tracks to layer - waveforms, audio file paths, or videos generated with a soundtrack |
| `audios` | Yes | The tracks to layer - waveforms, audio or video file paths, or videos generated with a soundtrack |
| `gains` | No | One plain multiplier per track, in the same order - not decibels. Defaults to unity on every track |
| `sample_rate` | With a raw waveform | Sample rate of the waveforms. Required unless every track brings its own; given here it wins |

Expand Down Expand Up @@ -556,7 +556,7 @@ supplied recording once, up front, feeds it what it already wants:

| Argument | Required | Description |
| -------- | -------- | ----------- |
| `audio` | Yes | Path or URL of an audio file, a video generated with a soundtrack (which brings its sample rate along), or a waveform |
| `audio` | Yes | Path or URL of an audio or video file, a video generated with a soundtrack (which brings its sample rate along), or a waveform |
| `target_sample_rate` | Yes | The rate to convert to |
| `sample_rate` | With a waveform | Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry |

Expand Down
8 changes: 5 additions & 3 deletions docs/WORKFLOW_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -557,7 +557,7 @@ once, not twice.
- **Ignored when the component's device is the CPU**, where there is nothing to move it
off of.

On a 24GB card, `MiniMaxH3Ref2VA.json` peaks at 18.9GiB of reserved VRAM with on-demand
On a 24GB card, `templates/minimax/reference-to-video.json` peaks at 18.9GiB of reserved VRAM with on-demand
VAEs against 23.2GiB resident, and the tighter resident fit costs 40 allocator retries -
cache flushes forced by a failed allocation - where the on-demand run has none. The
headroom is also what lets the chained variant run: its later segments carry an extra
Expand Down Expand Up @@ -815,7 +815,7 @@ and `load_components` pulls the weights:
"modules_to_not_convert": ["proj_in", "proj_out"]
}
},
"text_encoder": {
"language_model": {
"configuration": { "config_type": "transformers.TorchAoConfig" },
"arguments": { "quant_type": "torchao.quantization.Int8WeightOnlyConfig" }
}
Expand Down Expand Up @@ -1172,7 +1172,9 @@ directory being copied. `latest` only selects a run where run directories are; a
workflow or file that happens to be called `latest` is still named as itself.

Like `asset:`, a reference resolves to a path and then whatever loads paths loads it, so
it works under `image`, `video`, a `from_file`, or a list of them. It resolves against
it works under `image`, `video`, a `from_file`, or a list of them. The audio tasks take
a video file's path too and use the soundtrack muxed into it, which is how a finished
cut is scored in a later run without re-cutting it. It resolves against
the output directory the run was told to write to, and cannot leave it: `..`, an
absolute path, and a symlink pointing out are all refused.

Expand Down
2 changes: 1 addition & 1 deletion docs/WORKSPACES.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ it. Use them to keep work apart, not to keep it private.
## Where this is going

Workspaces were the first stage of the design in
[proposals/workspaces.md](proposals/workspaces.md). The resolver, the workflow
[proposals/server-workspaces.md](proposals/server-workspaces.md). The resolver, the workflow
search path with writes confined to the writable root, run directories with an
on-disk manifest, `asset:` and `output:` references, and server-side named
workspaces are all implemented. What remains from the proposal is an MCP
Expand Down
Binary file added docs/img/workflow-h3-with-mixed-audio.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading