From ee0f032c8b25b846fe3b52d7744c443884b55ef4 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Mon, 7 Sep 2026 18:53:50 -0500 Subject: [PATCH] Part 4 design: catalog repair and plugin skills specs, with the H3 and LTX-2.5 vendor audits Two vendor-source audits found the H3 Context-IR builtin is a faithful compression of MiniMax's published guides with three defects, and that the LTX-2.5 two-stage template skips its refinement pass and the LTX prompt library is in a genre the model was not trained on. Both vendors publish the prompt format themselves, so the plugin's skills teach composition and defer format to those sources. Co-Authored-By: Claude Fable 5.1 --- .../audits/2026-09-07-ltx-2.5-audit.md | 244 +++++++++++++ .../audits/2026-09-07-minimax-h3-audit.md | 339 ++++++++++++++++++ .../2026-09-07-dw-plugin-skills-design.md | 155 ++++++++ ...2026-09-07-ltx-h3-catalog-repair-design.md | 187 ++++++++++ 4 files changed, 925 insertions(+) create mode 100644 docs/proposals/audits/2026-09-07-ltx-2.5-audit.md create mode 100644 docs/proposals/audits/2026-09-07-minimax-h3-audit.md create mode 100644 docs/superpowers/specs/2026-09-07-dw-plugin-skills-design.md create mode 100644 docs/superpowers/specs/2026-09-07-ltx-h3-catalog-repair-design.md diff --git a/docs/proposals/audits/2026-09-07-ltx-2.5-audit.md b/docs/proposals/audits/2026-09-07-ltx-2.5-audit.md new file mode 100644 index 0000000..32d6c22 --- /dev/null +++ b/docs/proposals/audits/2026-09-07-ltx-2.5-audit.md @@ -0,0 +1,244 @@ +# LTX-2.5 knowledge audit — `diffusers-workflow` + +Research date: 2026-09-07. Repo branch `agent-legibility`, ltx2 templates last touched 2026-09-07. +Installed library: `diffusers 0.41.0.dev0` at +`/Users/don/src/dkackman/diffusers-workflow/venv/lib/python3.14/site-packages/diffusers`. + +## Sources + +| URL | Type | Date | Contributes | +| --- | --- | --- | --- | +| `venv/.../diffusers/pipelines/ltx2/utils.py` (installed 0.41.0.dev0) | primary (library source) | as installed, read 2026-09-07 | `DISTILLED_SIGMA_VALUES` (8), `STAGE_2_DISTILLED_SIGMA_VALUES` (3), `DEFAULT_NEGATIVE_PROMPT`, `GEMMA4_PROMPT_ENHANCEMENT_CONFIG`, `LTX2_5_T2V/I2V_DEFAULT_SYSTEM_PROMPT`, CRF 18, `MAX_CONDITIONING_FPS=60` | +| `venv/.../ltx2/pipeline_ltx2.py`, `_image2video.py`, `_condition.py`, `_ic_lora.py`, `_latent_upsample.py`, `_dfr.py` | primary (library source) | read 2026-09-07 | `__call__` defaults, `check_inputs` (w/h % 32), duration head, enhancer defaults, `LTX2VideoCondition`, `reference_downscale_factor` | +| https://huggingface.co/Lightricks/LTX-2.5-Diffusers | primary | fetched 2026-09-07 (raw README 401 — gated; HTML card fetched OK) | Distilled recipe: sigmas + guidance 1.0/STG 0/modality 1.0; "half resolution, x2 latent upsample, then a 3-sigma tail at full resolution"; prompt-style statement; dedicated Gemma-4 enhancer; arXiv 2601.03233 | +| https://huggingface.co/Lightricks/LTX-2.5 | primary | fetched 2026-09-07; release stated 2026-01-06 | `num_frames % 8 == 1`, w/h divisible by 32, 8-step distilled at CFG 1.0, component file list, upscalers | +| https://huggingface.co/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler | primary | fetched 2026-09-07 | Weight filename, LoRA strength 1.0, `reference_downscale_factor` 2, ~280p draft, "clean low-res render" requirement | +| https://github.com/Lightricks/LTX-2 `README.md` (via `gh api`) | primary | fetched 2026-09-07 | Prompting section (≤200 words, single flowing paragraph, chronological, 7-part structure); DFR as production path; full model list; Gemma-4-12B fine-tuned encoder | +| `.../LTX-2/packages/ltx-pipelines/docs/pipelines.md` | primary | fetched 2026-09-07 | DistilledPipeline = "two-stage, 8 predefined sigmas (8 steps stage 1, **4 steps stage 2**)"; DFR details; 8k+1 grid; DFR w/h % 64; fps snapping | +| `.../LTX-2/packages/ltx-pipelines/docs/pipeline-selection.md` | primary | fetched 2026-09-07 | Pipeline decision tree; DistilledPipeline "starting point", DFR "production quality"; ICLoraPipeline for video conditioning | +| `.../LTX-2/packages/ltx-pipelines/docs/conditioning.md` | primary | fetched 2026-09-07 | Image conditioning by replacing vs guiding latents; video conditioning listed ICLoraPipeline-only; generated keyframe slots | +| `.../LTX-2/CHANGELOG.md` | primary | 1.2.0 dated **2026-08-11** ("Support for LTX 2.5"); 1.3.0 dated **2026-08-25** | Dates the LTX-2.5 tooling release; DFR, duration head, dedicated enhancer, NVFP4, HDR | +| `.../LTX-2/packages/ltx-core/.../gemma4_t2v_system_prompt.txt` | primary | fetched 2026-09-07 | Byte-identical to diffusers' `LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT`; "roughly 150–220 words" | +| `.../LTX-2/packages/ltx-pipelines/src/ltx_pipelines/utils/constants.py` | primary | fetched 2026-09-07 | Upstream sigma lists (with trailing 0.0), CRF 33/18, stage-1 512x768 / stage-2 1024x1536, HQ 544x960, 121 frames @ 24 fps, guidance defaults | +| https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx2 | primary (library docs) | fetched 2026-09-07 | Explicit two-stage code recipe incl. `noise_scale=STAGE_2_DISTILLED_SIGMA_VALUES[0]` renoise before the 3-sigma tail; offload/tiling guidance | +| https://runware.ai/docs/models/lightricks-ltx-2-5-pro/guides/prompting | secondary | model release date given as 2026-08-11 | Present tense; six-part scaffold in one flowing paragraph; audio integral; camera motion in its own clause; "no music" to suppress score | +| https://ltx.io/blog/ltx-2-5-prompt-guide and /blog/prompting-guide-for-ltx-2 | secondary | — | **Fetch failed** (HTTP header overflow, both URLs). Content only reached indirectly via search snippets. | +| WebSearch summaries (fal.ai, HackerNoon, Dream Pixel Forge, Prompt Architects, ltx.io excerpts) | secondary | 2026 | Restate the model card / GitHub README: single chronological paragraph, audio always described, dialogue in quotes, match length to complexity | +| https://github.com/Lightricks/ComfyUI-LTXVideo `example_workflows/2.5/LTX-2.5_T2V_I2V_Two_Stage_Distilled.json` | primary (not fetched, surfaced by search) | 2026 | Corroborates 8 steps stage 1 → 2x upscale → +noise → 3 steps stage 2 | + +Note on secondary weight: every secondary prompting source found restates the GitHub README / +model card almost verbatim (single flowing paragraph, chronological, ~200 words, describe audio). +None contradicts a primary source, and none adds independent evidence. I therefore lean entirely on +the primary system prompt text (`gemma4_t2v_system_prompt.txt`), which is the actual training-caption +specification. + +--- + +## Claim-by-claim verdict + +### `workflows/templates/ltx2/README.md` + +| Claim | Verdict | +| --- | --- | +| "LTX-2.5" is the model | **CONFIRMED** — `Lightricks/LTX-2.5-Diffusers` is the current Diffusers-format repo; LTX-2.5 released 2026-01-06, tooling support 2026-08-11 (LTX-2 CHANGELOG 1.2.0). | +| Links "LTX-2.5" to `https://huggingface.co/Lightricks/LTX-Video` | **CONTRADICTED** — `LTX-Video` is the earlier, separate LTX-Video line. The LTX-2.5 repos are `Lightricks/LTX-2.5` (weights) and `Lightricks/LTX-2.5-Diffusers` (diffusers format). Wrong release entirely. | +| Every table row links `LTX2.json`, `LTX2I2V.json`, `LTX2Keyframes.json`, `LTX2I2VEnhancePrompt.json`, `LTX2TwoStage.json`, `LTX2ICLora.json`, `LTX2Extend.json`, `LTX2I2VChained.json` | **CONTRADICTED by the repo itself** — none of those files exist. The eight files are `text-to-video.json`, `image-to-video.json`, `keyframes.json`, `enhance-prompt.json`, `two-stage.json`, `generative-upscale.json`, `extend-clip.json`, `chained-segments.json`. Those are the workflow `id`s, not paths. Every link in the reading-order map is broken. | +| "fixed eight-step distilled schedule" | **CONFIRMED** — `DISTILLED_SIGMA_VALUES` has 8 entries (upstream's 9 includes the trailing 0.0 that the scheduler appends); pipelines.md: "8 predefined sigmas (8 steps in stage 1…)". | +| "quantized per component" | Repo-specific engineering; sources are silent (they document `--quantization fp8-cast` / `nvfp4-*`, not SDNQ). **UNSOURCED**, not contradicted. | +| "VAE tiling for the longer clip" | **CONFIRMED** — diffusers docs recommend `pipe.vae.enable_tiling()` for high resolution; the conv VAE is the only decoder that tiles. | +| Two-stage is "the recommended quality flow" | **PARTLY CONTRADICTED** — as of LTX-2 1.2.0 (2026-08-11) the recommended *quality* path is `DFRPipeline`; DistilledPipeline's two-stage is described as the "fastest inference … starting point". And the repo's two-stage omits stage 2 (see below). | +| IC-LoRA is "a generative 2x upscale … inventing detail" | **CONFIRMED** — the IC-LoRA card uses almost exactly that language ("re-rendering the scene at higher resolution, inventing plausible fine detail"). | +| Extend "conditioning on it in full" | **CONFIRMED as mechanically valid** by `LTX2VideoCondition` ("a single frame or a sequence of frames"). Lightricks' own `conditioning.md` restricts whole-video conditioning to `ICLoraPipeline`, so the flow is a diffusers capability rather than a documented Lightricks recipe. **UNSOURCED as a recommendation.** | +| Chained segments on last frame | Repo engine feature (`chain`, `trim_frames`, `crossfade_ms`). **UNSOURCED** — no Lightricks guidance either way. Note upstream would reach the same goal with the duration head (up to 20 s) or DFR. | + +### Shared template settings (all eight) + +| Claim | Verdict | +| --- | --- | +| `sigmas: constant:…DISTILLED_SIGMA_VALUES`, no `num_inference_steps` | **CONFIRMED** — model card: "No `num_inference_steps` parameter; use explicit sigma schedule instead"; `sigmas=DISTILLED_SIGMA_VALUES` (required for quality). | +| `guidance_scale: 1.0`, `audio_guidance_scale: 1.0` | **CONFIRMED** — model card lists exactly these for the distilled transformer. | +| `stg_scale: 0.0`, `audio_stg_scale: 0.0` | **CONFIRMED** — model card. | +| `modality_scale: 1.0`, `audio_modality_scale: 1.0` | **CONFIRMED** — model card. | +| `negative_prompt: constant:…DEFAULT_NEGATIVE_PROMPT` | **CONFIRMED** — the diffusers constant carries an upstream provenance comment pointing at `ltx-pipelines` `constants.py`. Note upstream's own current default begins `has_subtitles, has_blurbox, transition from black…`, so the two lists have drifted; the diffusers copy is pinned to commit `ae855f8`. Minor, low-risk. | +| `frame_rate: 24.0` / `result.fps: 24` | **CONFIRMED** — 24 fps is the documented default; the transformer is trained around 24/25/30 and 60. | +| `num_frames` values 121, 241, 481 | **CONFIRMED** — all satisfy `8k+1` (`num_frames % 8 == 1`), the VAE temporal grid. 121 ≈ 5.04 s at 24 fps. | +| Resolutions 960x544, 768x448, 768x768, 480x288, 960x576 | **CONFIRMED** — every one is divisible by 32, the documented spatial rule. 544x960 is upstream's named HQ stage-1 preset. | +| `torch_dtype: torch.bfloat16` throughout | **CONFIRMED** — bf16 is the published weight dtype and the recommended runtime dtype. | +| `text_encoder` = `transformers.Gemma4UnifiedForConditionalGeneration` from the LTX-2.5 repo's `text_encoder` subfolder | **CONFIRMED** — Gemma 4 12B, fine-tuned for LTX, bundled; the README warns stock Google Gemma 4 is *not* a substitute (version check against `gemma4-12b-ltx-v1`). | +| `duration_head` component present | **CONFIRMED** — LTX-2.5 addition (CHANGELOG 1.2.0, 2026-08-11). | +| SDNQ uint4 transformer / int8 text encoder, `group_offload leaf_level use_stream` | **UNSOURCED** — Lightricks documents fp8-cast / NVFP4 / `--offload cpu|disk`; SDNQ and diffusers group-offload are outside their guidance. Not contradicted, but nothing upstream validates uint4 for the 22B distilled transformer. | + +### `text-to-video.json` (id `LTX2`) + +- Everything above: **CONFIRMED**. +- Description "The distilled transformer runs a fixed eight-step schedule with guidance off": **CONFIRMED**. +- Default prompt `prompt:ltx2/fox_dawn_choir`: **CONTRADICTED as prompt style** — see the prompt section below. +- 960x544 as the base resolution: **CONFIRMED** as a tested/HQ-preset stage-1 size. + +### `image-to-video.json` (id `LTX2I2V`) + +- `LTX2ImageToVideoPipeline`, 768x768, 481 frames (20 s at 24 fps): **CONFIRMED** as legal (768 % 32 = 0; 481 = 8·60+1; 20 s is the enhancer/duration-head upper bound). +- **UNSOURCED / worth flagging**: the pipeline re-compresses a single-frame image conditioning at H.264 CRF **18** for LTX-2.5 (`resolve_default_image_crf`) to match training compression, and this requires a `PIL.Image.Image` plus PyAV — otherwise `check_inputs` raises. The repo carries no note of this anywhere. +- No mention that an already-compressed / degraded input still gets re-compressed. Not an error, but undocumented. + +### `keyframes.json` (id `LTX2Keyframes`) + +- `LTX2ConditionPipeline` with two `LTX2VideoCondition`s at `index: 0` and `index: -1`, `strength: 1.0`: **CONFIRMED** by the diffusers source and the docs page ("arbitrary latent frame conditioning (FLF2V, etc.)"). +- `strength: 1.0` for both endpoints: **CONFIRMED as the meaning** ("fully applied"). Lightricks' `conditioning.md` distinguishes *replacing* latents (strong control) from *guiding* latents ("better for smooth interpolation between keyframes") — for a first/last-frame interpolation their guidance leans toward the guiding variant, i.e. a strength below 1.0 on the endpoints. Compare `ANCHOR_KEYFRAME_STRENGTH = 0.95` in `utils.py`, used for exactly this reason ("pinned just short of fully clean so a tile can still settle its seam frame"). **Not contradicted, but arguably the wrong end of the range.** +- Uses `flf2v_input_first/last_frame.png` from the HF docs-images dataset — those are Wan FLF2V demo assets, unrelated to LTX. Cosmetic. + +### `enhance-prompt.json` (id `LTX2I2VEnhancePrompt`) + +- Dedicated `prompt_enhancer` = `google/gemma-4-E2B-it` with `AutoProcessor`, separate from the text encoder: **CONFIRMED** — model card: "a dedicated `google/gemma-4-E2B-it` `prompt_enhancer` component (the fine-tuned text encoder isn't trained for enhancement)"; CHANGELOG 1.2.0 added `--prompt-enhancer-gemma-root`. +- `enable_prompt_enhancement: true` with no `system_prompt`: **CONFIRMED correct** — `pipeline_ltx2_image2video.py:1270` defaults to `LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT`, which is byte-identical to upstream `gemma4_i2v_system_prompt.txt`. +- `prompt_max_new_tokens: constant:…GEMMA4_PROMPT_ENHANCEMENT_CONFIG.max_new_tokens` (600): **CONFIRMED but redundant** — the pipeline selects that config automatically when a dedicated enhancer is present (`pipeline_ltx2_image2video.py:603`). +- Does **not** pass `prompt_enhancement_kwargs`: **CONFIRMED harmless** — defaults resolve to `{"do_sample": False, "no_repeat_ngram_size": 5}` from the same config. +- Enhancer on `device: "cpu"` with `uint4` SDNQ: **UNSOURCED**; a 2B enhancer at uint4 on CPU is an aggressive combination for a component whose whole job is careful long-form generation. Nothing upstream speaks to it. +- Duration head via `min_seconds: 2.0` / `max_seconds: 8.0`: **CONFIRMED** as a legal range (pipeline defaults 1.0/20.0; `min < max` enforced). Predicted counts snap to the 8k+1 grid (~0.33 s at 24 fps). +- README's claim that the enhancer "rewrites a one-line idea into a trained-format prompt": **CONFIRMED** — that is precisely what `LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT` instructs (150–220 words, single paragraph, framing triple, integrated soundscape). + +### `two-stage.json` (id `LTX2TwoStage`) — the biggest gap + +- Claimed as "LTX-2.5's recommended quality flow: render small, then upscale in latent space rather than at full size." +- Implemented as: base 768x448 (8 sigmas) → `LTX2LatentUpsamplePipeline` → `pair_audio`. +- **CONTRADICTED.** Every primary source describes the distilled two-stage as *three* moves, not two: + - LTX-2.5-Diffusers model card: "half resolution, x2 latent upsample, **then a 3-sigma tail at full resolution**" — "Stage 2 decoding with `STAGE_2_DISTILLED_SIGMA_VALUES`". + - `pipelines.md` §4 DistilledPipeline: "Two-stage generation with 8 predefined sigmas (**8 steps in stage 1, 4 steps in stage 2**)." (Upstream's 4-value stage-2 list includes the trailing 0.0; diffusers' 3-value constant is the same schedule.) + - diffusers docs page gives the code: re-enter the *same* pipeline with `latents=upscaled_video_latent`, `audio_latents=audio_latent`, `sigmas=STAGE_2_DISTILLED_SIGMA_VALUES`, and crucially `noise_scale=STAGE_2_DISTILLED_SIGMA_VALUES[0]` to renoise before denoising. + - ComfyUI-LTXVideo's own `LTX-2.5_T2V_I2V_Two_Stage_Distilled.json` example corroborates. + - `STAGE_2_DISTILLED_SIGMA_VALUES` exists in the installed `utils.py` and is referenced **nowhere else in diffusers** — it is there for callers to use, and the repo never uses it. +- Consequence: the repo's "two-stage" is a bare latent upsample with no refinement pass. The upsampler alone produces a soft 2x; the detail is supposed to come from the 3-sigma tail. The claim that the result "is sharper than a single pass at 1536x896" (RECIPES_24GB) is therefore **UNSOURCED and probably false as built**. +- Also **CONTRADICTED**: the flow's framing as the *recommended quality* path. Since 2026-08-11, Lightricks route production quality to `DFRPipeline` (`LTX2DFRPipeline` exists in the installed diffusers), which is stage 1 at half res + generated keyframe slots + a full-res detailing pass with the same IC-LoRA. The repo ships neither. +- `pair_audio` re-attaching stage-1 audio: **CONFIRMED as the right instinct** — "Audio comes from stage 1… nothing refines audio after stage 1." But the correct implementation carries `audio_latents` into stage 2 so video↔audio cross-attention still has a stream; muxing the decoded soundtrack afterwards is a reasonable substitute only because there is no stage 2 here. +- `adain_factor: 0.0`, `tone_map_compression_ratio: 0.0` (both library defaults, i.e. off): **CONFIRMED as defaults**; no source recommends nonzero. +- Passing the *input* (base) `width`/`height` to the upsampler: **CONFIRMED correct** — the docstring says these are "the input video (not the generated video, which will have a larger resolution)". +- Passing `video=` pixels rather than `latents=`: **CONTRADICTED as best practice** — the diffusers recipe keeps stage 1 at `output_type="latent"` and hands latents straight across; the repo decodes to pixels and re-encodes, a lossy round trip through the VAE for no benefit. + +### `generative-upscale.json` (id `LTX2ICLora`) + +- LoRA repo, weight filename `ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors`, `scale: 1.0`: **CONFIRMED** exactly by the IC-LoRA model card ("LoRA strength 1.0, pre-scaled default") and the LTX-2 README download command. +- `reference_downscale_factor: 2`: **CONFIRMED** — card gives 2. +- `conditioning_attention_strength: 1.0`: **CONFIRMED as the library default**; card is silent. +- Base 480x288 → 960x576: **CONFIRMED and well-aimed** — the card names "~280p (draft generation)" as the base and 2x the reference as output. 288p is a good read of that. +- `reference_conditions` with `LTX2ReferenceCondition(frames=…, strength=1.0)`: **CONFIRMED** by the diffusers `pipeline_ltx2_ic_lora.py` example. +- Card note "fewer steps and lower guidance keep output closer to reference; more steps and higher guidance allow more creative hallucination" — the repo pins the 8-sigma distilled schedule at guidance 1.0 and offers no variable for this trade-off. **Missing knowledge**, not an error. +- Card note "reference should be a clean low-resolution render, not heavily compressed or degraded" — satisfied here (the reference is the in-memory stage-1 output), but not stated anywhere in the repo, so a user pointing this at a downloaded clip gets no warning. +- **Engineering flag (not a knowledge claim):** step 2 `reused_components` the transformer from step 1 and then loads a LoRA onto it. Whether the shared transformer is mutated for anything downstream depends on the engine; worth a look, but out of scope here. + +### `extend-clip.json` (id `LTX2Extend`) + +- `LTX2ConditionPipeline`, whole 121-frame opening as one condition at `index: 0`, `strength: 1.0`, `num_frames: 241`: **mechanically CONFIRMED** by `LTX2VideoCondition` (accepts a frame sequence) and by the docstring note "Multi-frame video conditions are not re-compressed." +- As a *recommended* extend recipe: **UNSOURCED**. Lightricks has no "extend" pipeline; `conditioning.md` scopes whole-video conditioning to `ICLoraPipeline`, and the nearest documented equivalent for making a longer clip is the duration head (to 20 s) or `RetakePipeline` for regenerating a region. Nothing contradicts the flow; nothing endorses it either. +- Two prompts (opening + continuation): sensible, unsourced. + +### `chained-segments.json` (id `LTX2I2VChained`) + +- Repo engine feature end to end (`chain.continuity: last_frame`, `trim_frames: 2`, `crossfade_ms: 80`). **UNSOURCED** in all Lightricks material. +- Worth noting against a primary source: each segment's start frame is a single image conditioning, so each segment pays the CRF-18 re-compression, and the LTX-2.5 duration head plus a 481-frame single pass (already used in `image-to-video.json`) reaches 20 s without stitching. The chain earns its place only past that. + +### Prompt style — `prompts/ltx2/*.json` (used as every template's default) + +This is where the repo diverges most from primary guidance. + +Lightricks' training-caption specification (`gemma4_t2v_system_prompt.txt`, byte-identical to +`LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT` in the installed diffusers) requires: + +1. Begin immediately with the action — never "The scene opens…", "We see…", "There is…". +2. Objective, observable description only; no inferred emotion. +3. Full visual detail: environment materials/textures/lighting/colors, character appearance, spatial positions. +4. A **framing triple woven into prose** for every shot: shot type (one of six named sizes), camera motion (always stated, explicitly "static" if none), and viewpoint relative to subject. +5. Complete soundscape integrated **chronologically alongside the action**, not appended. +6. Strict chronological real-time flow with connectors ("Initially…", "A moment later…", "Simultaneously…"). +7. **One continuous paragraph, roughly 150–220 words.** No labels, no bullets. + Dialogue quoted exactly. Present-progressive verbs. + +Against that: + +| Prompt | Words | Verdict | +| --- | --- | --- | +| `fox_dawn_choir` (default of `text-to-video.json`, `two-stage.json`) | ~35 | **CONTRADICTED**. Far under 150–220. Worse, it ends with tag-style modifiers — "cinematic lighting, 8k, ultra-detailed, photorealistic in the distance" — which is Stable-Diffusion-era booru style, not a caption. The training spec explicitly wants quality descriptors "woven naturally into the same observable prose", never as tags. The trailing "in the distance" is also a dangling fragment. No shot type, no camera-motion statement, no viewpoint. | +| `hummingbird_garden` (`generative-upscale.json`) | ~34 | **CONTRADICTED** — too short; no framing triple; audio is a separate trailing sentence rather than integrated chronologically. | +| `lighthouse_keeper` (`extend-clip.json`) | ~35 | **CONTRADICTED** — same shape. | +| `lighthouse_keeper_gallery` | ~55 | **CONTRADICTED** — closer (has a chronological "then", quoted dialogue "still burning" ✓) but still ~1/3 the target length, no framing triple. | +| `marmot_robot_overlords` (`image-to-video.json`, `chained-segments.json`) | ~90 | **Partly CONFIRMED, partly CONTRADICTED**. Best of the set: quoted dialogue ✓, present tense ✓, detailed environment ✓, subject appearance ✓. But it *is* an I2V prompt and the I2V spec says "describe only changes from the image; don't reiterate established visual details" — this reiterates them at length. It also has zero audio beyond the sung line, no camera-motion statement, and closes with image-generation vocabulary ("The image is highly detailed, with realistic textures and vibrant colors") — "image", not video, and "vibrant colors" is exactly the intensified-color phrasing the spec forbids ("Use plain terms ('red dress'), not intensified"). | +| `polaroid_lighthouse` (`keyframes.json`) | ~50 | **CONTRADICTED** on length and framing triple; the audio ("gulls call overhead") is appended rather than interleaved. | + +Also: `"intended_model": "ltx-2"` on all six, and several template `summary` fields say "LTX-2" where +the workflows target LTX-2.5. **Minor CONTRADICTION** — LTX-2, LTX-2.3 and LTX-2.5 are distinct +releases with different text encoders (Gemma 3 vs Gemma 4) and different image-conditioning CRF. + +`enhance-prompt.json`'s inline default `"a marmot in a top hat and monocle sings to the camera"` is +**CONFIRMED correct** for its purpose — the enhancer's whole job is to take a one-line idea, so a short +prompt is right *there* and only there. + +### `docs/RECIPES_24GB.md` LTX-2.5 section + +| Claim | Verdict | +| --- | --- | +| Per-component placement rather than a pipeline-level `offload` | **UNSOURCED** (repo engineering); not contradicted. | +| "`transformer` is the distilled model… fixed 8-step schedule at `guidance_scale: 1.0`, STG and modality guidance off, and the `sigmas`… are its trained schedule — not a knob" | **CONFIRMED**, and well put. | +| "`num_inference_steps`, `guidance_scale`, `stg_scale` and the rest only mean anything against `subfolder: "transformer_full"`, the dev model" | **CONFIRMED in substance** — the dev/full 22B transformer is what the guided TI2Vid/Keyframe/A2Vid pipelines take, at 30 steps / CFG 3.0 / STG 1.0 / modality 3.0 / rescale 0.7 / STG block 28 (or 29 in `constants.py`). The specific subfolder name `transformer_full` I could not verify against the gated Diffusers repo file listing — **UNVERIFIED**. | +| "the same ~38GB in bf16" for the dev transformer | 22B at bf16 ≈ 44GB; the cards say 22B (the Diffusers card page also says 19B, which is stale/2.3-era). **UNVERIFIED / likely off**. | +| "the checkpoint ships a diffusion decoder that `LTX2Pipeline` ignores" | **CONFIRMED** — LTX-2.5 ships both a `NADiffusionDecoder` video VAE and a conv variant; only the conv one tiles; `LTX2VideoDiffusionDecodePipeline` is the way to reach the diffusion decoder. | +| "70GiB at 1536x896x121, which no tile size reduces" and "a step that returns latents returns audio latents too, which nothing outside a pipeline call can vocode" | **UNSOURCED** but internally consistent and clearly measured locally. Upstream's answer to the same problem is different: keyframe-aware decode + `AUTO_TILING` (CHANGELOG 1.3.0, 2026-08-25), and installing the `natten` extra. **Missing knowledge.** | +| "Spend headroom on the two-stage flow rather than on base resolution: render at 768x448, upsample the latents 2x, and the result is sharper than a single pass at 1536x896" | **CONTRADICTED / incomplete** — as implemented there is no stage-2 refinement, so the sharpness claim is unsupported. With the 3-sigma tail it would be defensible. | +| Example links | All eight file paths here are **correct** (unlike the template README). | + +### `docs/ACCELERATION.md` / `docs/QUANTIZATION.md` + +- ACCELERATION's note that LTX-2's blocks return two hidden-state streams and need extra cache metadata beyond diffusers' defaults: **UNSOURCED** upstream (it's a diffusers-internals observation), not contradicted. +- QUANTIZATION's "the pattern the LTX-2 and MiniMax H3 [workflows use]": repo-internal. Fine. +- Neither doc mentions `torch.compile` / CUDA-graph capture, FP8 or NVFP4 for LTX-2.5, all of which upstream documents (CHANGELOG 1.2.0/1.3.0). **Missing knowledge**, CUDA-only. + +--- + +## Missing knowledge + +Recommendations the sources make that the repo does not carry, newest first. + +1. **The stage-2 refinement tail (dated: on the LTX-2.5-Diffusers card and `pipelines.md`; ComfyUI example workflow; diffusers docs page — all current as of 2026-09-07).** Renoise the upsampled latent with `noise_scale=STAGE_2_DISTILLED_SIGMA_VALUES[0]` and run 3 more steps at `sigmas=STAGE_2_DISTILLED_SIGMA_VALUES` through the same pipeline, carrying `audio_latents` across. This is *the* defining half of the distilled two-stage recipe and the repo has none of it. +2. **`DFRPipeline` — the current production-quality path (added 2026-08-11, CHANGELOG 1.2.0; refined 2026-08-25, 1.3.0).** `LTX2DFRPipeline` and `LTX2DFRTemporalRefinePipeline` are already present in the installed diffusers and unused. It is stage 1 at half res with generated keyframe slots, then a full-res detailing pass with the same IC-LoRA the repo already downloads. Its rules differ: **w/h divisible by 64** (128 with `--spatial-upscalings 2`), 4K is `3840x2176` not `3840x2160`, defaults 1024x1536 at 24 fps. +3. **Temporal upscaling for higher frame rates (2026-08-11, renamed `--temporal-upscalings` 2026-08-25).** 0/1/2 rounds → 121/241/481 frames at 24/48/96 fps for the same ~5 s. Requires `ltx-2.5-latent-temporal-upscaler-x2-bf16-1.0.safetensors`, which the repo never mentions. +4. **The fps trap (documented in the installed `utils.py`, `MAX_CONDITIONING_FPS` / `SNAP_CONDITIONING_FPS_ABOVE`, and in `pipelines.md`).** RoPE time is `pixel_frame / fps`; the transformer is trained around 24/25/30/60. 120 fps decodes as a motion spike per latent border then a stall; 48 stretches the other way. Condition at 60 and treat the frames at playback rate. The repo hardcodes 24 everywhere so it never hits this — but an agent composing from the repo's knowledge has no guardrail if a user asks for 60 fps. Also: some players read 96 fps H.264 as 24 and play in slow motion. +5. **Generated keyframe slots (2026-08-11).** Extra fully-denoised single-pixel-frame slots at interior positions, relaxing effective temporal compression where motion is too fast for the base grid. Available on `DistilledPipeline` stage 1. Costs ~+16% tokens for 5 slots at 512x768x241. This is the documented answer to fast-motion smearing, and the repo has no notion of it. +6. **Image-conditioning CRF (LTX-2.5 = 18, earlier = 33).** Auto-resolved from the text encoder's `model_type`, requires a PIL image and PyAV or `check_inputs` raises. Multi-frame video conditions are exempt. Undocumented in the repo despite three image-conditioned templates. +7. **Prompting: the actual training-caption spec.** The repo carries no prompting guidance at all — no `prompts/ltx2/README`, nothing in the templates' README, nothing in WORKFLOW_GUIDE for LTX. The full spec is sitting in the installed `utils.py` as `LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT` / `LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT` and could simply be pointed at. +8. **The I2V-specific prompting rule.** "Describe only changes from the image; don't reiterate established visual details. Inaccurate descriptions may cause scene cuts." The repo's I2V prompt does the opposite. +9. **IC-LoRA trade-off knob.** Fewer steps / lower guidance → closer to reference; more → more hallucination. And: the reference must be a clean low-res render, not compressed footage. Not for live-action fidelity work. +10. **Keyframe strength below 1.0** for smooth interpolation (`conditioning.md`'s "guiding latents"; cf. `ANCHOR_KEYFRAME_STRENGTH = 0.95`). +11. **Upstream memory tooling the repo doesn't discuss (2026-08-11 / 2026-08-25):** NVFP4 (`nvfp4-cast` / `nvfp4-prequant`, Blackwell + `ltx-kernels`), `fp8-cast`, `--offload cpu|disk`, CUDA-graph capture with `max_video_tokens`/`max_audio_tokens`, `AUTO_TILING` and keyframe-aware DiffVAE decode with the `natten` extra. Several are CUDA/Blackwell-only, but the repo's 24GB recipe is the natural place for at least fp8. +12. **HDR / EXR conditioning and output (2026-08-11)**, `HDRICLoraPipeline`, `--hdr`. `pipeline_ltx2_hdr_lora.py` is installed and unused. +13. **`RetakePipeline`** — regenerate a time region of an existing video; the documented "edit a clip" answer, distinct from the repo's extend/chain flows. +14. **Multishot generation** — the LTX-2.5 card names "native multishot… connected scenes in a single pass" as a headline capability, with its own prompting guidance. Nothing in the repo. + +--- + +## Assessment + +The repo's *mechanical* LTX-2.5 knowledge is unusually good. Everything that touches the distilled +schedule is right and right for the right reason: the 8-sigma constant referenced rather than copied, +`guidance_scale` 1.0 with STG and modality guidance zeroed, no `num_inference_steps`, `8k+1` frame +counts, resolutions on the 32-pixel grid, 24 fps, the dedicated Gemma-4 enhancer with its own config, +the IC-LoRA's filename / strength 1.0 / `reference_downscale_factor` 2 / ~280p draft. RECIPES_24GB's +distilled-vs-dev explanation is better than most of what is written about this model publicly. I would +trust all of that in a skill. + +Two things need correction before it teaches anything. + +**The two-stage flow is missing its second stage.** Every primary source describes it as 8 sigmas at +half res → 2x latent upsample → **renoise and 3 more sigmas at full res**. The repo upsamples and +stops, then calls the result the recommended quality flow and claims it beats a full-res single pass. +`STAGE_2_DISTILLED_SIGMA_VALUES` sits unused in the installed library. It also decodes to pixels +between the stages instead of passing latents. Separately, since 2026-08-11 the *recommended* quality +path is `DFRPipeline`, which is installed and unused. + +**The prompt library is the wrong genre.** LTX-2.5 was trained on 150–220-word single-paragraph +audio-visual captions carrying an explicit shot-type / camera-motion / viewpoint triple in prose and +a soundscape interleaved chronologically with the action. Five of six stored prompts are 35–55 words, +none states a shot type or camera motion, audio is appended rather than woven, and two end in +Stable-Diffusion tag style ("8k, ultra-detailed", "vibrant colors") that the spec explicitly forbids. +An agent taught to write LTX prompts from these examples would write bad ones. The correct spec is +already on disk in `utils.py`. + +Smaller: the template README links eight filenames that do not exist and points "LTX-2.5" at the +LTX-Video repo; prompts and summaries say "ltx-2" for a 2.5-only model. diff --git a/docs/proposals/audits/2026-09-07-minimax-h3-audit.md b/docs/proposals/audits/2026-09-07-minimax-h3-audit.md new file mode 100644 index 0000000..2625ec5 --- /dev/null +++ b/docs/proposals/audits/2026-09-07-minimax-h3-audit.md @@ -0,0 +1,339 @@ +# MiniMax-H3 knowledge verification — diffusers-workflow + +Research date: 2026-09-07. Repository branch `agent-legibility`, commit b2b9a8b. +No repository file was modified. + +## Headline + +MiniMax publishes two official prompt-writing guides and an official +`h3-prompt-writing` Skill. The repository's `dw/workflows/h3_context_ir.json` +system prompt is, to a very high degree, a **faithful and compressed +transcription of those two guides** — most sentences are near-verbatim. The +gaps are omissions rather than errors, plus one wholly invented layer +("continuity modes"). The `workflows/templates/minimax/README.md` numeric +conventions check out against the installed diffusers source. + +--- + +## Sources + +| URL | Type | Date | Contributes | +| --- | --- | --- | --- | +| https://huggingface.co/MiniMaxAI/MiniMax-H3/raw/main/README.md | PRIMARY | model card, current as fetched 2026-09-07 | Output specs (4–15 s, 24 fps, 32 kHz stereo, 768p default / 2K, 11 languages, aspect ratios), variant/input limits (≤9 images, ≤3 videos, ≤3 audio, ≤12 files), three-module system (Context-IR / Base / Regenerate-2K), CFG-distilled checkpoints, three worked Context-IR outputs (T2VA, I2VA, Ref2VA) as real reference prompts, links to the prompting guides and skills | +| https://huggingface.co/MiniMaxAI/MiniMax-H3/raw/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md | PRIMARY | shipped with the model card | **The** spec for T2VA/I2VA/FL2VA/L2VA: instruction-block sentences verbatim, three core fields, shot/cut rules, the full camera-motion table, speaker IDs, ``/voiceover/``/``, on-screen text, `overall_soundscape` / `non_diegetic_music` rules, four cases | +| https://huggingface.co/MiniMaxAI/MiniMax-H3/raw/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md | PRIMARY | shipped with the model card | **The** spec for Ref2VA: six sections and their order, the four label types, `summary` task-type table, `retention_analysis` marker tables, `detailed_description` rules incl. 350–500 words, speaker/audio-source rules, complete example | +| https://github.com/MiniMax-AI/MiniMax-H3 (`skills/h3-prompt-writing/`) | PRIMARY | last commit touching skills 2026-08-15 (`d21241f`); "Tips for better results" added 2026-08-11 (`a107547`) | Official agent-facing Skill wrapping the two guides; adds duration-matching and label-consistency tips; `references/base-en.txt` and `ref-en.txt` are the same guides | +| https://platform.minimax.io/docs/api-reference/video-generation-v2-h3-context-ir | PRIMARY | fetched 2026-09-07 | Hosted Context-IR endpoint: duration 4–15 (integer), ratio vocabulary, per-modality file limits; **no continuation/extension mode exists** | +| diffusers 0.41.0.dev0, `modular_pipelines/minimax_h3/` (installed in repo venv) | PRIMARY | as installed | `MINIMAX_H3_FPS = 24`, `min_duration = 5.0`, `max_duration = 15.0`, `align_num_frames` (`17n + 5`), canvas rules (multiple of 32, short edge 768, max 768×1344, aspect 1:4–4:1), `reference_image_short_edge = 2048`, reference caps | +| diffusers 0.41.0.dev0, `modular_pipelines/minimax_music3/` | PRIMARY | as installed | `audio_duration` = "Upper bound on the generated audio length… The language model may stop earlier. Capped at 9000 frames (six minutes)" | +| https://huggingface.co/docs/diffusers/main/en/api/pipelines/minimax_h3 | PRIMARY | main branch, fetched 2026-09-07 | "24 fps, 5 to 15 seconds", `17n+5`, 768 short edge, guidance-distilled (no `guidance_scale`, no `negative_prompt`), `num_inference_steps` counts sigma grid points incl. terminal 0, reference ordering is semantic, 960×544 endorsed as the speed lever | +| https://github.com/ModelTC/Minimax-H3-Turbo (README) | PRIMARY (LoRA author) | repo README as fetched 2026-09-07 | Turbo LoRA spec table: task, training resolution, video/audio shifts, distillation NFE, recommended NFE; reference-resize policies (`match` / `max` / `diffusers`) and the recommendation to use `match` | +| https://huggingface.co/api/models/lightx2v/Minimax-h3-Turbo | PRIMARY | `lastModified` 2026-09-04, created 2026-08-07 | Actual file list — includes files newer than the ModelTC README table | +| https://huggingface.co/MiniMaxAI/MiniMax-Music3/raw/main/README.md | PRIMARY | fetched 2026-09-07 | Music3 usage; only shows `audio_duration=60.0`, no ceiling semantics stated | +| https://docs.comfy.org/tutorials/video/minimax/minimax-h3 | SECONDARY (vendor-adjacent, cited by the LoRA authors) | fetched 2026-09-07 | Reference tagging "in the exact order it was connected"; explicit role assignment advice; 20 steps default / ~25 for motion / 8-step turbo; `ref_image_size` match-vs-max; multiframe `frame_idx` guidance | +| https://www.rundiffusion.com/minimax-h3-prompt-guide | SECONDARY | Aug 2026 | Restates the six sections, both marker sets, bracketed prefixes; adds a ten-item failure-mode list and a 7 000-character prompt cap | +| https://deapi.ai/blog/…, https://www.dreampixelforge.com/blog/minimax-h3-prompts, https://minimax-h3-ai.com/blog/minimax-h3-prompt-guide/, https://domoai.app/blog/minimax-h3-prompt-guide, https://leadde.ai/blog/mini-max-h3-prompt-guide | SECONDARY | Aug 2026 | All restate the model card and the two official guides (three-field / six-section split, `[Shot 1]` untimestamped, MM:SS.mmm). One adds an unsourced "350–450 words for complex, 150–250 for simple" figure that conflicts with the official 350–500. No independent information. | +| https://medium.com/…/minimax-h3-…, https://www.runpod.io/blog/minimax-h3-… | SECONDARY | Aug 2026 | Architecture restatement only | + +**Fetches attempted and not useful:** no arXiv/technical report for H3 exists as +of 2026-09-07 (searched; only unrelated omni-modal papers returned). The GitHub +mirror of the model repo has no `docs/` directory — the two guides live on the +Hugging Face repo only. + +--- + +## Claim-by-claim verdict — `dw/workflows/h3_context_ir.json` + +### Framing + +| Claim | Verdict | +| --- | --- | +| "H3-Base is trained to consume this exact format and degrades on anything else" | **PARTLY UNSOURCED.** The model card says H3-Context-IR "is critical to the quality of the final output, so we strongly recommend incorporating it into your generation pipeline or following the 'Prompting Guidance'". It never says the base model degrades on other input, and the base checkpoints accept free text. Directionally right, rhetorically stronger than any source. | +| Output only the rewritten prompt, no commentary | **UNSOURCED** (harness convention, not a model fact). | +| Structure = optional instruction block, one blank line, then the core fields | **CONFIRMED.** base guide §2.1: "The instruction must be the first line of the final prompt, followed by one blank line before the core fields." | + +### Instruction blocks + +| Claim | Verdict | +| --- | --- | +| T2VA: none, begin with the core fields | **CONFIRMED**, base §2.1. | +| I2VA: `For the target video, at 0.00 seconds into the target video, (from [Shot 1]) is fully referenced.` | **CONFIRMED verbatim**, base §2.1 and Case 2; also verbatim in the model card's own case-I2VA Context-IR output. | +| FL2VA: `How the reference pictures align with the target video - Picture 1 (from Shot 1) …; Picture 2 (from Shot N) aligns with the S.SS-second mark …` | **CONFIRMED**, base §2.1, with one cosmetic deviation: the guide uses an em dash (`—`) after "target video", the repo uses a hyphen. The guide's FL2VA form genuinely drops the angle brackets and square brackets that the L2VA form keeps; the repo reproduces that asymmetry correctly. | +| L2VA: `How the reference pictures align with the target video - (from [Shot N]) aligns with the S.SS-second mark of the target video.` | **CONFIRMED**, base §2.1 (same em-dash nit). | +| `S.SS` is the duration to two decimals; `N` is the index of the final shot | **CONFIRMED verbatim**, base §2.1. | +| Ref2VA has no instruction block | **CONFIRMED** — ref guide §1 lists six sections beginning at `subject_definitions`; the model card's case-Ref2VA output starts at `subject_definitions:`. | + +### Core fields (T2VA/I2VA/FL2VA/L2VA) + +| Claim | Verdict | +| --- | --- | +| Order `integrated_multimodal_description` → `overall_soundscape` → `non_diegetic_music`, blank-line separated | **CONFIRMED**, base §2.2. | +| Their three one-line definitions | **CONFIRMED near-verbatim**, base §2.2 and §4.1. | +| **Omission:** the `N/A` convention | **MISSING.** base §4.6: `N/A` for `overall_soundscape` only when the user asks for complete silence; §4.7: `N/A` when there is no non-diegetic music. Both official examples use it. The repo never mentions `N/A`, so a model following the repo prompt will invent a score for a silent brief. | +| **Omission:** length guidance (1–4 sentences soundscape, 1–3 music) and "do not repeat dialogue/singing/diegetic music in `overall_soundscape`" | **MISSING**, base §4.6/§4.7. | +| **Omission:** "do not use abstract mood words or explain the emotional function of the score" in `non_diegetic_music` | **MISSING**, base §4.7. | + +### Ref2VA six sections + +| Claim | Verdict | +| --- | --- | +| Six sections in order `subject_definitions, summary, retention_analysis, detailed_description, overall_soundscape, non_diegetic_music` | **CONFIRMED**, ref guide §1 table; matches the model card's case-Ref2VA output. | +| `` = reusable visible content (person, animal, object, scene, costume, style, action) | **CONFIRMED**, ref §2.1. | +| `` = concrete frame / composition anchor; an image that only defines a subject is cited inside that subject's line | **CONFIRMED verbatim**, ref §2.2 ("do not create a standalone picture entry"). | +| ``, "do not carry original dialogue over when only timbre is + referenced", "do not write `(Sx)` in `retention_analysis`", and the + `