Skip to content

TASKS.md

Latest commit

 

History

History
1209 lines (988 loc) · 51.3 KB

TASKS.md

File metadata and controls

1209 lines (988 loc) · 51.3 KB
Raw

Task Commands

Tasks are utility operations that run outside of pipeline inference. Use them for image preprocessing, data gathering, and other non-model operations.

{
    "name": "step_name",
    "task": {
        "command": "command_name",
        "arguments": { ... }
    },
    "result": { "content_type": "image/jpeg" }
}

Any task that runs a model accepts a "device" argument to pin where it runs - useful for keeping a helper model (a captioner, an upscaler) off the accelerator a loaded pipeline is using, or on a second one.

Task argument schemas are discoverable: GET /api/tasks/{command} on the server returns each command's arguments read from its registered implementation's real signature, the web editor builds task forms from them, and workflow validation flags task-argument typos the same way it flags pipeline ones.

Image Processing

ControlNet Preprocessors

Generate control images for ControlNet pipelines:

Command Description
canny Canny edge detection
canny_cv OpenCV Canny (alternative)
depth Depth estimation (DPT)
midas Monocular depth (MiDaS)
zoe Zoe depth estimation
zoe_depth Zoe depth with colorization
leres Relative depth (LeReS)
normal_bae Surface normal estimation
openpose Pose estimation
dw_pose DW pose estimation
mlsd Line segment detection
lineart Line art extraction
lineart_standard Standard line art
hed HED edge detection
scribble Scribble-style edges
pidi Boundary detection
shuffle Content-preserving shuffle
teed TEED edge detection
anyline Anyline edge detection
sam Segment Anything
segmentation Semantic segmentation
depth_estimator Depth hint generation
depth_estimator_tensor Depth hint as tensor

All accept an image argument with processing parameters:

{
    "task": {
        "command": "canny",
        "arguments": {
            "image": {
                "location": "https://example.com/photo.jpg",
                "low_threshold": 50,
                "high_threshold": 200,
                "detect_resolution": 1024,
                "image_resolution": 1024
            }
        }
    }
}

Image Manipulation

Command Description Extra Arguments
remove_background Remove image background
resize_center_crop Resize with center crop width, height
resize_resample Resample to nearest 64px multiple
resize_rescale Resize to exact dimensions width, height
resize_bucket Snap to closest model-native aspect ratio resolution, ratios, alignment
crop_square Center crop to square
add_border_and_mask Add border with alpha mask
add_border_and_mask_with_size Border with specific dimensions width, height
strip_exif Remove all EXIF/metadata from image
add_watermark Add visible text watermark text, position, opacity, font_size, color, margin
get_image_size Return {width, height} dict

EXIF Stripping

Remove all EXIF metadata, GPS coordinates, camera info, and timestamps from images for privacy-safe preprocessing:

{
    "task": {
        "command": "strip_exif",
        "arguments": {
            "image": "previous_result:input_image"
        }
    },
    "result": { "content_type": "image/png" }
}

Returns a clean copy with pixel data only — no embedded metadata. Useful as a first step when processing user-uploaded images.

Watermark Embedding

Add a visible text watermark to images for responsible AI compliance:

{
    "task": {
        "command": "add_watermark",
        "arguments": {
            "image": "previous_result:generate",
            "text": "AI Generated",
            "position": "bottom-right",
            "opacity": 128
        }
    },
    "result": { "content_type": "image/png" }
}
Argument Required Description
text No Watermark text (default: "AI Generated")
position No "bottom-right", "bottom-left", "top-right", "top-left", or "center" (default: "bottom-right")
opacity No Text opacity 0-255 (default: 128)
font_size No Font size in pixels, 0 = auto-scale ~3% of image height (default: 0)
color No RGB array for text color (default: white)
margin No Pixel margin from edges (default: 10)

Aspect Ratio Bucketing

The resize_bucket command snaps an image to the closest model-native aspect ratio, then resizes with 64-pixel alignment. This avoids distortion and ensures the model generates at a resolution it was trained on.

{
    "task": {
        "command": "resize_bucket",
        "arguments": {
            "image": "previous_result:input_image",
            "resolution": 1024
        }
    },
    "result": { "content_type": "image/png" }
}
Argument Required Description
resolution No Target short-side size in pixels (default: 1024)
ratios No Custom list of [w, h] ratio pairs (default: standard SDXL/Flux ratios)
alignment No Round dimensions to this multiple (default: 64)

Default ratios: 1:1, 4:3, 3:4, 3:2, 2:3, 16:9, 9:16, 21:9, 9:21

For example, a 1600x900 photo (16:9) at resolution 1024 becomes 1792x1024. A 800x600 photo (4:3) becomes 1344x1024.

Video Processing

Command Description Extra Arguments
get_first_frame Extract first video frame
get_last_frame Extract last video frame
get_frame Extract frame at index frame_index

The frame commands accept videos in any shape a result carries them: PIL frame lists, numpy or torch frame arrays, and audio+video pairs (LTX-2, MiniMax H3). The extracted frame is always a PIL image.

concat_videos

Concatenate videos - and the audio generated with them - into one video. The standalone counterpart of a chained pipeline step's stitching (see "Chained video generation" in the workflow guide):

{
    "task": {
        "command": "concat_videos",
        "arguments": {
            "videos": ["previous_result:shot_1", "previous_result:shot_2"],
            "trim_frames": 1,
            "crossfade_ms": 75,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}
Argument Required Description
videos Yes The videos to join, in order - previous_result references, or the path or URL of a video file an earlier run wrote, which is read with the audio muxed into it
trim_frames No Frames dropped from the head of every video after the first (default: 0)
crossfade_ms No Equal-power crossfade at each audio seam (default: 75)
audio_bleed_ms No How long the outgoing video's tail rings on over the head of the next one, at seams with nothing trimmed to crossfade (default: 0, off)
seam_fade_ms No Fade on each side of a seam that gets neither a crossfade nor a bleed (default: 3, just enough not to click)
fps No Frame rate of the videos - required to join audio when trimming

A video may also be named by path or URL, which is how shots an earlier run already wrote are joined without regenerating them - the file is read with the audio muxed into it, and its track is fitted to the frames' own duration so the codec's block padding does not walk the sound off the picture over a dozen seams:

{
    "task": {
        "command": "concat_videos",
        "arguments": {
            "videos": [
                "/path/to/outputs/shot_01.mp4",
                "/path/to/outputs/shot_02.mp4",
                "previous_result:shot_03_rerendered"
            ],
            "trim_frames": 0,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Give each video its own entry. One previous_result reference naming a step that produced several videos does not hand them all over at once - it fans the step out over them, one concatenation per video, which is what makes the list form above the way to join a run's shots.

trim_frames and audio_bleed_ms address opposite situations. A chain carries its keyframe forward, so the trimmed head is material that covers the same stretch of time as the outgoing tail and the two can be crossfaded. A cut generates each shot independently, so there is nothing to fade with - and generated shots tend to open on near-silence and end mid-sound, leaving a butt-join that drops a running laugh track or a ringing room into a hole. audio_bleed_ms fills it the way an audience carries across a picture cut: a decaying copy of the outgoing tail is laid over the incoming head, added to whatever is already there, shortening neither side. Reach for a few hundred milliseconds - the MiniMaxH3SitcomShort example uses 700:

{
    "task": {
        "command": "concat_videos",
        "arguments": {
            "videos": ["previous_result:shot_1", "previous_result:shot_2"],
            "trim_frames": 0,
            "audio_bleed_ms": 700,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

A bleed works because it copies ambience, which has no pitch and no attacks to give the copy away. It is the wrong tool for anything tonal - a copied musical phrase or half-spoken word reads as a stutter whichever direction it runs. When a shot ends on something tonal, either give the cut a continuous bed with slice_audio + pair_audio, which leaves no seam to treat at all, or fade the seam gracefully with seam_fade_ms (a hundred or so milliseconds) and accept the cut. audio_bleed_ms wins where both are set and there is material to bleed.

dissolve_videos

Join videos with a cross-dissolve at every seam, and fade the whole piece in from and out to a colour. Where concat_videos cuts - right for shots that each carry their own sound - this melts one shot into the next, which is what a montage cut to a score wants:

{
    "task": {
        "command": "dissolve_videos",
        "arguments": {
            "videos": ["previous_result:shot_1", "previous_result:shot_2"],
            "dissolve_frames": 12,
            "fade_in_frames": 12,
            "fade_out_frames": 24,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}
Argument Required Description
videos Yes The videos to join, in order - previous_result references, or the path or URL of a video file an earlier run wrote, one entry per video as with concat_videos
dissolve_frames No Frames of overlap at each seam, blended linearly (default: 12). 0 is a hard cut
fade_in_frames No Frames over which the first video rises out of fade_color (default: 0)
fade_out_frames No Frames over which the last video sinks into it (default: 0)
fade_color No The RGB colour the fades come from and go to (default: black)
fps No Frame rate of the videos - required to crossfade audio at a dissolve

Every seam shortens the result by one overlap, so eight 124-frame shots joined with 12-frame dissolves run 908 frames, not 992 - size a soundtrack slice to the joined length, not the sum. When every input carries audio, the tracks are crossfaded over exactly the seam's span so they stay in step with the picture; when any input is silent the result is, and pair_audio puts a score under it.

Example: dissolve-between-shots.json

stabilize_video

Remove a generated clip's accumulated framing drift - the slow wander a video model adds over a shot that was meant to hold still:

{
    "task": {
        "command": "stabilize_video",
        "arguments": {
            "clip": "variable:shot_1",
            "smooth": 0
        }
    }
}
Argument Required Description
clip Yes The video - a frame list, a frame array or tensor, an audio+video pair, or the path or URL of a video file, read with its audio, so a shot an earlier run wrote can be steadied without regenerating it
smooth No 0 (the default) locks the framing to the first frame, which is what a shot generated from a pinned keyframe wants. A window in frames instead removes only the wander faster than that window, so a slow deliberate camera move survives and the drift around it does not

The argument is clip, not video, on purpose: the engine loads an argument named video itself, as bare frames, which would strip the soundtrack off before the task ever saw it. Frames are shifted back and the result is cropped to the region every frame covers, then resized to the original size; a soundtrack passes through untouched.

Example: dissolve-between-shots.json

video_frames

The frames of a generated video, as one (frames, height, width, channels) uint8 array. That is the shape an argument taking frames rather than a video wants - LTX-2's keyframe conditions, which are mapped from 0-255 - and it is one artifact where a list of frames would become one artifact per frame and multiply the step that consumed it:

{
    "name": "opening_frames",
    "task": {
        "command": "video_frames",
        "arguments": { "video": "previous_result:opening" }
    },
    "result": { "content_type": "video/mp4", "save": false, "fps": 24 }
}
Argument Required Description
video Yes The video - a frame list, a frame array or tensor, or an audio+video pair

An argument that goes through diffusers' video processor instead - LTX-2's IC-LoRA references - wants the [0, 1] frames the pipeline returned rather than this array; hand those over with previous_result:step.frames.

Example: extend-clip.json

pair_audio

Pair a video with an audio track, so the two are saved as one muxed file. A pipeline that generates its own soundtrack returns the pair together; anything working on the frames alone - a latent upsampler, an interpolator, an upscaler - returns frames without it, and this puts it back:

{
    "task": {
        "command": "pair_audio",
        "arguments": {
            "video": "previous_result:upscale",
            "audio": "previous_result:base"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}
Argument Required Description
video Yes The frames - a frame list, a frame array or tensor, or an audio+video pair whose own soundtrack is replaced
audio Yes The soundtrack - a waveform, the earlier step whose video carried one, or the path or URL of an audio or video file; the last two bring their sample rate along
sample_rate No Sample rate of the waveform. Required unless audio carries one; given here it wins

Example: assemble-and-score.json

slice_audio

Cut a slice out of an audio track, addressed in seconds or in video frames. Slices reaching past the end of the track are zero-padded. Either half of a pair may be left out - an omitted start begins at the head of the track, an omitted duration runs to the end of it - so a workflow that trims only when it is given a length still passes the whole track along:

{
    "task": {
        "command": "slice_audio",
        "arguments": {
            "audio": "./soundtrack.wav",
            "start_frame": 124,
            "num_frames": 124,
            "fps": 24
        }
    },
    "result": { "content_type": "audio/wav", "sample_rate": 44100 }
}
Argument Required Description
audio Yes Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
start_seconds / duration_seconds One pair The slice in seconds; either may be omitted
start_frame / num_frames / fps One pair The slice in video frames; fps is required, start and count may be omitted
sample_rate With a waveform Sample rate of a directly passed waveform (files carry their own)

crossfade_audio

Join audio tracks with an equal-power crossfade. Each seam overlaps the two tracks by the fade window:

{
    "task": {
        "command": "crossfade_audio",
        "arguments": {
            "audios": "previous_result:slices",
            "crossfade_ms": 75,
            "sample_rate": 44100
        }
    },
    "result": { "content_type": "audio/wav", "sample_rate": 44100 }
}

fade_audio

Fade a track in from silence and out to it. A slice cut out of the middle of a piece ends on whatever was sounding at the cut; a fade turns that into an ending. The curve is the equal-power cosine the seam joins use:

{
    "task": {
        "command": "fade_audio",
        "arguments": {
            "audio": "previous_result:soundtrack",
            "fade_in_ms": 500,
            "fade_out_ms": 2500,
            "sample_rate": 44100
        }
    }
}
Argument Required Description
audio Yes Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
fade_in_ms No Length of the fade in, from the head of the track (default: 0)
fade_out_ms No Length of the fade out, to the tail of the track (default: 0)
sample_rate With a waveform Sample rate of a directly passed waveform (files carry their own)

Example: audio-trim-fade.json — slice a generated track to length, then fade the cut into an ending.

normalize_audio

Scale a track so its loudest sample sits at a level. Generated music comes out wherever the model happened to land - a quiet take needs lifting before it sits under a picture, a hot one needs headroom before the encoder. Only the gain changes, so the dynamics survive:

{
    "task": {
        "command": "normalize_audio",
        "arguments": {
            "audio": "previous_result:faded",
            "peak_dbfs": -1.0,
            "sample_rate": 44100
        }
    }
}
Argument Required Description
audio Yes Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
peak_dbfs No The level the loudest sample is moved to, in dB below full scale (default: -1.0). 0 is full scale
sample_rate With a waveform Sample rate of a directly passed waveform (files carry their own)

A silent track is returned unchanged.

Example: dissolve-between-shots.json

mix_audio

Layer tracks on top of one another. crossfade_audio puts tracks one after another; this puts them on top of each other - a score laid under a film's own sound, where the music runs unbroken while the world underneath it is replaced at every cut:

{
    "task": {
        "command": "mix_audio",
        "arguments": {
            "audios": ["previous_result:soundtrack", "previous_result:world"],
            "gains": [0.5, 1.0],
            "sample_rate": 44100
        }
    }
}
Argument Required Description
audios Yes The tracks to layer - waveforms, audio or video file paths, or videos generated with a soundtrack
gains No One plain multiplier per track, in the same order - not decibels. Defaults to unity on every track
sample_rate With a raw waveform Sample rate of the waveforms. Required unless every track brings its own; given here it wins

Tracks of different lengths are padded with silence to the longest, so a score shorter than the picture leaves the tail dry rather than cutting the picture down to fit. Summing can push peaks past full scale and the sum is not rescaled - follow it with normalize_audio to bring the peak back down.

Example: dissolve-between-shots.json — a generated score mixed under the shots' own audio.

resample_audio

Convert a track to a different sample rate. A pipeline that conditions on audio wants it at its own rate (MiniMax H3 at its audio VAE's), and resampling a supplied recording once, up front, feeds it what it already wants:

{
    "task": {
        "command": "resample_audio",
        "arguments": {
            "audio": "previous_result:edit",
            "target_sample_rate": 44100
        }
    }
}
Argument Required Description
audio Yes Path or URL of an audio or video file, a video generated with a soundtrack (which brings its sample rate along), or a waveform
target_sample_rate Yes The rate to convert to
sample_rate With a waveform Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry

A track already at the target rate is returned untouched. The conversion is PyAV's, which dw already needs for video - no torchaudio dependency.

Example: assemble-and-score.json

Data Gathering

gather_images

Load images from URLs and/or file glob patterns:

{
    "task": {
        "command": "gather_images",
        "arguments": {
            "urls": ["https://example.com/a.jpg", "https://example.com/b.jpg"],
            "glob": "./images/*.jpg"
        }
    }
}

Returns a list of images that can be referenced by later steps with previous_result:.

gather_videos

Same as gather_images but for video files. Each video comes back as one artifact holding its frames and whatever audio was muxed alongside them, so a step referencing this one iterates over videos rather than over frames.

To join videos that are already on disk, give their paths to concat_videos directly rather than gathering them first: a previous_result reference to a gather step fans the consuming step out over the gathered videos instead of handing it all of them at once.

gather_inputs

Pass through arguments directly. Useful for organizing data flow.

Videos in image tasks

Every image command - the upscalers, face restoration, segmentation and the image processors - takes a video where it takes an image: an AudioVideo from a generation, concat_videos or dissolve_videos step, or a frame array from video_frames. The command runs over the frames one at a time and returns one video artifact, its soundtrack carried through untouched, so a generated clip can be upscaled without losing what was generated alongside it:

{
    "task": {
        "command": "upscale",
        "arguments": {
            "image": "previous_result:generate_video",
            "model_name": "Kim2091/UltraSharp"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Captioning (image_to_text) is the exception - describe a frame, taken with get_first_frame, rather than a video.

Image Upscaling

Upscale images using spandrel-compatible super-resolution models (ESRGAN, SwinIR, HAT, DAT, and 40+ other architectures). Models are auto-detected from weight files.

{
    "task": {
        "command": "upscale",
        "arguments": {
            "image": "previous_result:generate",
            "model_name": "Kim2091/UltraSharp",
            "filename": "4x-UltraSharp.pth"
        }
    }
}
Argument Required Description
image Yes PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
model_name Yes HuggingFace repo ID or local file path
filename No Specific weight file in a HF repo (auto-detected if only one)
tile_size No Tile size for large images (default: 512)
tile_overlap No Overlap between tiles in pixels (default: 32)

Large images are automatically tiled to avoid GPU memory issues. Models can be loaded from HuggingFace Hub repos or local .pth/.safetensors files.

Examples:

Diffusion Upscaling

Upscale images using Stable Diffusion upscale pipelines. Text-guided upscaling with better detail recovery than traditional super-resolution, especially for faces and textures.

Two modes are available:

  • x4 (default): StableDiffusionUpscalePipeline — 4x upscale via stabilityai/stable-diffusion-x4-upscaler
  • x2: StableDiffusionLatentUpscalePipeline — 2x upscale via stabilityai/sd-x2-latent-upscaler
{
    "task": {
        "command": "diffusion_upscale",
        "arguments": {
            "image": "previous_result:generate",
            "prompt": "high quality, detailed",
            "negative_prompt": "blurry, low quality, artifacts",
            "mode": "x4"
        }
    }
}
Argument Required Description
image Yes PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
prompt No Text guidance for upscaling (default: "")
negative_prompt No Negative text guidance (default: none)
mode No "x4" or "x2" (default: "x4")
model_name No Override the default model for the selected mode
num_inference_steps No Denoising steps (default: 25)
guidance_scale No Classifier-free guidance scale (default: 9.0)
noise_level No Noise level for x4 mode (default: 20, ignored for x2)

Examples:

  • upscale-diffusion.json — Upscale any existing image. mode selects which: x4 (the default) reaches 2048px, x2 reaches 1024px through the latent upscaler.
  • upscale-diffusion.json — Prompt-guided upscale of an image you already have, with no generation step.

Face Restoration

Restore and enhance faces in images using spandrel-compatible face restoration models (GFPGAN, CodeFormer, RestoreFormer). Uses facexlib for face detection and alignment, then runs each detected face through the restoration model.

{
    "task": {
        "command": "restore_faces",
        "arguments": {
            "image": "previous_result:generate",
            "model_name": "leonelhs/gfpgan",
            "filename": "GFPGANv1.4.pth"
        }
    }
}
Argument Required Description
image Yes PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
model_name Yes HuggingFace repo ID or local file path
filename No Specific weight file in a HF repo (auto-detected if only one)
upscale_factor No Background upscale factor (default: 1, no upscaling)
face_size No Cropped face size in pixels (default: 512)
use_parse No Use face parsing for better blending (default: true)
only_center_face No Only restore the largest/center face (default: false)
detection_resize No Resize shorter side for detection speed (default: 640)
eye_dist_threshold No Skip faces with eye distance below this (default: 5)
upsample_img No Pre-upscaled background image (e.g., from a prior upscale step)

Models are loaded via spandrel, so any .pth/.safetensors face restoration weights work. CodeFormer requires pip install spandrel-extra-arches (non-commercial license).

Example: restore-faces.json — Generate a portrait, then restore faces with GFPGAN v1.4.

Combining with Upscaling

You can chain upscaling and face restoration. Generate first, upscale the background, then paste restored faces onto the upscaled image:

{
    "steps": [
        {
            "name": "generate",
            "pipeline": { "..." : "..." },
            "result": { "content_type": "image/jpeg" }
        },
        {
            "name": "upscale",
            "task": {
                "command": "upscale",
                "arguments": {
                    "image": "previous_result:generate",
                    "model_name": "Kim2091/UltraSharp",
                    "filename": "4x-UltraSharp.pth"
                }
            },
            "result": { "content_type": "image/jpeg" }
        },
        {
            "name": "restore",
            "task": {
                "command": "restore_faces",
                "arguments": {
                    "image": "previous_result:generate",
                    "model_name": "leonelhs/gfpgan",
                    "filename": "GFPGANv1.4.pth",
                    "upscale_factor": 4,
                    "upsample_img": "previous_result:upscale"
                }
            },
            "result": { "content_type": "image/jpeg" }
        }
    ]
}

This gives the best results: the super-resolution model handles background detail while the face model handles facial features, composited together at the upscaled resolution.

Object Segmentation

Detect and segment objects using text prompts via GroundingDINO + SAM2. Returns a binary mask image suitable for inpainting workflows.

{
    "task": {
        "command": "segment",
        "arguments": {
            "image": "previous_result:input_image",
            "prompt": "dog"
        }
    },
    "result": { "content_type": "image/png" }
}
Argument Required Description
image Yes PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
prompt Yes Text description of object(s) to detect (e.g., "dog", "red car")
model_name No GroundingDINO model ID (default: IDEA-Research/grounding-dino-base)
sam_model_name No SAM2 model ID (default: facebook/sam2-hiera-large)
threshold No Detection confidence threshold (default: 0.3)
invert No Invert the output mask (default: false)

Returns a grayscale PIL Image (mode "L") — white (255) for detected objects, black (0) for background. Use with inpainting pipelines like FluxFillPipeline.

Examples:

Image Captioning

Generate text captions from images using a vision-language model.

Transformers 5 removed the dedicated image-to-text pipeline this task used to build, along with the BLIP/ViT-GPT2/GIT captioning models that ran on it. Captioning now goes through the same image-text-to-text pipeline as any other VLM, so model_name needs a vision-language model (SmolVLM, Qwen2.5-VL, LLaVA, etc.) and prompt is a question put to the model rather than a text fragment to continue.

{
    "task": {
        "command": "image_to_text",
        "arguments": {
            "image": "previous_result:input_image"
        }
    },
    "result": { "content_type": "text/plain" }
}
Argument Required Description
image Yes PIL Image, URL/path, or previous_result: reference
model_name No HuggingFace vision-language model ID (default: HuggingFaceTB/SmolVLM-256M-Instruct)
prompt No What to ask about the image (default: Describe this image.) — ask a narrower question for a narrower caption
system_prompt No System instruction for the model
max_new_tokens No Maximum tokens to generate (default: 50)

The default model is deliberately tiny, matching the footprint of the old captioning default; it produces short, plain captions. Point model_name at something larger for detail.

Returns a caption string. Save as text/plain for .txt output, or pass to a downstream step via previous_result: as a prompt for image generation.

For a detailed caption, hand the image to text_generation with a question and a larger vision-language model; that is what describe-and-regenerate.json does ahead of its prompt expansion.

Examples:

Extracting Sections

Reduce generated text to a known set of labelled sections, dropping anything else:

{
    "task": {
        "command": "extract_sections",
        "arguments": {
            "text": "previous_result:expand",
            "sections": ["integrated_multimodal_description", "overall_soundscape", "non_diegetic_music"]
        }
    },
    "result": { "content_type": "text/plain" }
}
Argument Required Description
text Yes The generated text, usually a previous_result: reference
sections Yes Section labels to keep, in the order they should appear
keep_preamble No Keep any text before the first label (default: true)

A section runs from its label: to the end of that paragraph, so a blank line ends one and a single newline does not — a field holding one line per item stays intact. Repeats are dropped, missing sections are skipped, and text with no recognised label is returned unchanged.

This exists because a model asked for a rigid format usually produces it and then keeps going — restating the description, appending a summary, or looping until it runs out of tokens. Prompting against that is unreliable, and at small model sizes adding rules to an already long specification can make adherence worse. Trailing text is not free either: a prompt is conditioning, and a pipeline that does not truncate spends memory and attention on whatever arrives. Keeping the fields that were asked for is deterministic where prompting is not.

The built-in h3_context_ir workflow applies this to its own output, so a workflow delegating to it receives only the fields MiniMax H3 expects.

Text Generation / Prompt Expansion

Generate or expand text using a local language model. Useful for expanding short prompts into detailed image generation prompts, rewriting text, or other text-to-text tasks.

{
    "task": {
        "command": "text_generation",
        "arguments": {
            "prompt": "a cat on a windowsill",
            "system_prompt": "You are a helpful AI assistant that creates detailed prompts for text to image generative AI. When supplied input generate only the prompt, no other text."
        }
    },
    "result": { "content_type": "text/plain" }
}
Argument Required Description
prompt Yes The user message or short prompt to expand/transform
system_prompt No System instruction for the model (e.g., "expand this into a detailed image prompt")
model_name No HuggingFace model ID (default: Qwen/Qwen2.5-1.5B-Instruct, or HuggingFaceTB/SmolVLM-256M-Instruct when an image is supplied)
image No PIL Image, URL/path, or previous_result: reference — see below
repetition_penalty No Vision path only (default: 1.15) — see below
generate_kwargs No Anything else to pass to the model's generate() — no_repeat_ngram_size, top_p, min_new_tokens. Merged last, so it overrides the settings above
max_new_tokens No Maximum tokens to generate (default: 500)

Writing a prompt from a picture

Supplying image switches the task to a vision-language model, so the generated text describes what is actually in the picture instead of what the prompt guesses is there. model_name must then name a VLM — a text-only model cannot be loaded as one.

{
    "task": {
        "command": "text_generation",
        "arguments": {
            "prompt": "Write a video prompt that starts from this picture.",
            "image": "previous_result:input_image",
            "model_name": "Qwen/Qwen3-VL-4B-Instruct"
        }
    },
    "result": { "content_type": "text/plain" }
}

This matters most ahead of an image-conditioned generation step. Those pipelines pin the supplied picture as the first frame, so a prompt written without seeing it will describe a scene the keyframe contradicts and the two conditionings pull against each other. Pass the same image to both and the prompt agrees with the frame it opens on.

A vision model is large enough to be worth releasing before the generation model loads — see release_models in the workflow guide.

Generation stays greedy so a workflow reproduces, but greedy decoding against a long, rigid format specification makes these models loop — emitting a complete answer and then repeating its closing sections until the token budget runs out. The vision path applies a repetition_penalty of 1.15 to stop that. Measured on Qwen3-VL against the MiniMax H3 prompt spec, 1.05 still looped through the whole budget while 1.15 ended on its own at a length matching the format's own guidance. Raise it if a model still repeats itself, or set 1.0 to disable.

A penalty reins the looping in but does not guarantee the model stops where the format ends; for that, trim the output with extract_sections below.

There is a limit to what a small model will follow. Against the MiniMax H3 spec, neither Qwen3-VL-4B nor 8B produces the <d>[Language]...</d> dialogue tag or the (S1) speaker ids, whether the idea implies speech or supplies the line verbatim; the 8B is worse on layout, capitalising its section labels. Showing a complete worked example does produce them - by copying the example word for word, which is useless - and a placeholder skeleton does not produce them at all. The visual description these models write is grounded and usable; the dialogue markup is not. Write prompts by hand where a subject has to speak.

For anything the arguments above do not cover, generate_kwargs goes straight to generate():

"arguments": {
    "prompt": "a cat on a windowsill",
    "generate_kwargs": { "no_repeat_ngram_size": 25 }
}

It is merged after everything else, so it can override repetition_penalty and the sampling settings as well as add to them.

Examples:

Speech Generation

Speak a line of text with a local text-to-speech model. The result is a waveform carrying the rate its model generated at, so it composes with slice_audio, fade_audio and pair_audio directly (concat_videos and dissolve_videos join videos — pair the track onto a video first).

{
    "task": {
        "command": "generate_speech",
        "arguments": {
            "text": "The way ahead is longer still.",
            "voice_preset": "v2/en_speaker_6"
        }
    },
    "result": { "content_type": "audio/wav" }
}
Argument Required Description
text Yes The line to speak
model_name No HuggingFace model ID (default: suno/bark-small)
voice_preset No The speaker, for a model with presets — v2/en_speaker_0 through v2/en_speaker_9 for Bark. A model with no processor (a single-voice model such as facebook/mms-tts-eng) refuses a voice_preset with an error rather than ignoring it
forward_params No Passed to the model's forward/generate call
generate_kwargs No Ad-hoc generation settings for a generative model — temperature, do_sample

The default is Bark because its voice presets give distinct speakers, which is what two characters in a scene need; facebook/mms-tts-eng is a quarter the size and a good override where one voice will do. voice_preset is a preprocessing argument — it selects the speaker before generation rather than parameterizing it — so naming it here is what makes it reach the processor. Passed through forward_params it would be dropped and every character would sound the same.

The result needs no sample_rate. A generated track carries the rate its model produced it at, and that beats the 44100 default; declaring one still wins over both, for a track whose rate was reported wrong. Every TTS model runs at a different rate, so a declared rate that does not match plays the speech at the wrong speed and pitch without ever failing.

Generating a voice to condition on

The role this earns its place in is voice timbre reference, not the track a mouth follows. MiniMax H3 lip-syncs well when it generates the speech itself and poorly when it must follow supplied audio, so its MiniMaxH3AudioReference takes a few seconds of a voice to fix timbre, pitch and delivery while H3 still generates the line. Build the reference with from_previous_result and the clip's own sample rate comes across with it:

"references": [
    {
        "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3AudioReference",
        "from_previous_result": "voice"
    }
]

Referencing the same preset in every shot of a scene makes a character's voice a conditioning signal rather than a prose description that has to land identically a dozen times. The other honest uses are a voice that must be matched — a specific delivery H3 will not produce from description alone — and narration over shots where nothing has to lip-sync to it, muxed with pair_audio.

A speech model is worth releasing before a video model loads — set release_models on the step, as in the example below.

Examples:

Frame Interpolation

Increase video frame rate using RIFE (Real-Time Intermediate Flow Estimation). Takes a video and inserts intermediate frames between each pair. The result is one video artifact without a soundtrack - the frame count changed, so pair_audio is how the original track comes back. interpolate-frames.json shows the interpolation itself.

{
    "task": {
        "command": "interpolate_frames",
        "arguments": {
            "video": "previous_result:generate_video",
            "multiplier": 2
        }
    },
    "result": { "content_type": "video/mp4", "fps": 60 }
}
Argument Required Description
video Yes The frames - a frame list, a frame array, or an audio+video pair from a concat or dissolve step (its audio is dropped) - usually a previous_result: reference
multiplier No Frame count multiplier: 2, 4, or 8 (default: 2)
model_name No HuggingFace repo with RIFE v4.13 weights (default: imaginairy/rife-interpolation)
filename No Weights filename within the repo (default: rife-flownet-4.13.2.safetensors)

Uses vendored IFNet v4.13 architecture. Weights are downloaded from HuggingFace Hub on first use.

Example: interpolate-frames.json — Generate video with Mochi, then 2x interpolate from 30fps to 60fps.

Metadata Embedding

Embed generation parameters in saved images. Enable by setting embed_metadata: true in a step's result configuration:

{
    "result": {
        "content_type": "image/png",
        "embed_metadata": true
    }
}
Format Storage Notes
PNG Text chunk (parameters key) Always available
JPEG/WebP EXIF UserComment Requires pip install piexif

Metadata includes step name, model name, and generation arguments (prompt, steps, guidance scale, etc.) as JSON.

Example: embed-metadata.json — Generate with Flux and embed parameters in PNG.

QR Code Generation

{
    "task": {
        "command": "qr_code",
        "arguments": {
            "qr_code_contents": "https://example.com"
        }
    }
}
Argument Required Description
qr_code_contents Yes Data to encode (URL, text, etc.)
height No Used with width to derive output resolution (default: 768)
width No Used with height to derive output resolution (default: 768)

The QR code is generated then resampled to max(height, width), aligned to the nearest 64px multiple.

Example: qr-code.json — QR code with artistic ControlNet

Chat/Dict Plumbing

These small tasks glue together multi-step pipelines that mix raw transformers components with task steps — for the cases text_generation does not cover.

format_chat_message

Build a text_inputs chat message list from a system and user message, in the shape a transformers.pipeline text-generation call expects:

{
    "task": {
        "command": "format_chat_message",
        "arguments": {
            "system_prompt": "You are a helpful assistant.",
            "user_message": "variable:prompt"
        }
    }
}
Argument Required Description
system_prompt Yes System instruction
user_message Yes User message content

Returns {"text_inputs": [{"role": "system", ...}, {"role": "user", ...}]}. Pass the result to a transformers.pipeline step's text_inputs argument via previous_result:.

get_dict_value

Extract a single value from a dictionary result (e.g., a transformers pipeline's output) for use in a later step:

{
    "task": {
        "command": "get_dict_value",
        "arguments": {
            "dict": "previous_result:augment_prompt",
            "key": "generated_text"
        }
    }
}
Argument Required Description
dict Yes Dictionary (or previous_result: reference) to read from
key Yes Key to extract

Returns the value at key, or None if the key is absent.

batch_decode_post_process

Decode generated token IDs and run model-specific post-processing (e.g., Florence-2's task-token parsing), using the processor from an earlier pipeline step:

{
    "task": {
        "command": "batch_decode_post_process",
        "pipeline_reference": "describe_image_processor",
        "arguments": {
            "generated_ids": "previous_result:describe_image_model.generated_ids",
            "task": "<DETAILED_CAPTION>"
        }
    }
}
Argument Required Description
pipeline_reference Yes Name of an earlier pipeline step whose processor to reuse (sibling of command/arguments, not inside arguments)
generated_ids Yes Token IDs to decode (e.g., a model step's generated_ids output)
task Yes Task token to post-process for (e.g., <DETAILED_CAPTION>)

Calls processor.batch_decode(...) then processor.post_process_generation(..., task=task) and returns parsed_answer[task].

Multi-Step Example

Canny edge detection followed by ControlNet generation:

{
    "steps": [
        {
            "name": "edges",
            "task": {
                "command": "canny",
                "arguments": {
                    "image": {
                        "location": "photo.jpg",
                        "low_threshold": 50,
                        "high_threshold": 200
                    }
                }
            },
            "result": { "content_type": "image/jpeg" }
        },
        {
            "name": "generate",
            "pipeline": {
                "configuration": {
                    "component_type": "FluxControlPipeline",
                    "offload": "sequential"
                },
                "from_pretrained_arguments": {
                    "model_name": "black-forest-labs/FLUX.1-Canny-dev",
                    "torch_dtype": "torch.bfloat16"
                },
                "arguments": {
                    "control_image": "previous_result:edges",
                    "prompt": "a watercolor painting",
                    "num_inference_steps": 50
                }
            },
            "result": { "content_type": "image/jpeg" }
        }
    ]
}

Examples