Utilities

CLAUDE.md

@@ -27,7 +27,7 @@ python -m dw.test
 # Interactive REPL
 python -m dw.repl
 
-# Run all tests (134+ tests)
+# Run all tests (260+ tests)
 pytest -v
 
 # Run a single test file or test
@@ -52,7 +52,7 @@ JSON workflow → schema validation → variable substitution → sequential ste
 1. `workflow.py` loads JSON, validates against `workflow_schema.json`, substitutes `variable:name` references
 2. `step.py` executes each step — generating argument combinations via `previous_results.py` (cartesian product of `previous_result:step_name` references)
 3. Each step dispatches to one of: **Pipeline** (HuggingFace inference), **Task** (utility operation), or **Sub-Workflow** (recursive)
-4. `result.py` saves outputs as `{output_dir}/{workflow_id}-{step_name}.{index}.{ext}`
+4. `result.py` saves outputs as `{output_dir}/{workflow_id}-{step_name}.{index}.{ext}` — supports image, video, audio, text, and JSON content types. Optional `embed_metadata` stores generation parameters in PNG info chunks or JPEG/WebP EXIF.
 
 ### Key Modules
 
@@ -62,7 +62,12 @@ JSON workflow → schema validation → variable substitution → sequential ste
 | `dw/step.py` | Step executor: generates iterations, dispatches to pipeline/task/workflow |
 | `dw/pipeline_processors/pipeline.py` | Pipeline loading, components, quantization, LoRA, schedulers, offloading |
 | `dw/pipeline_processors/config_objects.py` | Quantization and group offload config creation |
-| `dw/tasks/task.py` | Task dispatcher (image processing, QR codes, gathering, video) |
+| `dw/tasks/task.py` | Task dispatcher (image processing, QR codes, gathering, video, segmentation, captioning, frame interpolation) |
+| `dw/tasks/segment.py` | GroundingDINO + SAM2 text-prompted object segmentation |
+| `dw/tasks/image_to_text.py` | Image captioning via transformers image-to-text pipeline (BLIP, BLIP-2, etc.) |
+| `dw/tasks/text_generation.py` | Text generation / prompt expansion via transformers text-generation pipeline |
+| `dw/tasks/interpolate_frames.py` | RIFE frame interpolation (2x/4x/8x) with vendored IFNet v4.6 |
+| `dw/tasks/rife_model.py` | Vendored RIFE IFNet v4.6 architecture (MIT License, Megvii Inc.) |
 | `dw/previous_results.py` | Cross-step data flow via cartesian products |
 | `dw/arguments.py` | Argument processing, resource loading, dynamic type conversion |
 | `dw/type_helpers.py` | Dynamic type loading: `"FluxPipeline"` → class, `"torch.bfloat16"` → dtype |

README.md

@@ -15,7 +15,8 @@ A declarative workflow engine for the [Hugging Face Diffusers library](https://g
 - **Prompt weighting** — A1111-style `(word:1.5)` syntax with long prompt support
 - **LoRA and IP-Adapter** support
 - **Composable workflows** from multiple JSON files with `builtin:` references
-- **Utility tasks** — background removal, upscaling, cropping, QR codes, LLM prompt augmentation
+- **Utility tasks** — upscaling, face restoration, segmentation, captioning, frame interpolation, QR codes, and more
+- **Metadata embedding** — store generation parameters in PNG/JPEG/WebP for reproducibility
 - **Interactive REPL** with persistent GPU model caching (2-4x faster iteration)
 - **Cross-platform** — CUDA, MPS (Apple Silicon), and CPU
 

TODO.md

@@ -0,0 +1,47 @@
+# diffusers-workflow TODO
+
+Utilities and enhancements from the stable diffusion ecosystem.
+
+## Post-Processing & Enhancement
+
+- [x] **CCSR / StableSR** — Diffusion-based upscalers with better detail than ESRGAN-family, especially for faces and textures. New task type alongside existing Spandrel upscaler. *(Implemented as `diffusion_upscale` task wrapping StableDiffusionUpscalePipeline x4/x2)*
+- [ ] **Real-ESRGAN Video** — Frame-consistent video upscaling with temporal smoothing. Current upscaler is image-only.
+- [x] **RIFE / FILM frame interpolation** — Generate intermediate frames for smoother video output. Video post-processing task.
+
+## Image Preprocessing & Conditioning
+
+- [x] **Marigold depth / DSINE normals** — Newer, more accurate depth/normal estimators than MiDaS/DPT. Better ControlNet conditioning maps.
+- [x] **GroundingDINO + SAM2** — Text-prompted object detection to segmentation. "Segment the dog" as a task, producing masks for inpainting workflows.
+- [x] **Florence-2** — Microsoft vision-language model for captioning, detection, segmentation. Powers an `auto_caption` task for img2img or IP-Adapter workflows.
+- [ ] **PuLID / InstantID** — Identity-preserving face conditioning (better than IP-Adapter for faces). Works with Flux and SDXL.
+
+## Video-Specific
+
+- [ ] **FramePack** — Context-aware video generation with efficient memory usage for long video generation.
+- [ ] **PySceneDetect keyframe extraction** — Smarter than `get_frame` for selecting keyframes from input video based on scene detection.
+- [x] **RIFE/IFRNet optical flow interpolation** — Optical flow-based frame interpolation as a post-processing step.
+
+## Workflow Utilities
+
+- [x] **Prompt expansion via local LLM** — Task that takes a short prompt and expands it using a small language model (Llama 3.2 1B, Qwen2.5, etc.).
+- [ ] **Image comparison / SSIM / LPIPS scoring** — Task that scores similarity between images for iterative refinement workflows.
+- [ ] **Color palette extraction / transfer** — Extract dominant colors from a reference image or apply color grading from one image to another.
+- [ ] **Tiled generation / outpainting helper** — Automate the border+mask+generation loop for progressive outpainting.
+
+## Model Management
+
+- [ ] **Automatic VRAM estimation** — Given a workflow JSON, estimate peak VRAM before running. Helps pick the right quantization/offload settings.
+- [ ] **Model predownload/warmup** — Dry-run mode that downloads all models without executing. Useful for deployment.
+
+## Quick Wins
+
+- [x] **Image metadata embedding** — Store generation params in PNG info chunks for reproducibility.
+- [x] **EXIF stripping** on input images — Privacy-safe preprocessing.
+- [ ] **Image hashing** (perceptual hash) — Dedup detection across workflow runs.
+- [x] **Watermark embedding/detection** — Responsible AI compliance.
+- [x] **Aspect ratio bucketing** — Auto-resize inputs to model-native aspect ratios.
+
+## Architectural Enhancements
+
+- [ ] **Conditional branching** — e.g., "if image has faces, run face restore; otherwise skip." Enables more sophisticated pipelines.
+- [ ] **Parallel step execution** — Steps with no data dependencies run concurrently. Matters for multi-GPU setups.