Utilities by dkackman · Pull Request #25 · dkackman/diffusers-workflow

dkackman · 2026-03-26T18:54:36Z

Summary

Marigold depth/normals — example workflows for MarigoldDepthPipeline and MarigoldNormalsPipeline (no code changes, native diffusers pipelines)
Segment task — new segment command using GroundingDINO + SAM2 for text-prompted object segmentation, returns binary mask for inpainting workflows
Frame interpolation task — new interpolate_frames command using vendored RIFE IFNet v4.6 (MIT, Megvii Inc.) for 2x/4x/8x video frame rate increase
Image metadata embedding — opt-in "embed_metadata": true in result config embeds generation parameters in PNG info chunks or EXIF

Covers Marigold depth/normals examples, GroundingDINO+SAM2 segmentation task, RIFE frame interpolation task, and image metadata embedding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

transformers is a hard dependency, no lazy import needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

13 tasks covering Marigold examples, segment task, frame interpolation task, and image metadata embedding with TDD approach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Example JSON workflows using MarigoldDepthPipeline and MarigoldNormalsPipeline for depth map and surface normal estimation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…masking Implements a new "segment" task command that takes an image and text prompt, detects matching objects with GroundingDINO, then generates precise binary masks using SAM2. Supports configurable thresholds and mask inversion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Segment: gather image + text-prompted segmentation mask. SegmentAndInpaint: segment object then inpaint with FluxFill. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Adds a new task command that takes video frames and returns interpolated frames at 2x, 4x, or 8x frame count using iterative 2x passes. The RIFE model loading is scaffolded (tests mock it) while the frame list logic is fully functional. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Video generation with Mochi followed by 2x frame interpolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When embed_metadata is true in a step's result config, generation parameters are embedded as PNG info chunks (or EXIF for JPEG/WebP via optional piexif). Step collects pipeline/task metadata and passes it to Result for serialization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Demonstrates embed_metadata: true for PNG generation parameter embedding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Mark Marigold depth, GroundingDINO+SAM2, RIFE interpolation, and image metadata embedding as done. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Adds a set of “ecosystem utilities” to the diffusers-workflow engine: new task commands for text-prompted segmentation and video frame interpolation, plus opt-in embedding of generation metadata in saved images, alongside several example workflows and accompanying docs/tests.

Changes:

Add segment (GroundingDINO + SAM2) and interpolate_frames (RIFE IFNet v4.6) task commands and register them in the task registry.
Add opt-in embed_metadata result configuration, collecting metadata in Step and embedding it on image save (PNG info chunk / EXIF).
Add example workflows and tests covering the new tasks and metadata embedding.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`tests/test_segment.py`	Adds unit tests for `segment_image` and command registration.
`tests/test_result.py`	Adds tests for opt-in PNG metadata embedding behavior.
`tests/test_interpolate_frames.py`	Adds unit tests for interpolation frame-count logic and command registration.
`examples/SegmentAndInpaint.json`	Example workflow chaining `segment` into inpainting.
`examples/Segment.json`	Minimal example workflow for segmentation.
`examples/MetadataEmbed.json`	Example workflow demonstrating `embed_metadata: true`.
`examples/MarigoldNormals.json`	Example workflow for `MarigoldNormalsPipeline`.
`examples/MarigoldDepth.json`	Example workflow for `MarigoldDepthPipeline`.
`examples/InterpolateFrames.json`	Example workflow generating a video then interpolating frames.
`dw/workflow_schema.json`	Extends result schema with `embed_metadata` boolean flag.
`dw/tasks/task.py`	Registers new `segment` and `interpolate_frames` task commands.
`dw/tasks/segment.py`	Implements GroundingDINO + SAM2 segmentation returning a mask image.
`dw/tasks/rife_model.py`	Vendors the RIFE IFNet v4.6 network definition.
`dw/tasks/interpolate_frames.py`	Implements frame interpolation logic, model loading, and weights download.
`dw/step.py`	Collects metadata for embedding when `embed_metadata` is enabled.
`dw/result.py`	Adds metadata storage and embeds metadata during image saves.
`docs/superpowers/specs/2026-03-26-ecosystem-utilities-design.md`	Design spec for the new utilities.
`docs/superpowers/plans/2026-03-26-ecosystem-utilities.md`	Detailed implementation plan for the utilities.
`TODO.md`	Updates TODO checklist to reflect completed utilities.

Comments suppressed due to low confidence (1)

dw/step.py:56

embed_metadata is handled once before get_iterations(), and _collect_metadata() reads from the step definition template (which can still contain previous_result: references). If a step runs multiple iterations with different resolved arguments, every saved artifact will embed the same (possibly unresolved) metadata, which is incorrect for reproducibility. Consider storing metadata per-iteration/per-artifact (e.g., parallel list aligned to results/artifacts, or wrapping each iteration result with its own metadata) and using that during Result.save().

            # Collect metadata for embedding if enabled
            result_def = self.step_definition.get("result", {})
            if result_def.get("embed_metadata", False):
                metadata = self._collect_metadata()
                result.set_metadata(metadata)

            # Log what type of action we're executing (Pipeline/Task/Workflow)
            action_type = type(step_action).__name__
            logger.info(f"Running {action_type} {step_name}:{step_action.name}...")

            # Get all possible argument combinations for this step
            # This expands any references to previous results into concrete values
            iterations = get_iterations(step_action.argument_template, previous_results)
            logger.debug(f"Generated {len(iterations)} argument combinations")

Copilot · 2026-03-26T19:00:19Z

+            # Collect metadata for embedding if enabled
+            result_def = self.step_definition.get("result", {})
+            if result_def.get("embed_metadata", False):
+                metadata = self._collect_metadata()
+                result.set_metadata(metadata)
+


There are existing unit tests for dw.step.Step, but the new embed_metadata behavior isn’t covered. Adding a test that enables embed_metadata and asserts the resulting Result carries the expected metadata (and behaves correctly with multiple iterations) would prevent regressions.

@copilot apply changes based on this feedback

Added 6 tests in commit a6fda74 covering the embed_metadata behavior:

embed_metadata absent or False → Result.metadata stays None

Pipeline step with embed_metadata: True → metadata includes step_name, model_name, and arguments

Task step with embed_metadata: True → metadata includes step_name, task_command, and arguments

Pipeline step missing model_name → no model_name key in metadata

Multiple iterations → metadata is set once on the single Result object and all iterations run correctly

Copilot · 2026-03-26T19:00:19Z

+"""
+Video frame interpolation via RIFE (Real-Time Intermediate Flow Estimation).
+
+Takes a list of video frames and generates intermediate frames to increase
+frame rate. Supports 2x, 4x, and 8x multipliers.
+
+Model weights are downloaded from HuggingFace Hub on first use.
+"""


The module docstring says the RIFE weights are downloaded from the HuggingFace Hub, but _download_weights() downloads from a GitHub release when model_name is None. Please update the docstring (or change the default download source) so the documented behavior matches the implementation.

@copilot apply changes based on this feedback

Copilot · 2026-03-26T19:00:20Z

+        h, w = t1.shape[2], t1.shape[3]
+        ph = ((h - 1) // 32 + 1) * 32
+        pw = ((w - 1) // 32 + 1) * 32
+        padding = (0, pw - w, 0, ph - h)
+        t1_padded = torch.nn.functional.pad(t1, padding)
+        t2_padded = torch.nn.functional.pad(t2, padding)
+
+        # Precompute warp grid and flow divisors for the padded resolution
+        tenFlow_div = torch.tensor([(pw - 1.0) / 2.0, (ph - 1.0) / 2.0], device=device)
+        backwarp_tenGrid = torch.cat(
+            [
+                torch.linspace(-1.0, 1.0, pw, device=device)
+                .view(1, 1, 1, pw)
+                .expand(-1, -1, ph, -1),
+                torch.linspace(-1.0, 1.0, ph, device=device)
+                .view(1, 1, ph, 1)
+                .expand(-1, -1, -1, pw),
+            ],
+            1,
+        )
+
+        timestep = torch.full((1, 1, ph, pw), 0.5, dtype=torch.float32, device=device)
+        scale_list = [8, 4, 2, 1]


In the returned inference() callable, tenFlow_div, backwarp_tenGrid, and timestep are recomputed on every frame-pair inference even when the padded resolution doesn’t change. For long videos this adds significant overhead. Consider caching these tensors per (device, ph, pw) within the closure (or via an LRU cache) so repeated calls reuse the precomputed grid/divisors.

Copilot · 2026-03-26T19:00:20Z

+            try:
+                import piexif
+
+                exif_dict = {"0th": {}, "Exif": {}, "GPS": {}, "1st": {}}
+                if hasattr(image, "info") and "exif" in image.info:
+                    exif_dict = piexif.load(image.info["exif"])
+                exif_dict["Exif"][piexif.ExifIFD.UserComment] = (
+                    piexif.helper.UserComment.dump(metadata_json)
+                )


_save_image_with_metadata uses piexif.helper.UserComment.dump(...) but only does import piexif. piexif.helper is a separate submodule and won’t be available unless explicitly imported, so this will raise AttributeError at runtime and fail saving JPEG/WebP when metadata embedding is enabled. Import piexif.helper (or from piexif.helper import UserComment) and consider catching broader exceptions from EXIF writing so a metadata failure can fall back to a normal save.

@copilot apply changes based on this feedback

Copilot · 2026-03-26T19:00:20Z

+    def _collect_metadata(self):
+        """Collect step metadata for embedding in saved images."""
+        metadata = {"step_name": self.name}
+
+        if "pipeline" in self.step_definition:
+            pipeline_def = self.step_definition["pipeline"]
+            pretrained_args = pipeline_def.get("from_pretrained_arguments", {})
+            if "model_name" in pretrained_args:
+                metadata["model_name"] = pretrained_args["model_name"]
+            metadata["arguments"] = dict(pipeline_def.get("arguments", {}))


Metadata collection doesn’t include the workflow ID, but the design docs and tests treat workflow_id as part of the embedded metadata. Consider passing workflow_id into Step (or into _collect_metadata) from Workflow.run() and adding it to the metadata dict so embedded metadata is uniquely attributable to a workflow run.

@copilot apply changes based on this feedback

Agent-Logs-Url: https://github.com/dkackman/diffusers-workflow/sessions/7078d921-2f46-4c61-a0da-369dac1ab9ba Co-authored-by: dkackman <5160233+dkackman@users.noreply.github.com>

dkackman and others added 12 commits March 26, 2026 10:51

Add design spec for ecosystem utilities

94f1210

Covers Marigold depth/normals examples, GroundingDINO+SAM2 segmentation task, RIFE frame interpolation task, and image metadata embedding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use direct imports for transformers in segment task spec

022ce29

transformers is a hard dependency, no lazy import needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add implementation plan for ecosystem utilities

aedffc7

13 tasks covering Marigold examples, segment task, frame interpolation task, and image metadata embedding with TDD approach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: add Marigold depth and normals example workflows

1752942

Example JSON workflows using MarigoldDepthPipeline and MarigoldNormalsPipeline for depth map and surface normal estimation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: add Segment and SegmentAndInpaint example workflows

ec2fc9c

Segment: gather image + text-prompted segmentation mask. SegmentAndInpaint: segment object then inpaint with FluxFill. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: add InterpolateFrames example workflow

b4bedab

Video generation with Mochi followed by 2x frame interpolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: add MetadataEmbed example workflow

62cb5bc

Demonstrates embed_metadata: true for PNG generation parameter embedding. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs: update TODO.md with completed items

0d99119

Mark Marigold depth, GroundingDINO+SAM2, RIFE interpolation, and image metadata embedding as done. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: vendor RIFE IFNet v4.6 for frame interpolation

a384d42

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

dkackman requested a review from Copilot March 26, 2026 18:54

Copilot started reviewing on behalf of dkackman March 26, 2026 18:55 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

Copilot started work on behalf of dkackman March 26, 2026 19:01 View session

test: add embed_metadata tests for Step class

a6fda74

Agent-Logs-Url: https://github.com/dkackman/diffusers-workflow/sessions/7078d921-2f46-4c61-a0da-369dac1ab9ba Co-authored-by: dkackman <5160233+dkackman@users.noreply.github.com>

Copilot finished work on behalf of dkackman March 26, 2026 19:06

dkackman added 7 commits March 26, 2026 17:50

rife tests

5e214fc

image to text

29a343e

docs

e4504ea

text generation tasks

b5b76bb

auto bucket resize

e1f6c70

strip exif nd add watermark

c2ab328

make upscale a task

9dbd6da

dkackman merged commit f082319 into master Apr 5, 2026
4 checks passed

Navigation Menu

Utilities - #25

Utilities#25
dkackman merged 20 commits into
masterfrom
Utilities

dkackman commented Mar 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

dkackman Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

dkackman Mar 26, 2026

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

dkackman Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

dkackman Mar 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dkackman commented Mar 26, 2026

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

dkackman Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

dkackman Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

dkackman Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

dkackman Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants