Add JoyAI-Video-Edit pipeline

[feice-huang]

We are the JoyAI Team, and this is the Diffusers implementation for the JoyAI-Video-Edit model.

• Reference implementation: https://github.com/jd-opensource/JoyAI-Video-Edit
• Original checkpoint: https://huggingface.co/jdopensource/JoyAI-Video-Edit
• Diffusers checkpoint: https://huggingface.co/jdopensource/JoyAI-Video-Edit-Diffusers
• MiMo-VL: https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-RL-2508

What does this PR do?

Fixes #14524

Overview

This PR adds Diffusers support for JoyAI-Video-Edit.

JoyAI-Video-Edit is an instruction-guided video editing model built around causal autoregressive diffusion. It combines:

• MiMo-VL for multimodal prompt conditioning
• a causal video VAE
• a dual-stream multimodal diffusion transformer
• autoregressive chunk-wise denoising with a bounded KV cache

The model supports open-ended editing instructions such as subject modification, local object editing, background replacement, style transfer, motion changes, and reference-image-guided editing.

Its causal design allows each generated chunk to reuse previously denoised context without depending on future generated frames. A sliding context window keeps the attention state bounded, while an optional global sink chunk helps retain long-range consistency.

What is added

This PR introduces:

• JoyVideoEditTransformer3DModel
• AutoencoderKLJoyVideoEdit
• JoyVideoEditPipeline
• JoyVideoEditKVCacheConfig and its Diffusers cache hook
• a checkpoint conversion script
• model and pipeline documentation
• model, pipeline, conversion, and cache lifecycle tests

The converted checkpoint can be loaded through the standard Diffusers API:

pipeline = JoyVideoEditPipeline.from_pretrained(
    "jdopensource/JoyAI-Video-Edit-Diffusers",
    text_encoder=text_encoder,
    processor=processor,
    dtype=torch.bfloat16,
)

Example

• input video: https://raw.githubusercontent.com/jd-opensource/JoyAI-Video-Edit/main/assets/input.mp4
• prompt: "Transform the scene into a British castle royal aristocratic style. Modify the characters' clothing to aristocratic attire: dress the man in a tailored velvet suit with a ruffled cravat, and the women in elegant silk gowns with lace details and embroidered bodices. Change their hairstyles to classic aristocratic styles, such as elaborate updos with subtle jewels for the women and a neatly styled classic cut for the man. Change the environmental decoration to a British castle interior: replace the plain walls and abstract painting with stone walls and antique oil paintings in gilded frames, and replace the white window curtains with heavy velvet drapes. The characters' ages and facial features must remain completely unchanged. The dining table, white tablecloth, plates of food, wine glasses, water glasses, and the characters' positions and actions must remain unchanged."
• output video: https://raw.githubusercontent.com/jd-opensource/JoyAI-Video-Edit/main/assets/output.mp4
• environment: torch==2.8.0 transformers==5.15.0 diffusers installed from source

processor = AutoProcessor.from_pretrained(MIMO_PATH)
text_encoder = Qwen2_5_VLForConditionalGeneration.from_pretrained(MIMO_PATH, dtype=torch.bfloat16)
pipeline = JoyVideoEditPipeline.from_pretrained(
    MODEL_PATH,
    text_encoder=text_encoder,
    processor=processor,
    dtype=torch.bfloat16,
)
pipeline.enable_model_cpu_offload()

video = load_video(VIDEO_PATH)

output = pipeline(
    video=video,
    prompt="Transform the scene into a British castle royal aristocratic style. Modify the characters' clothing to aristocratic attire: dress the man in a tailored velvet suit with a ruffled cravat, and the women in elegant silk gowns with lace details and embroidered bodices. Change their hairstyles to classic aristocratic styles, such as elaborate updos with subtle jewels for the women and a neatly styled classic cut for the man. Change the environmental decoration to a British castle interior: replace the plain walls and abstract painting with stone walls and antique oil paintings in gilded frames, and replace the white window curtains with heavy velvet drapes. The characters' ages and facial features must remain completely unchanged. The dining table, white tablecloth, plates of food, wine glasses, water glasses, and the characters' positions and actions must remain unchanged.",
    num_inference_steps=2,
    generator=torch.Generator(device="cpu").manual_seed(0),
)
export_to_video(output.frames[0], OUTPUT_PATH, fps=24)

Self-review

Blocking issues

None.

Non-blocking issues

None.

Dead code review

Location	Status	Reason
autoencoder_kl_joyvideoedit.py:162	Retained	self.in_channels provides conventional channel metadata for module introspection and custom attention processors. It is not considered dead code.

Summary

Verdict: READY

• The PR contains exactly one commit relative to origin/main.
• The complete make quality check passes.
• Tests pass: 142 passed, 16 skipped.
• VAE output parity and state-dict compatibility were preserved.
• No blocking or non-blocking issues remain.
• No changes need to be deferred to the reviewer.

This self-review can be added to the PR description or posted as a PR comment.

Before submitting

• Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the Coding with AI agents guide?
  • Did you run the self-review skill on the diff?
  • Did you share the final self-review notes in the PR description or a comment?
• Did you read the contributor guideline?
• Did you read our philosophy doc? (important for complex PRs)
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?
• Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

• Pipelines and models: @yiyixuxu @dg845 and @asomoza