[pipeline] CogVideoX-Fun Control

[a-r-r-o-w]

What does this PR do?

Adds support for CogVideoX-Fun Pose Control models

• alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose
• alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose

Usage

import torch
from diffusers import CogVideoXFunControlPipeline, DDIMScheduler
from diffusers.utils import export_to_video, load_video

pipe = CogVideoXFunControlPipeline.from_pretrained(
    "alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose", torch_dtype=torch.bfloat16
)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")

control_video = load_video(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/hiker.mp4"
)
prompt = (
    "An astronaut stands triumphantly at the peak of a towering mountain. Panorama of rugged peaks and "
    "valleys. Very futuristic vibe and animated aesthetic. Highlights of purple and golden colors in "
    "the scene. The sky is looks like an animated/cartoonish dream of galaxies, nebulae, stars, planets, "
    "moons, but the remainder of the scene is mostly realistic."
)

video = pipe(prompt=prompt, control_video=control_video).frames[0]
export_to_video(video, "output.mp4", fps=8)

hiker.mp4

output.mp4

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@bubbliiiing @yunkchen

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

thanks! awesome! so we did not need to update the model at all?
I left comments about scheduler

[a-r-r-o-w]

so we did not need to update the model at all?

nope, no changes at all :)

[bubbliiiing]

Thank you for the cc. How can I help you?

[a-r-r-o-w]

Hi @bubbliiiing. Thanks for your awesome work! I would be grateful if you could give this PR a review and verify its correctness and if it's compatible with the original pipeline/scripts. Happy to merge this if you think it's good (and subsequently the inpainting models in a different PR) :)

Also would love to collaborate on any future releases you are planning on and if we can help with training/finetuning in any way as @sayakpaul and I are looking into it too alongside the CogVideoX team

[stevhliu]

LGTM, thanks!

[yiyixuxu]

awesome!

[sayakpaul]

Awesome. Just a single a comment!

[bubbliiiing]

LGTM.

[bubbliiiing]

This is the test result after I simply modified the prompt words, it is very good.
https://github.com/user-attachments/assets/daddc0eb-e2c4-44ac-b86e-24a668308de9

[sayakpaul]

@bubbliiiing thanks for reviewing and for testing this PR.

Would it be possible to also add a note about diffusers support in your repo?

[bubbliiiing]

@bubbliiiing thanks for reviewing and for testing this PR.

Would it be possible to also add a note about diffusers support in your repo?

Of course! Could you provide me with an example of adding "diffusers support"? I would like to add it in the same way.

[sayakpaul]

Of course! Could you provide me with an example of adding "diffusers support"? I would like to add it in the same way.

Maybe something like:

CongVideoX-Fun Control is now supported in diffusers. Thanks to a-r-r-o-w who contributed the support in this PR. Check out the docs to know more.

[bubbliiiing]

Of course! Could you provide me with an example of adding "diffusers support"? I would like to add it in the same way.

Maybe something like:

CongVideoX-Fun Control is now supported in diffusers. Thanks to a-r-r-o-w who contributed the support in this PR. Check out the docs to know more.

I understand. I'll submit a PR to the CogVideoX-Fun repository.

[bubbliiiing]

Mission completed.