Luo-Yihong/TDM_CogVideoX-2B_LoRA

Instalación y uso básico con Diffusers:

pip install -U diffusers transformers accelerate

import torch
from diffusers import DiffusionPipeline
from diffusers.utils import export_to_video

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("THUDM/CogVideoX-2b", dtype=torch.bfloat16, device_map="cuda")
pipe.load_lora_weights("Luo-Yihong/TDM_CogVideoX-2B_LoRA")

prompt = "A man with short gray hair plays a red electric guitar."
output = pipe(prompt=prompt).frames[0]
export_to_video(output, "output.mp4")

Ejemplo específico de TDM-CogVideoX-2B-LoRA:

import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXPipeline.from_pretrained("THUDM/CogVideoX-2b", torch_dtype=torch.float16)
pipe.vae.enable_slicing() # Save memory
pipe.vae.enable_tiling() # Save memory
pipe.load_lora_weights("Luo-Yihong/TDM_CogVideoX-2B_LoRA")
pipe.to("cuda")

prompt = (
    "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The "
    "panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other "
    "pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, "
    "casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. "
    "The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical "
    "atmosphere of this unique musical performance")

# We train the generator on timesteps [999, 856, 665, 399].
# The official scheduler of CogVideo-X using uniform spacing, may cause inferior results.
# But TDM-LoRA still works well under 4 NFE.
# We will update the TDM-CogVideoX-LoRA soon for better performance!
generator = torch.manual_seed(8888)
frames = pipe(prompt, guidance_scale=1, num_inference_steps=4, num_frames=49, generator = generator).frames[0]
export_to_video(frames, "output-TDM.mp4", fps=8)