rhymes-ai/Allegro-T2V-40x360P

Instalación básica con Diffusers:

pip install -U diffusers transformers accelerate

Ejemplo simple:

import torch
from diffusers import DiffusionPipeline

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("rhymes-ai/Allegro-T2V-40x360P", dtype=torch.bfloat16, device_map="cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

Requisitos recomendados: Python >= 3.10, PyTorch >= 2.4 y CUDA >= 12.4.

Instalación para el ejemplo completo:

conda create -n rllegro python=3.10 -y
pip install git+https://github.com/huggingface/diffusers.git torch==2.4.1 transformers==4.40.1 accelerate sentencepiece imageio imageio-ffmpeg beautifulsoup4

Inferencia completa:

import torch
from diffusers import AutoencoderKLAllegro, AllegroPipeline
from diffusers.utils import export_to_video

vae = AutoencoderKLAllegro.from_pretrained("rhymes-ai/Allegro-T2V-40x360P", subfolder="vae", torch_dtype=torch.float32)
vae.tile_overlap_t = 8
vae.tile_overlap_h = 144
vae.tile_overlap_w = 64
vae.stride = (16,112,192)

pipe = AllegroPipeline.from_pretrained(
    "rhymes-ai/Allegro-T2V-40x360P",
    vae=vae,
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")
pipe.vae.enable_tiling()

prompt = "A seaside harbor with bright sunlight and sparkling seawater, with many boats in the water. From an aerial view, the boats vary in size and color, some moving and some stationary. Fishing boats in the water suggest that this location might be a popular spot for docking fishing boats."

positive_prompt = """
(masterpiece), (best quality), (ultra-detailed), (unwatermarked), {}
emotional, harmonious, vignette, 4k epic detailed, shot on kodak, 35mm photo,
sharp focus, high budget, cinemascope, moody, epic, gorgeous
"""

negative_prompt = """
nsfw, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit,
fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,
signature, watermark, username, blurry.
"""

prompt = prompt.format(prompt.lower().strip())

video = pipe(
    prompt,
    negative_prompt=negative_prompt,
    guidance_scale=7.5,
    max_sequence_length=512,
    num_inference_steps=100,
    generator=torch.Generator(device="cuda:0").manual_seed(42)
).frames[0]

export_to_video(video, "output.mp4", fps=15)

Para reducir memoria GPU puede usarse pipe.enable_sequential_cpu_offload(), aunque aumenta de forma importante el tiempo de inferencia. Para interpolar de 15 FPS a 30 FPS, la tarjeta recomienda EMA-VFI.