rhymes-ai/Allegro

Instalación rápida con Diffusers:

pip install -U diffusers transformers accelerate

import torch
from diffusers import DiffusionPipeline

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("rhymes-ai/Allegro", dtype=torch.bfloat16, device_map="cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

Ejemplo completo recomendado por la ficha del modelo:

conda create -n rllegro python=3.10 -y
pip install git+https://github.com/huggingface/diffusers.git torch==2.4.1 transformers==4.40.1 accelerate sentencepiece imageio imageio-ffmpeg beautifulsoup4

import torch
from diffusers import AutoencoderKLAllegro, AllegroPipeline
from diffusers.utils import export_to_video

vae = AutoencoderKLAllegro.from_pretrained("rhymes-ai/Allegro", subfolder="vae", torch_dtype=torch.float32)
pipe = AllegroPipeline.from_pretrained(
    "rhymes-ai/Allegro",
    vae=vae,
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")
pipe.vae.enable_tiling()

prompt = "A seaside harbor with bright sunlight and sparkling seawater, with many boats in the water. From an aerial view, the boats vary in size and color, some moving and some stationary. Fishing boats in the water suggest that this location might be a popular spot for docking fishing boats."

positive_prompt = """
(masterpiece), (best quality), (ultra-detailed), (unwatermarked), {}
emotional, harmonious, vignette, 4k epic detailed, shot on kodak, 35mm photo,
sharp focus, high budget, cinemascope, moody, epic, gorgeous
"""

negative_prompt = """
nsfw, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit,
fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,
signature, watermark, username, blurry.
"""

prompt = prompt.format(prompt.lower().strip())
video = pipe(
    prompt,
    negative_prompt=negative_prompt,
    guidance_scale=7.5,
    max_sequence_length=512,
    num_inference_steps=100,
    generator=torch.Generator(device="cuda:0").manual_seed(42)
).frames[0]

export_to_video(video, "output.mp4", fps=15)

Requisitos indicados: Python >= 3.10, PyTorch >= 2.4 y CUDA >= 12.4. Para reducir memoria de GPU se puede usar pipe.enable_sequential_cpu_offload(), con mayor tiempo de inferencia.