Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers

NVIDIA and Hugging Face have announced a deep integration between the NVIDIA NeMo Automodel library and the Hugging Face Diffusers ecosystem. This collaboration delivers a production-grade, distributed training framework for large-scale image and video diffusion models directly from the Hub. The integration allows developers to scale fine-tuning tasks from a single GPU to hundreds without needing to convert model checkpoints or rewrite underlying model code, addressing a significant operational bottleneck in deploying custom generative AI.

The technical foundation of this partnership is NeMo Automodel, an open-source, PyTorch DTensor-native library designed to be "Hugging Face native." It directly uses Diffusers model classes and pipelines for loading and generation, ensuring seamless interoperability. The workflow involves pre-encoding a dataset into cached VAE latents, launching training with a YAML configuration file, and then using the fine-tuned checkpoint for inference. This system already supports prominent open-source models like FLUX.1-dev, Wan 2.1, and HunyuanVideo.

Key Technical Capabilities

• Native Hub Integration: Use any Diffusers model directly from the Hugging Face Hub without checkpoint conversions.
• Declarative Parallelism: Configure various sharding strategies such as FSDP2, tensor, and pipeline parallelism via YAML files rather than complex code rewrites.
• Efficient Training: Utilizes methods like latent caching, multiresolution bucketing, and a flow-matching objective to maximize throughput.
• Flexible Adaptation: Supports both full model fine-tuning for maximum quality and parameter-efficient methods like LoRA for resource-constrained environments.

This integration lowers the technical barrier for customizing state-of-the-art diffusion models. By embedding NVIDIA's scalable training logic into the standard Diffusers workflow, the collaboration provides researchers and organizations with access to high-performance computing capabilities previously confined to specialized frameworks. This is expected to accelerate the development and deployment of fine-tuned models for niche commercial and scientific applications, as the path from a base model on the Hub to a specialized, scaled-out version becomes significantly more direct.

This collaboration standardizes the MLOps pipeline for diffusion models, allowing organizations to leverage the vast Hugging Face model ecosystem with NVIDIA's proven, scalable training infrastructure. It bridges the critical gap between open-source model availability and the operational demands of enterprise-grade fine-tuning and deployment.