Fine-tuning a world model so it stops drawing human hands on robots

NVIDIA and Hugging Face published a hands-on guide to adapting Cosmos Predict 2.5, a 2-billion-parameter world model that generates physically plausible video from text or an image, for one specific job: producing synthetic robot manipulation footage. The motivation is practical. Recording real robot trajectories is slow and expensive, and a video model that can generate them gives robot-learning teams cheaper training data.

Out of the box the model has a tell. Shown a robot gripper, it tends to fall back on what it saw most in training and draw human hands instead, and the motion jitters. The fix is LoRA or DoRA fine-tuning, which freezes the base model and trains small adapter modules. That avoids the catastrophic forgetting you risk by updating all 2 billion parameters, keeps memory low enough for a single 80GB GPU, and produces adapter files small enough to swap per domain. With rank 32 the run touches only about 50 million parameters, and a 100-epoch job takes roughly 17 hours on one H100 or 2.5 hours across eight.

The guide also takes evaluation seriously. It measures geometric consistency with Sampson error across frames and viewpoints, and uses a separate model, Cosmos Reason2, as a judge that scores physical plausibility and instruction following. After fine-tuning on 92 example videos, the hands render correctly and the motion settles.

Why it matters

If you train robot policies, this is a concrete recipe for generating domain-matched video data without buying robot time. The reusable lesson is that a general world model usually needs a small, cheap adaptation step before its output is trustworthy for a narrow task, and the guide shows how to check that it worked rather than assume it did.