LoRA is the default fine-tuning method, but maybe not the best one

LoRA has become the reflex choice for fine-tuning large models, and a Hugging Face post argues the reflex is mostly habit rather than evidence. The authors first show how total the dominance is: 98.4% of model cards that name a single parameter-efficient method name LoRA, 95% of image-generation checkpoints are LoRAs, and 71.3% of GitHub code examples use it. Their worry is that popularity feeds on itself, so people pick LoRA because everyone else does, not because they measured it against the alternatives.

So they measured. Running more than 40 techniques through the PEFT library on the same tasks, they found no single winner, just different tradeoffs. On a math reasoning task (MetaMathQA on Llama-3.2-3B), LoRA reached 53.2% accuracy at 22.6 GB of memory; a method called Lily got to 54.9% at slightly higher memory, while BEFT used less memory but fell to 32.9%. For learning a single image concept on FLUX.2, a technique called OFT beat LoRA on both axes at once: higher similarity (0.708 versus 0.697) and lower memory (9.01 versus 9.97 GB). Which method wins depends on what you are optimizing, whether that is accuracy, memory, runtime, or checkpoint size, and the unified PEFT API makes switching between them a one-line config change.

Why it matters

If you fine-tune models, defaulting to LoRA may quietly cost you accuracy or memory that a different method would give you for free. Since switching techniques is a config change, it is worth benchmarking two or three on your own task before you commit to one.