IBM's new Granite embeddings beat much larger models

IBM has released the second generation of its Granite multilingual embedding models, and the interesting part is how much it gets out of small models. There are two, at 97 million and 311 million parameters, both Apache 2.0 and both built on the ModernBERT encoder rather than the older XLM-RoBERTa. The context window jumps from 512 tokens to 32,768, a 64-fold increase, which matters if you embed long documents instead of short passages.

The numbers IBM reports are concrete. On the multilingual MTEB suite across 18 languages the 97M model scores 60.3, up 12.2 points over R1, while the 311M model reaches 65.2 and ranks second among open models under 500M parameters. On long-document retrieval the 311M model is first overall. The small model runs at roughly 2,500 documents per second and its weights are 195 MB, so it fits in places a 300M model does not. The 311M model also supports Matryoshka embeddings, so you can cut the vector from 768 to 128 dimensions and keep about 97 percent of quality while using six times less storage. IBM credits the gains to distillation from several teacher models, contrastive fine-tuning across 52 languages and code, and merging checkpoints. The models work out of the box with Sentence Transformers, LangChain, LlamaIndex, and others, with details on the Hugging Face blog.

Why it matters

If you run retrieval or RAG, a 97M Apache 2.0 model that beats 300M-class baselines and handles 32K context is a direct cost and latency win, and Matryoshka lets you trade a little accuracy for much smaller vector storage.