MODEL 01 · Open models for Indian languages
Before You Invent a New Architecture, Try a Bigger Encoder
Building Yama, a 586M zero-shot classifier for 11 Indian languages and Hinglish, and why the encoder swap beat every clever idea.
TL;DR
Yama is a 586M-parameter, Apache-2.0 zero-shot classifier for 11 Indian languages, Hinglish and English that returns calibrated probabilities for labels you define at run time. While building it, looped transformers, targeted data and continued pretraining each added under a point; switching to a bigger encoder added 5.3. On 33 unseen datasets it scores 0.814, against 0.770 for the best open alternative tested.
- 586Mparameters
- +5.3from a bigger encoder
- 0.814 vs 0.770unseen tasks
- ~17 msper decision, H100
Summary
This article is the build log of Yama (arunp0/yama-indic), an open-weights, Apache-2.0, 586M-parameter zero-shot classifier. It reads text in 11 Indian languages as well as Hinglish and English, takes questions and labels written at inference time, and it returns a calibrated probability for each label in one forward pass. It answers three kinds of question: pick one option, yes or no, and a rating on a scale you define.
The main lesson is in the title. Starting from an mmBERT-base encoder that scored 0.800, the author tried targeted reasoning data, looped transformer blocks in the decision head and in the encoder, and continued pretraining. Each moved the 67-task benchmark by less than a point. Swapping in the larger XLM-RoBERTa-large encoder moved it by 5.3 points. One reason: only 110M of mmBERT-base's 307M parameters sit outside its vocabulary embeddings.
That swap needed care. The first XLM-R run collapsed around step 28,000 when the loss roughly doubled; moving from fp16 to bf16 and halving the learning rate fixed it.
On 33 pick-one-label tasks from datasets it never trained on, Yama scores 0.814 against 0.770 for bge-m3-zeroshot-v2.0, winning 29 of the 33. The article is open about limits: reasoning tasks such as IndicCOPA and Hindi BoolQ sit near 0.62, inputs are capped at 512 tokens, about half the labels come from language models, and an unseen dataset is not an unseen task type. A single decision takes about 17 ms on an H100.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
What is a zero-shot text classifier for Indian languages?
It is a model that sorts text into labels you invent at run time, with no task-specific training. Yama, released as arunp0/yama-indic under Apache-2.0, accepts input in Hindi, Bengali, Tamil and eight other Indian languages, Hinglish or English, with an English question and labels, and gives back a calibrated probability for every label from one forward pass. It handles pick-one questions, yes/no questions and ordinal ratings, at about 17 ms per decision on an NVIDIA H100.
Should I try a bigger encoder before changing the model architecture?
The article's results say yes. On a 67-task benchmark, targeted reasoning data, looped transformer blocks in the head, looped encoder layers and continued pretraining each changed the score by less than a point. Replacing mmBERT-base with XLM-RoBERTa-large on the same data added 5.3 points. The author's takeaway is to run the bigger-encoder baseline first, because depth and wiring were not the bottleneck; capacity was.
Why did my XLM-RoBERTa-large fine-tuning run collapse?
One common cause is fp16 overflow. In the Yama project, the first XLM-R-large run looked fine until around step 28,000, when the running loss jumped from about 0.65 to 1.35 and stayed there. Switching to bf16, which has a wider numeric range, and halving the learning rate fixed it. A safety check that reverts to an earlier checkpoint and cuts the learning rate on any loss spike was added afterwards.
How does Yama compare with bge-m3-zeroshot and bart-large-mnli?
On 33 pick-one-label tasks drawn from datasets Yama never trained on, it averaged 0.814 accuracy against 0.770 for bge-m3-zeroshot-v2.0, the strongest open zero-shot model tested, and won 29 of the 33 tasks. Of the NLI-based baselines, three of the four scored only 0.35 to 0.59 on the Indian-language tasks, and bart-large-mnli is English-only. Calibration error on the unseen tasks was a tie at 0.092.
What are Yama's limitations?
Reasoning is still weak: IndicCOPA and Hindi BoolQ both score about 0.62, and BoolQ at that level is close to always answering yes. Inputs are capped at 512 tokens, about half the training labels were produced by language models, and there is no Odia test set. Its strong results are on unseen datasets, not unseen task types, and it is not a tested moderation system, so recalibrate before setting thresholds.