MODEL 05 · Open models for Indian languages
My Model Thought Tamil Was Telugu, Part 2: How 51,000 YouTube Comments Fixed It
Real YouTube comments, labelled by an open-weights LLM after a rule labeller failed, took Bhasha's Tamil score from 0.72 to 0.94, plus a test-set leak.
TL;DR
Bhasha v2 retrains a 278M-parameter open model for word-level Indian language ID with 51,071 real YouTube comments labelled by Gemma 4 26B-A4B. Tamil F1 on human-labelled comments rose from 0.72 to 0.94 and Tamil words tagged as Telugu fell to 0.8%, though a 5-word overlap check means the Malayalam test is no longer independent.
- 0.72 → 0.94CoLI Tamil F1
- 36.4% → 0.8%Tamil words called Telugu
- 51,071real comments added
- 7.5%LLM label word error
- 278Mparams, runs on CPU
Summary
Part 2 of the Bhasha story fixes the problem Part 1 exposed. Bhasha is a 278M-parameter, Apache-2.0 model that tags the language of every word in code-mixed Indian text. Version 1 was trained only on sentences generated by code, and on human-labelled Tamil YouTube comments from CoLI-Dravidian 2024 it scored 0.72, calling 36.4% of the Tamil words Telugu.
The fix was real comments. The author took 51,071 Tamil, Malayalam and Kannada comments from the training split of DravidianCodeMix and wrote down a stop rule (at most 10% wrong word labels) and a ship rule before starting. A hand-written rule labeller looked fine on the comments it was tuned on but failed on 150 fresh ones, with 11.6% wrong. An open-weights labeller, Gemma 4 26B-A4B on vLLM, passed at 7.5% once every word was sent with its own number, after two request formats failed.
Bhasha v2 adds those comments to the original 1.65 million training rows. On CoLI Tamil it scores 0.94, and Tamil words tagged as Telugu fall below 1%; other human-labelled tests moved by small amounts that the author reads as no worse. A later check found 24% of CoLI Malayalam's 5-word runs inside the new training text, so that test no longer counts as independent. Kannada against the 2022 shared-task winner, Telugu chat and Urdu remain open weaknesses.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
What is Bhasha v2?
Bhasha v2 is a free, Apache-2.0 language-identification model that tags every word of a message with its language: English, any of 11 Indian languages, a name, or other. It handles romanised and native-script text such as Tanglish and Kanglish, has 278M parameters and runs on a laptop CPU. It is version 1 retrained with 51,071 real YouTube comments added.
Why did the first Bhasha model confuse Tamil with Telugu?
Version 1 never saw real chat. Its 1.65 million training sentences were news and web text converted to English letters by code, and romanised news Tamil looks very different from spoken comments such as “padam semma mass”. On 8,064 human-labelled Tamil words it called 36.4% Telugu and only 36.3% Tamil, and telling it which language to expect helped only slightly.
Can an LLM label training data for language identification?
In this project it could, with checks. Gemma 4 26B-A4B, an open-weights model, labelled 51,071 comments word by word with 7.5% word error on 150 fresh comments. It worked only after each word was sent with a number, because plain lists led to merged or split words. A simpler rule labeller failed the 10% stop line on fresh data.
How much did Bhasha v2 improve on Tamil?
On the CoLI-Dravidian 2024 Tamil test, word-level macro-F1 rose from 0.72 to 0.94 in a single training run, and the share of Tamil words tagged as Telugu fell from 36.4% to 0.8%. The article notes that part of the gain may simply come from seeing YouTube comments for the first time, which it cannot separate out.
What test-set leak was found after training?
Deduplication compared each comment with whole 12-word test windows, so short copies slipped through. Checking every 5-word run found 24% of CoLI Malayalam's runs in the new training text and 1.9% of CoLI Tamil's. Scores on the clean windows barely changed, but CoLI Malayalam is now treated as not independent for v2.