MODEL 04 · Open models for Indian languages
My Model Thought Tamil Was Telugu: Open-Source Language ID for Hinglish, Tanglish and Kanglish
Bhasha, an Apache-2.0 model that labels the language of every word in Hinglish, Tanglish and Kanglish, and why human labels exposed its Tamil gap.
TL;DR
Bhasha is a free, 278M-parameter, Apache-2.0 model that tags the language of every word in code-mixed Indian text such as Hinglish, Tanglish and Kanglish, in Roman or native script. Trained only on code-built data, it reaches 0.84 to 0.95 on human-labelled Kannada and Malayalam sets, but Tamil scored 0.72 because it confused spoken Tamil with Telugu.
- 278Mparameters
- 11Indian languages + English
- 1.65Mcode-built training rows
- 0.84–0.95Kannada and Malayalam, human labels
- 0.72Tamil, human labels
Summary
This article presents Bhasha (arunp0/bhasha-lid-indic), a 278M-parameter, Apache-2.0 model that gives a language label to each word of a message rather than one label for the whole text. It covers English and 11 Indian languages, plus tags for names and other tokens, and reads Roman letters, native scripts or both in one sentence. That matters for comments like Tanglish film reviews, where half the words are Tamil typed in Latin script. The author could not find any licensed model that tags words in Roman-script Tamil, Malayalam or Kannada blended with English, so he built one.
Every training row was generated by code: native IndicCorp sentences were romanised the way people actually spell, English spans were spliced in, and noise such as emoji and stretched words was added, giving 1.65 million rows with no hand labels and no LLM-written text.
On 425 real YouTube comments with LLM labels, Bhasha scored 0.87 to 0.89 macro-F1, well above released Hinglish models. Human-labelled sets then held up for Kannada and Malayalam but showed Tamil at 0.723, and counting its errors revealed that 36.4% of Tamil words were tagged Telugu. The likely cause is that it learned only news-style Tamil. A language-pair hint helps slightly, and a v2 trained on real comments lifts Tamil to 0.94.
This page is a short summary written for this site. Every figure, table, code sample and reference is in the original article on Towards AI.
Questions this article answers
How do I detect the language of each word in a Hinglish or Tanglish sentence?
Use a word-level language identification model rather than a sentence-level one. Bhasha, released under Apache-2.0, takes a message and returns a language tag for every word, along with the main language, the script type, whether the text is mixed and each language's share. Where romanised Hindi and Urdu look identical it answers hi/ur instead of guessing, and a langs option restricts output to a known language pair.
How is word-level language ID different from GlotLID or IndicLID?
GlotLID and IndicLID assign a single language to a whole piece of text, which is fine for routing messages but loses detail when one sentence switches between, say, Kannada and English. Bhasha labels each word instead. According to the article it also finds the main language of real code-mixed comments better than GlotLID, 0.71 against 0.53 macro-F1, though GlotLID wins on whole romanised sentences written in one language.
Why does a language ID model confuse Tamil with Telugu?
In Bhasha's case, the training Tamil came only from news and web writing, which differs sharply from the spoken Tamil people type in comments. Faced with unfamiliar colloquial forms, the model reached for the nearest South Indian language it knew. On human-labelled Tamil words, 36.4% came out as Telugu. Blocking Telugu mostly shifted the errors to English, which pointed to a data problem rather than a decoding one.
Is there an open-source language identification model for Kannada and Malayalam code-mixed text?
Yes. Bhasha scores 0.84 to 0.90 macro-F1 on human-labelled Kannada sets and 0.91 to 0.95 on Malayalam ones, and the author says it is the only model he found above 0.84 on all of them. It is not top everywhere: distilka, the 2022 shared-task Kannada model, does better on a test from its own series, 0.954 versus 0.903.
Can you train a code-mixed language ID model without hand-labelled data?
Bhasha shows it can work, with a catch. Code took real native-script sentences, converted them to Roman spellings weighted by how often each appears online, inserted English spans of one to six words and recorded every word's language. The labels are therefore set by the generator, not guessed. The catch is the text itself: news-style sentences were enough for Malayalam and Kannada but not for colloquial Tamil.
What are Bhasha's limitations?
Spoken Tamil remains weak in the first release, so pass langs with the Tamil code and expect mistakes. Telugu, Marathi, Punjabi, Urdu and Odia lack a real code-mixed test, and Gujarati scored only 0.678 in mixed text. GlotLID is better on single-language romanised sentences, names are weak at 0.23 to 0.56 F1, and very short messages give it too little context.