Indian Social Media Platform Promotes Multilingualism
Koo, an Indian microblogging social media platform that was launched in late 2019, has announced that it is currently looking to expand the social network’s…
→ Continue ReadingLanguage Technology
Why Generative AI Still Struggles With Indian Languages
By Shrushti Chhapia
I
t almost feels like there was one world before ChatGPT and another after it. From the stories emerging worldwide about artificial intelligence (AI)-powered chatbots passing medical exams, writing poetry, and even mimicking human emotions, it seems like AI has cracked the code of human language. But beneath this impressive progress lies a significant gap — one that becomes glaringly obvious when we look beyond English and widely spoken Latin-script languages.
How does AI truly navigate the complexities of non-English languages, each with its own grammar, idioms, and cultural nuances? This question is especially pertinent when it comes to Indian languages. With over 22 official languages and hundreds of dialects, India is one of the most linguistically diverse countries in the world. But AI systems struggle to accurately understand, translate, and generate text in many Indian languages, especially those with complex grammar, rich oral traditions, and non-Latin scripts like Devanagari, Tamil, and Bengali.
This article explores the reasons why generative AI still struggles with Indian languages, detailing linguistic intricacies, data limitations, sociocultural factors, and technological gaps.
Advertisement
Data Scarcity — The Main Bottleneck
A big obstacle preventing generative AI from mastering Indian languages is the lack of large, high-quality datasets for training. Unlike English, which has billions of digitized texts, transcripts, and annotated datasets, most Indian languages remain underrepresented in the digital world. Many Indian languages, especially regional ones, have a strong oral tradition but a weak digital footprint. For example, while Hindi and Bengali have some presence online, languages like Bodo, Konkani, and Santali have minimal digitized resources. Even widely spoken languages like Punjabi and Marathi lack a comprehensive corpus of diverse text (e.g., legal documents, scientific papers), limiting AI’s ability to train on rich, varied linguistic inputs.
Nonstandardized Spelling and Script Variations
Another significant challenge is nonstandardized languages. Unlike English, where spelling is largely standardized, many Indian languages lack uniformity in writing. For instance, Bengali and Assamese share the same script but have different linguistic rules, making it harder for AI to distinguish between them. Similarly, Urdu and Hindi share vocabulary but use different scripts (Perso-Arabic and Devanagari, respectively), which adds another layer of complexity. AI models trained on inconsistent or fragmented data struggle to learn accurate linguistic patterns, resulting in flawed translations and unnatural text generation.
Code-Switching and Contextual Ambiguities
Code-switching (blending elements of two or more languages) and contextual ambiguities are common in India, and AI models struggle to handle these linguistic complexities. Many Indian people are at least bilingual, seamlessly switching between their native language, for example, Hindi (the country’s lingua franca) and English.
In the sentence in Figure 1, “amazing” and “movie” are English words embedded in a Marathi sentence. AI faces multiple challenges here:
- Language tagging: The model must correctly identify which words belong to which language and apply appropriate linguistic rules.
- Grammar adaptation: “Amazing” is an adjective following English grammar, but in Marathi, adjectives often agree in gender with the noun. AI needs to adjust its grammatical processing accordingly.
- Multilingual processing: This sentence includes both Marathi and English, requiring AI to recognize and process multiple linguistic frameworks simultaneously.
Because most AI training datasets are monolingual, they fail to handle these dynamic, real-world language patterns. Moreover, resource-poor languages like Marathi lack high-quality annotated datasets for code-switching, making it difficult for AI to learn the correct rules. This results in AI-generated translations sounding robotic, making them ineffective for real-life use cases in multilingual India.
Technological Limitations in AI Models
Finally, even the most advanced AI models today struggle with low-resource languages and dialects due to fundamental technological barriers. Most AI models rely on two key techniques to handle languages with limited training data:
Few-shot learning: AI learns from a small number of examples and generalizes patterns. Few-shot learning works best when at least a small but diverse dataset is available for AI to detect patterns. However, many Indian dialects have little to no written resources; most exist primarily as spoken languages with few digital texts, dictionaries, or formal linguistic studies.
Zero-shot learning: In this way of learning, AI attempts to interpret text without any prior training by transferring knowledge from similar high-resource languages. This works by leveraging similarities between an unknown language and a known one.
So, for instance, an AI trained on Spanish can understand Catalan quite well because the two languages share many similarities. Hindi and Bhojpuri also have overlapping vocabulary and grammar; however, AI struggles with Bhojpuri-specific expressions, pronunciation, and informal structures that don’t exist in standard Hindi.
In the example in Figure 2, Bhojpuri uses “चलल जाईं” (shall go), a verb structure that doesn’t exist in standard Hindi. AI, lacking Bhojpuri-specific training, incorrectly assumes a progressive tense “जा रहे हैं” (are going), distorting the sentence’s meaning.
Advertisement
Training AI Models
What are we doing to train the models? Despite the challenges, several innovative efforts to improve AI’s understanding of Indian dialects are underway. Here’s a look at how researchers, tech companies, and linguistic communities are working to bridge the gap:
- Government initiatives
- The Indian government’s Bhashini project under the Digital India initiative aims to create AI-powered translation models and speech recognition tools for all 22 scheduled languages and several dialects.
- The National Language Translation Mission seeks to make government documents and educational materials accessible in regional languages using AI.
- Crowdsourced language data collection
- Tech companies and research institutions are encouraging native speakers to contribute text and speech samples to improve AI’s accuracy in regional languages.
- AI research and development focused on Indian dialects
- Institutions like the Indian Institute of Technology Madras and the International Institute of Information Technology Hyderabad are developing AI-driven linguistic innovations tailored to Indian dialects.
- Microsoft Research India’s Project ELLORA is using deep learning to create adaptable AI models for dialect-heavy regions.
- Advanced AI models like mBERT and XLM-R are being trained to improve code-mixed and dialect-heavy conversations.
- Development of AI models that support code-switching
- Meta’s No Language Left Behind models are being trained on Hinglish (Hindi + English), Tanglish (Tamil + English), and Benglish (Bengali + English) to improve AI’s ability to process mixed-language inputs naturally.
- Chatbots and voice assistants are being optimized to interpret and respond to code-mixed queries without forcing users to choose between English and their native language.
Although generative AI has made remarkable strides in language processing, its struggle with Indian dialects highlights the deep-rooted challenges of data scarcity, code-switching complexities, and low digital representation of regional languages. The future of AI in India isn’t just about supporting dominant languages, it’s about ensuring that every dialect, every voice, and every linguistic identity finds its place in the digital world.
Shrushti Chhapia is the CEO and cofounder of eLanguageWorld, a language service provider specializing in Indian and Southeast Asian languages for more than 10 years. Currently based in London, Chhapia is a council member at the Association of Translation Companies, UK. She holds a B.Tech and an MA in Translation, combining tech expertise with deep industry insight.
Advertisement
Related Articles
Understanding India’s Complex Linguistic Landscape
The Southern Asian nation with more than a billion people has found a way to maintain its traditional linguistic identity at home while navigating diverse…
→ Continue ReadingHow AI Bridges the Gap Across 22 Indian Languages
AI in Indian languages is reshaping access in India, enabling translation in 22 tongues for services from e-commerce to government.
→ Continue Reading