I
t almost feels like there was one world before ChatGPT and another after it. From the stories emerging worldwide about artificial intelligence (AI)-powered chatbots passing medical exams, writing poetry, and even mimicking human emotions, it seems like AI has cracked the code of human language. But beneath this impressive progress lies a significant gap — one that becomes glaringly obvious when we look beyond English and widely spoken Latin-script languages.
How does AI truly navigate the complexities of non-English languages, each with its own grammar, idioms, and cultural nuances? This question is especially pertinent when it comes to Indian languages. With over 22 official languages and hundreds of dialects, India is one of the most linguistically diverse countries in the world. But AI systems struggle to accurately understand, translate, and generate text in many Indian languages, especially those with complex grammar, rich oral traditions, and non-Latin scripts like Devanagari, Tamil, and Bengali.
This article explores the reasons why generative AI still struggles with Indian languages, detailing linguistic intricacies, data limitations, sociocultural factors, and technological gaps.
Advertisement
A big obstacle preventing generative AI from mastering Indian languages is the lack of large, high-quality datasets for training. Unlike English, which has billions of digitized texts, transcripts, and annotated datasets, most Indian languages remain underrepresented in the digital world. Many Indian languages, especially regional ones, have a strong oral tradition but a weak digital footprint. For example, while Hindi and Bengali have some presence online, languages like Bodo, Konkani, and Santali have minimal digitized resources. Even widely spoken languages like Punjabi and Marathi lack a comprehensive corpus of diverse text (e.g., legal documents, scientific papers), limiting AI’s ability to train on rich, varied linguistic inputs.
Another significant challenge is nonstandardized languages. Unlike English, where spelling is largely standardized, many Indian languages lack uniformity in writing. For instance, Bengali and Assamese share the same script but have different linguistic rules, making it harder for AI to distinguish between them. Similarly, Urdu and Hindi share vocabulary but use different scripts (Perso-Arabic and Devanagari, respectively), which adds another layer of complexity. AI models trained on inconsistent or fragmented data struggle to learn accurate linguistic patterns, resulting in flawed translations and unnatural text generation.
Code-switching (blending elements of two or more languages) and contextual ambiguities are common in India, and AI models struggle to handle these linguistic complexities. Many Indian people are at least bilingual, seamlessly switching between their native language, for example, Hindi (the country’s lingua franca) and English.
In the sentence in Figure 1, “amazing” and “movie” are English words embedded in a Marathi sentence. AI faces multiple challenges here:
Because most AI training datasets are monolingual, they fail to handle these dynamic, real-world language patterns. Moreover, resource-poor languages like Marathi lack high-quality annotated datasets for code-switching, making it difficult for AI to learn the correct rules. This results in AI-generated translations sounding robotic, making them ineffective for real-life use cases in multilingual India.
Finally, even the most advanced AI models today struggle with low-resource languages and dialects due to fundamental technological barriers. Most AI models rely on two key techniques to handle languages with limited training data:
Few-shot learning: AI learns from a small number of examples and generalizes patterns. Few-shot learning works best when at least a small but diverse dataset is available for AI to detect patterns. However, many Indian dialects have little to no written resources; most exist primarily as spoken languages with few digital texts, dictionaries, or formal linguistic studies.
Zero-shot learning: In this way of learning, AI attempts to interpret text without any prior training by transferring knowledge from similar high-resource languages. This works by leveraging similarities between an unknown language and a known one.
So, for instance, an AI trained on Spanish can understand Catalan quite well because the two languages share many similarities. Hindi and Bhojpuri also have overlapping vocabulary and grammar; however, AI struggles with Bhojpuri-specific expressions, pronunciation, and informal structures that don’t exist in standard Hindi.
In the example in Figure 2, Bhojpuri uses “चलल जाईं” (shall go), a verb structure that doesn’t exist in standard Hindi. AI, lacking Bhojpuri-specific training, incorrectly assumes a progressive tense “जा रहे हैं” (are going), distorting the sentence’s meaning.
Advertisement
What are we doing to train the models? Despite the challenges, several innovative efforts to improve AI’s understanding of Indian dialects are underway. Here’s a look at how researchers, tech companies, and linguistic communities are working to bridge the gap:
Although generative AI has made remarkable strides in language processing, its struggle with Indian dialects highlights the deep-rooted challenges of data scarcity, code-switching complexities, and low digital representation of regional languages. The future of AI in India isn’t just about supporting dominant languages, it’s about ensuring that every dialect, every voice, and every linguistic identity finds its place in the digital world.
Shrushti Chhapia is the CEO and cofounder of eLanguageWorld, a language service provider specializing in Indian and Southeast Asian languages for more than 10 years. Currently based in London, Chhapia is a council member at the Association of Translation Companies, UK. She holds a B.Tech and an MA in Translation, combining tech expertise with deep industry insight.
Advertisement
Related Articles
The theme of this year’s International Mother Language Day was “Multilingual education — a necessity to transform education,” emphasizing the role education can have in fostering…
→ Continue ReadingThe Southern Asian nation with more than a billion people has found a way to maintain its traditional linguistic identity at home while navigating diverse…
→ Continue ReadingAmazon expands its streaming strategy in India through Prime Video and MX Player, using dubbing to reach multilingual audiences across the country.
→ Continue Reading
