Modern tech meets Indian Sign Language

Indian Sign Language (ISL), also known as Indo-Pakistani Sign Language (IPSL), is one of the most common sign languages in South Asia, used by millions of speakers in India, Pakistan and Bangladesh. However, as is the case with some sign languages, it is difficult to accurately estimate the number of people who practice ISL. The study on sign languages in India meets a massive roadblock in particular, as the actual number of deaf people has not been ascertained. This barrier creates an enormous problem for individuals and organizations who want to have new technologies implemented around the usage of sign languages in public spheres.

The 2011 Indian census stated that roughly 1.3 million people in India experience hearing impairment. These numbers are quite different from that of India’s National Association of the Deaf, which puts this number at 18 million, or roughly 1% of India’s population. But even these numbers are low when they are compared to global standards — it has been reported that 5% of the world’s population and 3.5% of Americans experience hearing loss.

The pattern of underreporting is not surprising when it is compared to other forms of disability. The 2011 Indian census reported that 2.21% of India’s population is disabled, while the global average for that year stood at 15%.

The reason for such underreporting is partially due to the social stigma around disability. This means census takers experience an inability to accurately report the cases of disability. The resulting data then can impair the formation of an effective roadmap for the implementation of sign languages in universities and other public spheres.

A rudimentary stage

Unlike the sign languages in the west, ISL is at a rudimentary stage, where it is still struggling to be acknowledged as a language for minorities. Acknowledgement would help it garner more investment of resources so that it could expand and adapt to modern platforms.

Even though ISL is used by numerous deaf people in India, it is still not an integral part of the teaching curriculum. It was only as recently as 2005 that the National Curriculum Framework (NCF) in India acknowledged sign languages and hinted at them qualifying as an optional choice of language for the hearing-impaired. Since then, numerous independent and organizational efforts have been made at implementing sign languages in different curriculums for different languages, including a chapter on sign languages in by NCERT in a third-grade textbook.

ISL is unrelated to either Hindi or English, although it draws some signs from British Sign Language and its alphabet is based on English. It has its own dialects, too. “Indian Sign Language is very scientific and has its own grammar, but lack of awareness has meant that many deaf people are not even aware of institutions where they can learn it and equip themselves for public communication,” said Andesha Mangala, assistant professor at the government-funded Indian Sign Language Research and Training Centre (ISLRTC) in a BBC report.

The need for a comprehensive system around sign language starts with an official dictionary for it. The ISL dictionary (now upgraded to a second edition) was launched in March 2018. Presently, it contains 6,000 words, in categories such as academic, medical, technical, legal and everyday terms.

However, the presence of a dictionary certainly doesn’t put an end to the challenges. There are not enough schools in India that cater to the needs of disabled students, and the ones that exist face problems with funding and expensive technologies that are imported from other countries.

Retired colonel Kamrinder Singh is the secretary of the Society for Welfare of the Handicapped, which was established in 1967, and manages three schools — Patiala School for the Deaf, Patiala School for the Blind and Patiala School for the Deaf-blind — in Saidipur village near the city of Patiala in Punjab. Currently, there are 400 students in these NGO-run schools.

“There haven’t been any technologies that cater well to the needs of our students in India, and we have been importing most of our technology from countries like USA, Japan and Germany, and they tend to be expensive. The Indian versions are free of cost, but they are not up to the mark compared to their western counterparts,” said the retired colonel.

Singh hopes that the software gets better with time, as it would add to the cost-effectiveness of these programs that are not funded by the government. “Disability is a state subject, and more often than not, these roles are not played well due to the lack of funding,” he added.

Although the launch of the dictionary has opened up doorways to mass-implementation of ISL on primary fronts, the main challenge still is the absence of easily accessible tech-based solutions around sign languages that could work alongside human interpreters. India’s deaf population is largely illiterate, so even reading public signs can be a challenge.

However, home-grown solutions on these fronts are emerging.

Researchers at the Punjabi University in Patiala, India, have developed an automatic translation system based on ISL that takes English text as input. The program then machine translates it to ISL using synthetic animation in the real world.

“Around four years ago, I was visiting the school in Saidipur village in Patiala, and I came across a lot of students who are hard of hearing. This is when the idea came to me. I thought that we need to have a system through which they can read the texts in their curriculum — a simple interface that would allow them to copy things to the interface and have them converted to a sign language,” said Vishal Goyal, who has been working on this project with the aid of his colleague Lalit Goyal and students at Punjabi University.

Vishal Goyal has worked extensively in machine translation (MT) systems for Indian languages, including developing a Hindi to Punjabi  MT system copyrighted in 2011 and available for use at http://h2p.learnpunjabi.org/. “Earlier my research was in natural language processing, and this idea came to my mind that since machine translation is my research area, I should use this system for the community,” he said.

How the program works

The program, which was copyrighted in 2018 and is in its final stages of development, uses an English parser for parsing texts. The input in English is broken down to get the phrases, structure and grammar of the sentence. After that, the phrase-reordering module reorders the words of English sentence according to the rules or ISL grammar.

Then, the eliminator module of the program gets rid of unwanted words from a sentence such as linking words, articles and so on. The output of the eliminator module is then sent to the lemmatization module, which converts the words into the root form. Now, each root word is checked for availability in the English ISL dictionary. The unknown words in the sentences are replaced with their synonyms. If a synonym is not available, the fingerspelling of the words is performed character by character. The words are then played out through an animation tool.

This application requires less memory as compared to human sign language videos, and is highly flexible and programmable as it can be altered using synthetic animation of various kinds — animated kids, grownups or animals.

This program happens to be the world’s first working system for ISL, and the scope for its applications are huge, especially in the public spheres.

Goyal and his team are pushing for the program to be implemented at a national level through various organizations operating in the field. For instance, if the program is implemented through the University Grants Commission (UGC) in Indian universities, it can cater to the needs of students who are hard of hearing. The application of such program is not just limited to universities, but also at public places such as hospitals, airports and railway stations in India that do not yet have a functional system to cater to the needs of hearing-impaired people.

This system stands out from the past work that has been done in this field as it has more animations along with a better flow in them. It also has more vocabulary which makes the program more efficient.

A promising scope

This work in the field of sign languages has started seeing fruitful results. The government of India has approved a pilot project using ISL with the collaboration of Punjabi University. It has already asked the board for a proposal which includes the scope, utility, specifications and expenditures for the project, which is to be piloted at the New Delhi railway station.

The project will serve as a model for other railway stations in the future. Henceforth, it is expected that the program will garner interest from other organizations that operate in the public sector.

Once the program gets more funding from government bodies, it will be capable of working with not just English, but also other Indian languages to provide more regional context in the areas of its operation.

For his contributions to the field, Vishal Goyal received a state award on World Disability Day,  December 3, 2019. Presently, he is overseeing other sub-projects that can further extend the capabilities of the program, including an automatic telecast of news in ISL through synthetic animations; a machine translation system from Punjabi; text to ISL synthetic animations; and translating complex and compound English text sentences to ISL synthetic animations. Along with his team, he is also working on a similar model which could be implemented for airport announcements. This program has already been funded by a private organization, and if things go well, it could implemented in airports.

However, the biggest challenge still lies with the acceptance of this program with core organizations that are responsible for standardizing the ISL, such as the ISLRTC, which was set up by the government of India 2015.

“The biggest challenge is to work together with the ISLRTC because they have the people and resources to test this system on the ground. It will be great to work with their resources and expertise,” said Vishal Goyal.

The Ministry of Social Justice and Empowerment, which oversees ISLRTC, said that it is open to new ideas around technologies that can help benefit people with disabilities. “If someone wants a collaborative effort with the government, they can write to the secretary with a detailed plan,” said Mrityunjay Jha, the deputy secretary at the Department of Empowerment of Persons with Disabilities, operating under the umbrella of the Ministry of Social Justice and Empowerment.

“I gave a demonstration of the software to ISLRTC a few years ago. Even though they liked our work, their only concern was with introducing facial expressions, that they were not up to the mark. However, if they show interest in a project like this, this is something that we can improvise on as per their needs, as the foundation for such a system is set,” explained Lalit Goyal when asked about working with ISLRTC to expand the applications of the program.

There has also been parallel work done to have a mobile application which contains a video catalog of all the words released by ISLRTC. The app, simply named SignLearn – ISL learning app, is available on the Google Play Store for free download and has already been downloaded over 5,000 times. It is based on the video catalog that is released by ISLRTC.

Jestin Joy, the developer of the app who has been working on it as a part of his research for around five years, thinks that such a platform can be helpful in communicating with hearing impaired individuals, as it has a vast catalog of around 4,500 signs.

Since learning sign language requires a lot of hand movements, learning about it through text books is not easy, and this is where multimedia platforms can come in handy.

“This is a dictionary for those who are hard of hearing. If you have a spoken language, then a normal dictionary is okay, but we need to describe words as signs and fingers movements, and this is where multimedia solutions will be helpful,” Joy said.

When asked about the implementation of this synthetic animation program to modern platforms such as mobile apps, Vishal Goyal seemed quite positive. “That’s not a big deal. Once we gain some traction and acceptance by more organizations in India, these are minor things that can easily be taken care of,” he said.