The Present and Future of Natural Language Processing

Babita Jain has been in the technology industry for over 25 years and for the past decade has focused on helping brands develop software solutions with the use of AI and NLP to move into new markets. She is currently the director of enterprise and mobility at RWS Moravia.


Hardik Dwivedi is a data scientist with a track record of driving international projects to support major brands in their globalization journey. A project manager at RWS Moravia, Hardik has 15 years of industry experience in deep learning, machine learning and NLP.


Since COVID-19 began to hold the world in its vice-like grip last year, artificial intelligence (AI), and more specifically its language-related applications where natural language processing (NLP) plays an indisputable role, have become more relevant than ever. There are numerous areas of AI that had to evolve quickly over the past months to keep pace with everyone in the world craving information immediately and on their terms.

Take chatbots, for example. According to the World Economic Forum, the two most authoritative voices of the pandemic, the World Health Organization (WHO) and the Centers For Disease Control and Prevention (CDC), have also included chatbots in their websites to provide up-to-date information to billions on the spread of disease and its symptoms.

NLP and its business applications

NLP is a field of artificial intelligence that takes human language and makes it understandable to machines. Computer science then transforms this linguistic knowledge into rule-based and machine learning algorithms that can solve specific problems and perform desired tasks. NLP programs computers to process and analyze large amounts of natural language data. Good examples of this are machine translation, voice recognition in virtual assistants, chatbots, digital phone calls, predictive text, and email filters — like when Gmail categorizes incoming emails into social, promotions, or primary inboxes based on its content.

NLP can convert free-flow lingual text into a structured format. Free-flow lingual text is just natural speech or writ-ing. Like this sentence: “All fragments and, um, pauses and different ideas and punctuation all over the place and repeated words and, well, the way we speak, basically.” NLP transforms this linguistic chaos into easily understandable datasets in order for us to build applications that rely on this digested language information and turn the numerical output into meaningful text that humans can understand.

NLP applications access the metadata and the keywords in the text it processes. From this data, it generates a graphical view and provides a textual answer. For example, if Kentucky Fried Chicken wants to market a product only to Hispanic people, and they want to know how many went to the restaurant in the last year, they can ask a system developed with NLP, and it will understand what they want, extract the data from the system, and provide it to the team at Kentucky Fried Chicken.

People could be using NLP applications without knowing it. For instance, with Siri or Alexa, if you ask them some-thing, like to play a certain song or search something on the internet, it will understand that. This is just one example of how people, without realizing it, are benefitting from NLP. Also, if you are driving a car and you get a text message, your car, with the help of NLP, could read the text to you. You can then reply by speaking. The car identifies you, identifies what you want to do, and understands your voice command.

Solving problems with NLP

Right now, in machine learning and AI, NLP is the most searched topic. Companies are researching NLP because they want to create tech that can be used by people to communicate more easily. They are especially interested in on-the-fly translation.

More global content is driving demand for NLP solutions. NLP becomes even more important with multiple languages. In these situations, there is a huge amount of data that slows business because multiple linguistic teams are needed. Companies that want to analyze this data are looking to do it efficiently and at a reasonable cost.

For example, if you have written an article that has been translated into multiple languages, and each language version is going to be read by thousands of people who will provide comments on it, that is a huge amount of feedback that would require multiple linguists to analyze any trends in an analog way. It would not be financially feasible. But NLP can analyze thousands of comments and come to a conclusion about the feedback on the article quickly and without human intervention.

There are other business problems clients solve with NLP. Sentiment analysis is a great example. When you have a lot of data — it could be thousands of comments a day, coming in from a variety of social media platforms — you need to get an overview of how people engaging with your brand feel about your product. NLP can determine if these comments are generally positive, negative, or neutral. So, the NLP technology is not just translating text, natural speech, or the written word; it is also understanding the emotion behind it and giving you that crucial data.

NLP can handle big data like nothing else. Say a company has half a million employees, and management wants to know what those employees think about the company. We can create a system that can gather feedback from all the employees, determine a consensus by looking at the language used, and then that data can be presented to management.

NLP makes it easier, less expensive, and faster for businesses to go global with local languages. For example, in India, there are 22 languages. It is not viable to have your content available in 22 languages, so you just create it in English to cover all of India. But by doing this, you are not able to reach the remote corners of the country because they do not speak English.

For example, with the advent of mobile devices, everyone can afford to have a computer in their pocket. Mobile providers in India expect people to know English or another main language. But your mother or your grandmother might now be using a mobile without main-language knowledge. So, for these situations, you need a system that understands multiple languages and NLP is the best choice: if your phone tech is intelligent enough (via NLP), then it doesn’t matter what language you speak.

Take my mother. She is very fond of baking, but she cannot type in English. So, she just speaks into her phone: “tell me how to bake a cake,” but in Hindi. The system understands what she is saying and presents the whole recipe to her in spoken Hindi and, because it is just voice, she is comfortable using the mobile technology.

Another example is Google Pay in India. Many people are not good with English, but with Google Pay, they can use their local language to make a payment on their mobile phone. As Google improved its language knowledge and increased the number of languages it caters to, the number of users has increased by 400%. This shows where major technology players, like Google, are investing a lot of resources — and are seeing significant results.

Other applications of NLP systems

Some applications of NLP might be unknown to readers, such as the fact that NLP is also a major tool for quality assurance (QA). In software development, it speeds up the QA process by identifying issues for applications that need to be international. It will help identify issues that do not show up in English or other main languages but would be a problem as your software is localized into, for example, the 22 languages in India.

For example, if I am writing a multilingual system, I need to follow standards so that my code is easily maintainable if any new language is added. Using NLP, we can write an engine to analyze the code and tell if it’s written in the standard way or not. The human variance in the process is eliminated, and the work is more international and easier to localize in the future.

A second example of NLP helping with QA involves deter-mining which words must remain in the native language when translating large blocks of text or speech. For example, if we have a website developed in English that also needs to work in Hindi, the NLP application can scan the whole web-site and determine if there are any words that should remain in English. If the term “Cambridge University” appears on the site, we know we don’t want this translated into Hindi — not even the word “University” — so the NLP software will realize it should not be translated.

There’s a certain amount of acceleration in the field — NLP and AI have been used more because of the COVID-19 pandemic. COVID-19 has sped up investment in automation of the workplace, machine learning and AI. NLP has been more widely used since parts of the world went into lockdown — specifically, the use of tools such as voice recognition. People were at home, and they wanted to get information and entertain themselves. There are always people who do not use technology as much to connect with others, and would rather meet in person, but many still cannot do so because of restrictions and barriers. For example, in the past, people would visit a doctor to get basic medical information, and now they can now do this with chatbots. Also, in the COVID climate, chatbots were used to help people track their symptoms, addressing our need to feel safe by giving us access to relevant information during this health crisis. This makes the ability of NLP to understand the way we speak naturally, in our native language, even more important.

The business of NLP

There are certain steps businesses can take to deploy NLP, but first and foremost, it depends on what the business needs to achieve. You should study your situation to come up with a problem statement. Do you need to analyze big data? Do you need customer service chatbots, or do you need something else? After doing the needs analysis, you will know what type of NLP technology to put in place. Next, you will look at how to build the technology on top of an existing solution or whether you need to create something completely customized. So, it is really analysis, development, testing, deployment, continual monitoring, and updating.

Of course, businesses may ultimately be wondering what the future holds for this kind of tech. There are two areas to address when answering this question. First, the emerging tech, and second, how current NLP applications are improving. On the emerging side, we will see NLP grow in healthcare, education, and agriculture — at the moment, these areas of NLP are still new and not being used to their full potential. Companies that can understand how to use NLP to their advantage in these industries will be early adopters of the technology, and could have a competitive advantage.

The current technology is improving because, today, Siri or Google Assistant will understand your language and will answer in that language. So, if you ask a question in Dutch, it will reply in Dutch. This is the future — it will not matter what language Siri is configured for, because if you start speaking in a language, it will reply in that language.

Additionally, NLP is becoming more realistic and accurate. Currently, when I ask Siri or Google Assistant a question, I know that I will get my answer, but it may not be totally accurate. So NLP will play a particularly important role anytime you are looking for a computer to answer your queries. If you tell it that you are not happy with the answer it has given you, it can use that as feedback and will try to supply you with an even more accurate answer next time. The technology becomes more accurate when there is more data — this is always an ongoing process.

How intelligent will AI become?

Let’s consider an example of one application of this tech. Right now, with voice recognition, when I ask it something, the application gives me the information I need. But as it becomes more intelligent and understands the context of data that I put into it, it will start to recommend things without me asking. This proactive nature is a paradigm shift in our interaction with NLP, and it will shape our lives in the future.

For example, if you order one type of sandwich frequently, regular NLP-enabled technology could understand this and prompt you the next time it thinks you might want to eat. It could also suggest other products or even better solutions. Maybe you need potato chips or a cookie to go with your meal.

The other use is in a business setting, where NLP can be used to track everything we do on our work computers or mobile phones, and it can analyze this data to see, for example, how long it takes us to reply to an email. If there are thousands of employees receiving the same email, then this becomes big data, and technology could suggest ways to structure the email, or suggest another type of communication as time and cost savings for the business.

This kind of proactive NLP is only beginning to be explored. In the future, it will be able to enhance our lives in ways that we probably cannot imagine right now.

It is true that being able to use NLP to crunch data, under-stand sentiment, comprehend, and understand any language in a voice application — and therefore allow businesses to more easily reach their customers in all markets — is a game changer.

One thing is for certain: the importance of NLP cannot be overstated. After all, AI and machine learning — the technologies that take up the tasks of humans and reduce error, increase efficiency and allow businesses to be productive around the clock — lean on the field of NLP to process natural language so these technologies learn and make more accurate and relevant predictions.

What does this mean ultimately? In theory, tech like this can help us get safely to work in self-driving cars or explore deep space in crafts that can go on unoccupied and error-free for years. From a game of chess with a computer to bomb disposal in the battlefield, the potential uses are increasing all the time.