How to Analyze Game Localization Quality
Using Reviews

interview BY Dmitry Antonov and Tatyana Veryasova

It’s pretty obvious that localization affects player reviews. A bad translation can lead to bad reviews, and great work can, of course, win high praise from gamers. In turn, all of this feedback can be really useful. Reviews are important information sources to assess and improve the quality for any game localization professional.

While we do have an internal localization quality-control system that we use to digitize the results of our linguists’ work and objectively evaluate them, this isn’t always enough. It’s possible to make a translation that is perfect from a technical point of view but leaves players unsatisfied. This is where working with review analysis comes into play. And the good news is that any operation can conduct review analysis to incorporate valuable user feedback into localization processes.

First, let’s break down what exactly localization quality is and why it should be evaluated. Projects are localized in order to ensure players get more enjoyment from a game and developers get more profit by capturing new markets and increasing lifetime value. This means that reviews from both pleased and angry players can be used as the basis for evaluating the quality of our work.

Advertisement

Why Are Reviews So Important?

Reviews are more than just words that a developer reads in the store comment section to revel in praise (or get upset about criticism). First and foremost, they affect the game’s rating, which in turn impacts the game’s position in the app store search results. If you look at the ratings of games listed first in the search results, they’re always above average.

According to data from statistical services like Asodesk, games in the top five rarely have a rating lower than 4.4 stars on the Apple App Store. In Google Play, the star rating of leading apps is slightly lower, but still none of the most successful apps has a rating lower than 3.6.

If you can identify trends in negative reviews that develop over time, you can work to remove the cause of users’ dissatisfaction, which can then significantly boost the game’s rating. That means a higher position in the search results or even getting featured by the platform.

Collection and Analysis of Reviews

So what exactly can we do with the reviews? To set up the system, we need an algorithm of actions that lays out the stages and indicates the final result of the work.

At our company, we analyze game reviews and extract useful feedback in several stages.

  • First, we download the reviews from open sources like Google Play, the Apple App Store, Steam, or other distribution platforms. To do this, we wrote a parser that collects data from open sources based on the given parameters.
  • Next, the results are combined into a single document and sorted. We’re interested in reviews involving localization, so only segments that contain phrases like “localization,” “translation,” and “language” are taken for further analysis. Every language has its own set of keywords. We include slang expressions, as well as common typos of words, to cover all the bases.
  • The collected reviews are then translated into English using machine translation. It doesn’t provide a perfect translation, but it’s sufficient for this task.
  • Finally, a “sentiment” analysis is done to find out what players wanted to say with their reviews. They might have been satisfied with the localization, upset over a poor translation, or dreaming of playing the game in their own native languages.

The analysis results in a document containing relevant localization reviews as well as their translation and rating. The project team and client can then use the data however it’s needed.

What to Do with the Results

Now, we have a selection of relevant localization reviews sorted into various categories. This is not the final result. We want to use this data to determine a plan of action. We use the data to perform the following tasks:

  • Quality control
  • Creating teams
  • Locating and fixing errors
  • Improving our own work processes
  • Support for completed projects

What we do with the reviews depends on the category they fall under.

A Positive Review

First of all, reviews like this make us happy.

This is a very interactive and realistic game. The translation was very good.

Second of all, we note that the translation team working on this project (in this case, translating into Brazilian Portuguese) is skilled at translating games of this type. We will invite those translators back to work on similar projects in the future.

A Negative Review

Negative reviews are classified into different groups. Some reviews are overly emotional. The user might have been dissatisfied with the overall game (or individual mechanics), so they target everything in their review, including the translation, even if it’s not necessarily a bad translation.

I’m giving it a 4 and not a 5 because the combat and the Russian translation aren’t very good. But at least they’re in the game. Cool graphics, cool story, and cool character creation. The game is good but kinda boring.

This does not mean it’s a poor-quality localization, but we want to keep it in mind. The next time reviews are collected, we will need to do an especially thorough analysis of player opinion on localization into the mentioned language. For our current projects, we conduct a review analysis every three months. This is one of the ways to keep our finger on the pulse, perform quality control, and make localization that players will like, and that will work to benefit the developer.

Advertisement

It’s also possible to receive more than one negative localization review. If a clear trend emerges — which we can identify by conducting regular review analyses — then it’s time to start figuring out what the issue is.

In a perfect world, we would ask the developer to get in touch with players to share specific examples of in-game errors (preferably with screenshots), so we could validate the errors and make adjustments to the localization. In reality, the developer has several more important things on their plate. Instead, we start by trying to locate the problem on our own. For example, we can send the game for localization quality-assurance testing, during which it could easily turn out that everything’s properly translated but just doesn’t fit in the fields or line breaks aren’t formatted properly.

On occasion (though not as often as we’d like), the player describes the mistake in the review on their own. This makes our work easier and takes a bit of the edge off the sad fact that there is indeed a mistake.

I play in Japanese, but sometimes the text shows up in Hangul (the Korean alphabet), and I can’t read it. There are a lot of incorrect translations of the things I’m looking for and in other parts of the game too. The translation of “accessories” is too abstract, and the translation of “flask” is incorrect. I really don’t know what “pin,” “glare,” “gloves,” and “gummy fish” are. The common name “calla lily” should be used for “calla lilies.” It’s good that some things were fixed in the update, but “Lily Cross” turned into “Iris Cross,” and now it makes no sense.

If there are actual errors in the translation, we fix them and send the client an updated translation. If the problem isn’t in the quality of the translation, then we send an email explaining the current problems and suggesting ways to fix them. For example, players might be dissatisfied with the level of formality or unhappy being addressed as the wrong gender.

I think this is a good game. I met great people, although the German translation is a little off, as if it was made using Google Translate. And I was always addressed as “he” in the game, although I’m actually female.

This isn’t a translation mistake, but eliminating details that annoy gamers will reward you with a more loyal community and significantly boost players’ engagement.

Requests for Translation

It’s common to see reviews like this:

The game needs to add support for Turkish. It’s not fun playing without it.

Does this mean that the game needs to be translated into Turkish or any other language that can be mentioned in the review? Not necessarily. First, you should evaluate whether the residents of a target region are willing to pay. To help you do this, you can use reports from analytical services and agencies. Also consider their current income, the number of downloads, and the average purchase amount. Still, requests like these are definitely one more reason to consider adding a new language.

It’s worth checking the number of such requests before deciding whether to expand into a new market. For example, if a developer has a profitable game that has been translated into English, French, Italian, German, Spanish, Chinese, Japanese, and Korean, they’ll look for growth opportunities. One such opportunity is expanding into new language markets, and aside from metrics, the presence of requests for localization is a signal to choose that market in particular.

Tatyana Veryasova is a marketing manager at Allcorrect, a content maker, and a fan of great texts.
Dmitry Antonov is an account manager at Allcorrect and has worked in localization for more than 11 years.

Advertisement

RELATED ARTICLES

WEEKLY DIGEST

Subscribe to stay updated between magazine issues.