01 | What languages did you have in your MT initiative? Also, have you added new ones since you launched the MT?

We’ve always had 11 target languages: English, French, Italian, German, Spanish, Portuguese, Simplified Chinese, Traditional Chinese, Japanese, Korean, and Turkish. And we did not add new ones — we rarely add locales at all, preferring to translate projects into all languages simultaneously. We started experimenting with MT in August 2019, after talking with the guys from EA and Supercell. At first, it was a question of competition, like, if they can do it, why can’t we? In October 2019, the Russian to English pair on our existing projects completely switched to MTPE, with the exception of marketing materials. For those, transcreation is needed rather than translation. The rest of the languages switched to MTPE in February 2020. Before that, we played with all the MT engines we could reach. Now we are experimenting with trainable engines, but the result leaves much to be desired. It’s also worth mentioning that we translate games where quality depends on the translator’s qualification; the linguist’s desire to work on the project; adequate turnaround time requirements; and the quality of the original text. All these points are primarily tied to us, the localization managers.

02 | So you compared the MT engines on the principle of time spent on post-editing? Did you track how much time did you actually save?

Yes, we compared the engines based on this principle. But we also used a “like/dislike” approach when comparing them as well, as a “yes, I do want to work with this engine” kind of metric. After all the wanderings, the team chose DeepL, mostly motivated by the fact that it had the least amount of post-editing required.

It’s worth mentioning here that we have a team of perfectionists. If the translator doesn’t like the phrase even slightly, he deletes it and writes the translation from scratch, performing human translation instead of post-editing. With DeepL, the number of segments rewritten from scratch is two to three times less than with any other MT engine that we have tried. But this does not mean that the general number of rewritten segments is small. You should not perceive MT as a solution to all your problems, or DeepL as the king of MT engines suitable for any purpose.

03 | So exactly how cool was DeepL? Was its output 5% or 50% better as compared with other MT engines?

One way to measure this is to ask the translators. I collected translators’ feedback about our current MT experience, keeping their original wording. The results are in Table 1, although they’re all talking about time savings compared to translating from scratch.

The gained increase in translation speed is difficult to measure objectively for many reasons.

Source texts have different content and complexity; fuzzy matches from the translation memory (TM) may be different. Also, things like a Mercury retrograde period or a linguist’s bad mood might change the outcome. So let’s just say, the speed gain ranged between 5% to 50% depending on these and other factors.

We actually have not tested the engines in a way that Intento does, for example. What we did was MTPE of our current games updates (10-1500 source words per iteration) and then analyzed of the results together with a team of translators and editors.

I really don’t like to complicate processes, especially if there is no clear goal behind it. In this case, as there was no ultimate goal to have a set of MT evaluation criteria and strict metrics for academic or commercial research purposes, why would we do this? So we didn’t.

I was given a simple task: to reduce the cost with minimal loss in quality, while the timing was of secondary importance. The most obvious solution seemed to cut out editing. So we had had a full translation, editing, and proofreading (TEP) cycle, with at least two translators working in each language pair. The problem was that without the editing step, the quality dropped noticeably: typos and misspellings were not corrected, and actual errors became more frequent.

Let me stop here for a moment and tell you more about how we actually measure the translation quality: there are major errors, and there are minor errors. Major ones either contain a major factual, political, or cultural error, or do not allow the player to perform the required action and correctly understand the essence of the game task. Such errors are subject to immediate correction, as they affect the monetization and general image of the project. And there are minor errors, which encompass everything that has to do with the grammar rules but will not drastically affect the gameplay or generate negative feedback from the users. These errors are being fixed in a general order.

In general, we adhere to an industry-wide rule for TEP: a maximum of one major error and three acceptable (minor) errors per 1200 words of the source text, excluding tags and placeholders — provided that the linguist made these errors without the influence of external factors, meaning they had the needed context or at least time for clarifying questions.

So here comes what I like about MTPE: an acceptable MT is easier for a translator to edit rather than to come up with another version from scratch. The post-editors get less tired and make fewer mistakes, since they only need to assess how correctly the meaning is conveyed, and then adjust the wording if necessary. In this scenario, the number of both unacceptable and acceptable errors was 50% less as compared to the scenario when a translator does a translation from scratch and without further editing.

What is also worth mentioning: we do not use MT on our new projects or non-standard tasks. We use MT only where a TM, a glossary, and a styleguide are available, and the linguists already know the project.