Technology agnostic machine translation: Part two

In part one of this article, I discussed the main approaches to machine translation (MT): statistical (SMT) and rules-based (RBMT), as well as hybrid, which combines the two. Using case  studies from my own company, LexWorks, as well as those from other technology agnostics such as PayPal, Symantec, Autodesk and Adobe, I demonstrated that for many experienced MT practitioners, it’s not a case of which engine is best, but rather which engine best suits the content and language combinations. This second article of the series provides more concrete examples of fitting the right approach to the right situation.

Outstanding MT results — everyone wants them, but what’s the best way to achieve them? Just running a text through an engine and handing it off for post-editing is one approach. But it’s not very effective, and can burn a lot of bridges with your post-editors. The ideal is to give them the kind of quality output that has them saying, “I can’t believe how little there was to correct!” For MT output that is to be published raw, as in the case of customer support content, it’s even more important to have the highest possible quality right from the get-go. Easier said than done?

No, not really. The basic path to outstanding MT results is fairly straightforward. Fred Hollowood, an independent localization and language technology consultant who directed Symantec’s MT research for nine years, summarizes the steps as follows. First, look for the best tool for the job. Second, develop skills across the toolset, and third, put the emphasis on training. We will concentrate on the first of these: identifying the best tool for the job. “Combining your expertise in MT with the most suitable engine maximizes the benefit to you, your customer and your language suppliers,” says Hollowood. “It stands to reason that you need expertise across the technology portfolio to optimize this process.”

With expertise built over nearly a decade of MT deployments, LexWorks has been able to draw some conclusions about the most suitable engine for given contexts and create some rules of thumb. How do we work with the rules of thumb in a real world production environment? To make sure that these rules of thumb hold true in a given situation, we train two to four engines at project rollout, then test to see which approach actually works best. And, yes, sometimes the results surprise us, such as when an RBMT engine outperforms a hybrid one for Japanese software!


Four engines compared

Although it’s generally accepted that there are three main MT approaches, RBMT, SMT and hybrid, for comparison purposes we usually add a fourth: a trained online Software-as-a-Service engine such as Microsoft Translator Hub. For simplicity, I call this online SMT and have included it in our rules of thumb.

Online SMT refers to a subscription-based engine available via application programming interface, which you can train on your material. The advantage of this is that these online engines are already built on rich resources of data. Few organizations have sufficient data to train their engines so fully.

As used in this context, online SMT is distinct from the free generic engines such as Bing and Google Translate, which in our experience are of limited use since they are not trained on your material. I also distinguish online SMT from other SMT engines such as Moses, since online SMT comes already trained with significant in-domain and, more importantly, out-of-domain data. This makes a big difference for certain kinds of content. If you are short of training data, an online SMT engine can significantly expand the number of language pairs you can handle.

In addition to the SMT and online SMT engines — Moses and Microsoft Translator Hub respectively — Systran (RBMT and hybrid) fills out the list of the engines covered subsequently.

The first consideration when choosing an MT engine is the content (Table 1). This is because content in a narrow domain with set terminology tends to respond best to a hybrid or RBMT engine. Why? Because you can be sure the right terminology is being employed. This includes documentation, reports, online help, user interface and similar material.

On the other hand, frequently asked question (FAQ) pages, forums and other user-generated content (UGC) is best suited to SMT, which is less likely to be thrown off track by poor spelling or grammar or colloquial language.

The patent domain is another area where SMT excels, due to the broad range of subject areas and terminology, which becomes impractical to encode as you would for RBMT. As for marketing materials, we don’t consider any MT approach suitable!

What about other considerations? Not having enough training data in a particular language pair absolutely limits your choices. Without a good quantity of data, training your own SMT engine will be out of the question; you will need an engine that is usable out of the box.

Language combination is probably the biggest determiner of MT engine performance (Table 2). Some languages, such as French, Spanish and Italian, work well in any approach. Other languages, such as Russian, Japanese and German, tend to respond better to an RBMT or hybrid engine built to accommodate their tricky morphology. Still other languages may not exist (at least not yet) in an RBMT engine, and so SMT is the only choice. Additionally, if, as previously mentioned, you have limited training data, then that limits your choices even further; in this case, online SMT may be your only suitable engine.

As another decision aid in determining the best-of-breed engine, comparing features of RBMT versus SMT performance in different areas yields the results in Table 3.


Look for the best tool for the job

These rules of thumb, as well as the conclusions you draw from your own testing and measurement, should make it easier to “look for the best tool for the job,” as Hollowood puts it. Whether our customers, internal or external, want the “fastest, or most beautiful, or most appropriately styled translations,” the broad range of solutions available “obliges us to seek out the most appropriate technology for the benefit of both our clients and our translators,” says Hollowood.

In the last ten years we have seen that under the right conditions, any of the main approaches to MT will yield workable results. In that time, MT use has grown exponentially. However, MT is still far from mainstream. One of the reasons for this may be the uncertainty as to which engine to employ under what circumstances. As I see it, our task for the next five or ten years is to grow our competence in all MT approaches so that we are sure to deliver whatever our customers want — whether, as Hollowood says, they want the fastest translation, the most beautiful translation or simply one in a style that fits the purpose.