Edwin Starr’s 1970 Number 1 hit “War” repeatedly asks the question, “War, what is it good for?” and the response is always “nothing.” You could ask the same question of many things — garlic crushers, Roombas, automatic evaluation metrics for machine translation (MT) — apply the same answer, and have the majority of people agree with you.

Table 1

The fundamental flaws

The single biggest flaw with these metrics is the concept that MT output is compared to and judged based on a “correct” reference translation. As I’m sure most readers will agree, there’s no such thing as a single correct translation. There are often very many ways to adequately translate a given source text. Therefore, even if it is correct, if the MT differs from the reference translation, it will be penalized. As we’ll see shortly, some metrics have tried to account for this in their algorithms with varying degrees of success. Another suggested workaround for this issue is to have multiple reference translations, but that is not very practical in most scenarios and is rarely used.

The next biggest flaw with these metrics is that the scores they produce have very little meaning in absolute terms. So your Russian to English MT engine has a BLEU score of 45 — what does that mean? Absolutely nothing, literally. Scores might be informative for a relative comparison, like if another Russian to English MT engine tested on the exact same documents had a BLEU score of 41, we could say with some degree of certainty that the first engine is probably better than the second one. But we cannot say if either of them are actually any good. Beware anyone trying to tell (sell) you otherwise!

Scores generally aren’t comparable across different languages or on different test documents either, so their application is quite narrow. Even when you try to use common sense, they can confound. For example, you might think the higher the score the better, and you’d be correct. However, experience tells us that if a score is too high, the more suspicious it is, and maybe there was some mistake made during testing!

These flaws are openly acknowledged by developers and users, and there has been a concerted effort to refine and improve them over the past 15 years. This has resulted in a large number of different metrics in the field with varying degrees of adoption.

What are all these metrics and what is the difference between them? Table 1 offers a non-exhaustive (but not far from it) attempt to explain; questionable acronyms included.

Other metrics that exist but are not detailed here include GTM, AMBER, BEER, GLEU, MP4IBM1, YiSi, MTeR-ATER, and PER.

The present and future of automated metrics

Despite their known flaws, these metrics are still used extensively. Particularly the BLEU score, which was even acknowledged by its developers to be limited. The reason for this is really a case of sticking with the familiar in the absence of a silver bullet alternative. Not to be hypocritical, I used them back in my own research days and we still use them at RWS Iconic. Typically, practitioners will tend not to rely on a single metric, but rather calculate scores from a variety of different metrics, as a kind of sanity check. However, important decisions, particularly when it comes to buyers choosing whether to use MT (and/or which MT), will additionally rely on human assessments.

Fortunately, all is not lost when it comes to automated metrics. There is a lot of R&D effort spent on trying to create new and better metrics. As part of the annual WMT Conference, there has been a shared task related to new metric development running since 2008, with the next edition in November this year. This task focuses on evaluating the evaluation metrics themselves, by seeing how well they correlate with human assessments.

The oracle of evaluation, however, is quality estimation. A perpetual hot topic in MT, quality estimation aims to predict the accuracy of machine translations without needing to com-pare them to reference translations. Not only is this very useful as a product feature, it would be ideal as an absolute evaluation metric.

This will no doubt be an upcoming topic in these pages, and research in this area is ongoing to make this a reality. Until then, use automated metrics in the wild with caution! Correctly interpreting what they are telling you takes a dollop of experience along with a serving of intuition. Without this, a little knowledge can be a dangerous thing.