At the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Welocalize and Duke University unveiled LangMark, a multilingual dataset designed to advance research in automatic post-editing (APE) of neural machine translation (NMT). With more than 200,000 human-annotated samples, LangMark is one of the largest resources of its kind and aims to become a new benchmark for evaluating how effectively large language models (LLMs) can refine machine-translated text.

LangMark includes 206,983 triplets — each composed of a source English segment, an NMT output, and a human post-edited version. The dataset spans seven languages: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish. Professional linguists with subject-matter expertise performed the annotations to ensure accuracy and domain relevance.

Much of the content is drawn from marketing documents, making the dataset highly representative of real-world business communication. According to the authors, this context reflects practical use cases where terminology, tone, and audience appropriateness matter as much as literal correctness.

Why Post-Editing Still Matters

Machine translation (MT) has become increasingly accurate with the rise of large transformer-based models and the integration of LLMs. Yet errors persist. Translations may be technically correct but contextually wrong — for example, rendering “our people” in Spanish as “our nation” instead of “our staff.” APE addresses this gap by correcting MT output to professional standards without requiring full human retranslation.

Previous datasets, such as SubEdits or WMT APE collections, were limited in scale or language diversity. LangMark hopes to distinguish itself as a multilingual, large-scale, human-annotated corpus for both industry and academia.

Benchmarking LLMs

Using LangMark, the researchers tested commercial systems and LLMs under a consistent experimental setup. Results showed that GPT-4o achieved the strongest improvements, particularly in Japanese and Russian. Open-source models like Qwen2.5-72B also delivered competitive results, sometimes outperforming larger proprietary systems.

However, most models struggled with a fundamental challenge: deciding when an edit is necessary. Conservative models, which made fewer changes, often outperformed aggressive ones that introduced unnecessary edits. This highlights a need for evaluation metrics that account not just for accuracy but also for editing behavior.

Implications for the Industry

For the localization sector, LangMark offers a way to evaluate whether systems can meet the nuanced demands of professional translation. By releasing the dataset publicly, Welocalize and Duke University want to encourage wider collaboration between academic researchers and language service providers.

The dataset also emphasizes the growing role of LLMs in translation workflows. While models can already rival or surpass MT in some areas, human judgment remains the gold standard. LangMark helps measure how close machines are to replicating that judgment — and how far they still have to go.