A human-parity claim in translation fails under document-level evaluation

Assembled from 33 blocks · 1 evidence · 28 related

1. Story
2. Läubli, Sennrich and Volk re-examined a claim that machine translation had reached parity with professional human translation on the Chinese–English news task, and found the claim depended on the evaluation unit. Raters comparing isolated sentences saw little difference; the same raters comparing whole documents preferred human translation clearly, because the failures that separate the two are failures of cohesion, reference and consistency across sentences — invisible by construction to a sentence-level protocol. The paper is a template for the whole Journey: the system did not change between the two evaluations, and the finding reversed.
3. Knowledge
4. Construct validity in AI evaluation
5. Machine translation
6. The difficulty of evaluating machine learning systems
7. A human-parity claim in translation fails under document-level evaluation
8. Connections
9. Machine translation
10. The difficulty of evaluating machine learning systems
11. Construct validity in AI evaluation
12. Labour-market effects of machine learning
13. The transformer architecture
14. The transformer is presented at NeurIPS
15. A human-parity claim in translation fails under document-level evaluation
16. The second International AI Safety Report reports mixed labour findings
17. CASP — Critical Assessment of Structure Prediction
18. Construct validity in AI evaluation
19. Benchmark saturation
20. METR
21. European Union Artificial Intelligence Act
22. A human-parity claim in translation fails under document-level evaluation
23. A randomised trial finds experienced developers slower with AI assistance
24. An audit finds the most-cited preference leaderboard is optimised rather than merely measured
25. A systematic review of 445 benchmarks reports pervasive validity failures
26. The Commission proposes deferring the high-risk obligations
27. The AI Index records both fast benchmark movement and doubts about benchmarks
28. The difficulty of evaluating machine learning systems
29. A human-parity claim in translation fails under document-level evaluation
30. An audit finds the most-cited preference leaderboard is optimised rather than merely measured
31. A systematic review of 445 benchmarks reports pervasive validity failures
32. Evidence
33. Testing a human-parity claim on the WMT Chinese–English news task with alternative evaluation protocols, human raters assessing adequacy and fluency showed a stronger preference for human over machine translation when evaluating whole documents than when evaluating isolated sentences, indicating that errors decisive for quality are frequently invisible at sentence level. V55 verification basis: the paper was not fetched; search results carried the title, the ACL Anthology identifier D18-1512, the EMNLP 2018 Brussels venue and the substance of the finding and attributed them to this paper.