(Meta-) Evaluation of Machine Translation
Chris Callison-Burch,Cameron Shaw Fordyce,Philipp Koehn,Christof Monz,Josh Schroeder +4 more
- 23 Jun 2007
- pp 136-158
TL;DR: An extensive human evaluation was carried out not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process, revealing surprising facts about the most commonly used methodologies.
read more
Abstract: This paper evaluates the translation quality of machine translation systems for 8 language pairs: translating French, German, Spanish, and Czech to English and back. We carried out an extensive human evaluation which allowed us not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process. We measured timing and intra- and inter-annotator agreement for three types of subjective evaluation. We measured the correlation of automatic evaluation metrics with human judgments. This meta-evaluation reveals surprising facts about the most commonly used methodologies.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Further Meta-Evaluation of Machine Translation
Chris Callison-Burch,Cameron Shaw Fordyce,Philipp Koehn,Christof Monz,Josh Schroeder +4 more
- 19 Jun 2008
TL;DR: This paper analyzes the translation quality of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish and uses the human judgments of the systems to analyze automatic evaluation metrics for translation quality.
Reference bias in monolingual machine translation evaluation
Marina Fomicheva,Lucia Specia +1 more
- 01 Jan 2016
TL;DR: This paper shows that this practice has a serious issue - annotators are strongly biased by the reference translation provided, and this can have a negative impact on the assessment of MT quality.
Discourse Structure in Machine Translation Evaluation
TL;DR: This article first design discourse-aware similarity measures, which use all-subtree kernels to compare discourse parse trees in accordance with the Rhetorical Structure Theory (RST), and shows that a simple linear combination with these measures can help improve various existing machine translation evaluation metrics regarding correlation with human judgments.
A Smorgasbord of Features for Automatic MT Evaluation
Jesús Giménez,Lluís Màrquez +1 more
- 19 Jun 2008
TL;DR: This document describes the approach by the NLP Group at the Technical University of Catalonia (UPC-LSI) for the shared task on Automatic Evaluation of Machine Translation at the ACL 2008 Third SMT Workshop.
Human and Automatic Evaluation of English to Hindi Machine Translation Systems
Nisheeth Joshi,Hemant Darbari,Iti Mathur +2 more
- 01 Jan 2012
TL;DR: This work presents the MT evaluation results of some of the machine translators available online for English-Hindi machine translation, measured on automatic evaluation metrics and human subjectivity measures.
References
The measurement of observer agreement for categorical data
J. R. Landis,Gary G. Koch +1 more
TL;DR: A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
76.1K
Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni,Salim Roukos,Todd Ward,Wei-Jing Zhu +3 more
- 06 Jul 2002
TL;DR: This paper proposed a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run.
•Proceedings Article
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments
Satanjeev Banerjee,Alon Lavie +1 more
- 01 Jun 2005
TL;DR: METEOR is described, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machineproduced translation and human-produced reference translations and can be easily extended to include more advanced matching strategies.
A systematic comparison of various statistical alignment models
Franz Josef Och,Hermann Ney +1 more
TL;DR: An important result is that refined alignment models with a first-order dependence and a fertility model yield significantly better results than simple heuristic models.
Statistical phrase-based translation
Philipp Koehn,Franz Josef Och,Daniel Marcu +2 more
- 27 May 2003
TL;DR: The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.