(Meta-) Evaluation of Machine Translation
Chris Callison-Burch,Cameron Shaw Fordyce,Philipp Koehn,Christof Monz,Josh Schroeder +4 more
- 23 Jun 2007
- pp 136-158
TL;DR: An extensive human evaluation was carried out not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process, revealing surprising facts about the most commonly used methodologies.
read more
Abstract: This paper evaluates the translation quality of machine translation systems for 8 language pairs: translating French, German, Spanish, and Czech to English and back. We carried out an extensive human evaluation which allowed us not only to rank the different MT systems, but also to perform higher-level analysis of the evaluation process. We measured timing and intra- and inter-annotator agreement for three types of subjective evaluation. We measured the correlation of automatic evaluation metrics with human judgments. This meta-evaluation reveals surprising facts about the most commonly used methodologies.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Improving evaluation and optimization of MT systems against MEANT
Chi-kiu Lo,Philipp Dowling,Dekai Wu +2 more
- 01 Sep 2015
TL;DR: It is shown that, consistent with MEANTtuned systems that translate into Chinese, MEANT-tuned MT systems that translates into English also outperforms BLEUtuned systems across commonly used MT evaluation metrics, even in BLEU.
Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery
TL;DR: In this article , a model-agnostic method for training and inference using a conditional independence classifier is proposed, which is trained by a constrained self-training method, coined ConSTrain, to overcome data sparsity.
Automatic Metrics for Machine Translation Evaluation and Minority Languages
Daša Munková,Michal Munk +1 more
- 01 Jan 2016
TL;DR: The results of the reliability analysis showed that these automatic metrics for MT evaluation are reliable and valid, whereby the validity and reliability were verified for one translation direction: from the minority language into English.
How to evaluate machine translation: A review of automated and human metrics
TL;DR: The most up-to-date, influential automated, semiautomated and human metrics used to evaluate the quality of machine translation (MT) output are presented and provides the necessary background for MT evaluation projects.
Proceedings Article
The Authenticity Gap in Human Evaluation
Kawin Ethayarajh,Dan Jurafsky +1 more
- 24 May 2022
TL;DR: This paper proposed a new human evaluation protocol called system-level probabilistic assessment (SPA), which can recover the ordering of GPT-3 models by size, with statistically significant results.
References
The measurement of observer agreement for categorical data
J. R. Landis,Gary G. Koch +1 more
TL;DR: A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
76.1K
Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni,Salim Roukos,Todd Ward,Wei-Jing Zhu +3 more
- 06 Jul 2002
TL;DR: This paper proposed a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run.
•Proceedings Article
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments
Satanjeev Banerjee,Alon Lavie +1 more
- 01 Jun 2005
TL;DR: METEOR is described, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machineproduced translation and human-produced reference translations and can be easily extended to include more advanced matching strategies.
A systematic comparison of various statistical alignment models
Franz Josef Och,Hermann Ney +1 more
TL;DR: An important result is that refined alignment models with a first-order dependence and a fertility model yield significantly better results than simple heuristic models.
Statistical phrase-based translation
Philipp Koehn,Franz Josef Och,Daniel Marcu +2 more
- 27 May 2003
TL;DR: The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.