Open Access
Evaluating Evaluation Measures
Ines Rehbein,Josef van Genabith +1 more
- 23 May 2007
- pp 372-379
TL;DR: An analysis of specic error types indicates that the dependency-based evaluation is most appropriate to reect parse quality, and shows that PARSEVAL should not be used to compare parser performance for parsers trained on treebanks with different annotation schemes.
read more
Abstract: This paper presents a thorough examination of the validity of three evaluation measures on parser output. We assess parser performance of an unlexicalised probabilistic parser trained on two German treebanks with different annotation schemes and evaluate parsing results using the PARSEVAL metric, the Leaf-Ancestor metric and a dependency-based evaluation. We reject the claim that the T¤ uBa-D/Z annotation scheme is more adequate then the TIGER scheme for PCFG parsing and show that PARSEVAL should not be used to compare parser performance for parsers trained on treebanks with different annotation schemes. An analysis of specic error types indicates that the dependency-based evaluation is most appropriate to reect parse quality.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Proceedings Article
Overview of the SPMRL 2013 Shared Task: A Cross-Framework Evaluation of Parsing Morphologically Rich Languages
Djamé Seddah,Reut Tsarfaty,Sandra Kübler,Marie Candito,Jinho D. Choi,Richárd Farkas,Jennifer Foster,Iakes Goenaga,Koldo Gojenola Galletebeitia,Yoav Goldberg,Spence Green,Nizar Habash,Marco Kuhlmann,Wolfgang Maier,Joakim Nivre,Adam Przepiórkowski,Ryan M. Roth,Wolfgang Seeker,Yannick Versley,Veronika Vincze,Marcin Woliński,Alina Wróblewska,Éric Villemonte de la Clergerie +22 more
- 18 Oct 2013
TL;DR: This paper presents and analyzes parsing results obtained by the task participants, and provides an analysis and comparison of the parsers across languages and frameworks, reported for gold input as well as more realistic parsing scenarios.
201
Effects of data set features on the performances of classification algorithms
Ohbyung Kwon,Jae Mun Sim +1 more
TL;DR: This research experimentally examines how data set characteristics affect algorithm performance, both in terms of accuracy and in elapsed time, and uses a multiple regression method to evaluate the causality between dataSet characteristics as independent variables, and performance metrics as dependent variables.
171
Discontinuous Incremental Shift-reduce Parsing
Wolfgang Maier
- 01 Jul 2015
TL;DR: This work presents an extension to incremental shift-reduce parsing that handles discontinuous constituents, using a linear classifier and beam search, and achieves very high parsing speeds and accurate results.
•Proceedings Article
Information Retrieval Meta-Evaluation: Challenges and Opportunities in the Music Domain.
Julián Urbano
- 01 Jan 2011
TL;DR: A survey of past meta-evaluation work in the context of Text Information Retrieval argues that the music community still needs to address various issues concerning the evaluation of music systems and the IR cycle, pointing out directions for further research and proposals in this line.
•Proceedings Article
Direct Parsing of Discontinuous Constituents in German
Wolfgang Maier
- 05 Jun 2010
TL;DR: This paper uses a parser for Probabilistic Linear Context-Free Rewriting Systems (PLCFRS), a formalism with high expressivity, to directly parse the German NeGra and TIGER treebanks, and shows that an output quality can be achieved which is comparable to the output quality of PCFG-based systems.
29
References
Statistical learning theory
Vladimir Vapnik
- 01 Jan 1998
TL;DR: Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
30.4K
•Book
A Probabilistic Theory of Pattern Recognition
Luc Devroye,László Györfi,Gábor Lugosi +2 more
- 01 Jan 1996
TL;DR: The Bayes Error and Vapnik-Chervonenkis theory are applied as guide for empirical classifier selection on the basis of explicit specification and explicit enforcement of the maximum likelihood principle.