Evaluating structural pattern recognition for handwritten math via primitive label graphs
Richard Zanibbi,Harold Mouchère,Christian Viard-Gaudin +2 more
- 04 Feb 2013
- Vol. 8658, pp 865817
TL;DR: This work defines new metrics obtained by Hamming distances over label graphs, which allow classification, segmentation and parsing errors to be characterized separately, or using a single measure, at the primitive level.
read more
Abstract: Currently, structural pattern recognizer evaluations compare graphs of detected structure to target structures(i.e. ground truth) using recognition rates, recall and precision for object segmentation, classi cation andrelationships. In document recognition, these target objects (e.g. symbols) are frequently comprised of multipleprimitives (e.g. connected components, or strokes for online handwritten data), but current metrics do notcharacterize errors at the primitive level, from which object-level structure is obtained. Primitive label graphsare directed graphs de ned over primitives and primitive pairs. We de ne new metrics obtained by Hammingdistances over label graphs, which allow classi cation, segmentation and parsing errors to be characterizedseparately, or using a single measure. Recall and precision for detected objects may also be computed directlyfrom label graphs. We illustrate the new metrics by comparing a new primitive-level evaluation to the symbol-level evaluation performed for the CROHME 2012 handwritten math recognition competition. A Python-basedset of utilities for evaluating, visualizing and translating label graphs is publicly available.Keywords: Evaluation, Performance Metrics, Structural Pattern Recognition, Math Recognition, GraphicsRecognition
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Table 1: CROHME Part III Dataset Statistics 
Figure 4: Undirected Structure for a Hand-Drawn Chemical Diagram and Flowchart. (b) represents an iterative process with a flowchart diagram. (c) identifies which ends of detected arrows connect to a box in (b) using “tail” or “head”, and which primitives “label” boxes 
Table 2: CROHME 2012 Part III Metrics (%) for Submitted Systems.4 Participants are sorted by expression rate (EXP) 
Figure 1: “2+2” Written Using Four Strokes. (a)-(c) are primitive label graphs, and (d) shows trees over objects (symbols) whose structure are equivalent to (b) and (c). Strokes are named in writing order as s1, s2, s3 and s4 with the vertical and horizontal strokes for the ’+’ indicated by (ver.) and (hor.). Dashed edges indicate strokes merged into a symbol. Nodes are labeled with the class of the symbol associated with a stroke. Remaining edges represent relationships: R for adjacent-at-right, and Arg1 and Arg2 for operator arguments 
Table 3: CROHME part III Error Metrics for Label Graph Representations. Participant systems are ranked by average percentage error in label graphs (∆Bn). Shown are sums over all expressions for each Hamming distance (∆B, ∆C, ∆L and ∆S), along with distributions for ∆Bn and ∆E distributions shown by means (µ) and standard deviations (σ). The number of expressions (MathML) with formatting errors (*Expr.) and the number of concerned symbols (*Symbs.) in system outputs are provided. Systems IV and I and II and V switch positions relative to Table2 
Figure 3: Different Interpretations for “2+2” in Fig. 1. In (a) we have recognized “2+” and compare this with Figure 1(b). The missing stroke in the interpretation is treated as unrelated to strokes present in the expression, with an undefined label and undefined relationships with other strokes. In (b) the ’+’ is split, with a superscript between strokes s2 and s4. In (c) the exponent “12” is represented by the edge labeled “Expt.”
Citations
ICFHR 2014 Competition on Recognition of On-line Handwritten Mathematical Expressions (CROHME 2014)
Harold Mouchère,Christian Viard-Gaudin,Richard Zanibbi,Utpal Garain +3 more
- 01 Sep 2014
TL;DR: The outcome of the latest edition of the CROHME competition, dedicated to on-line handwritten mathematical expression recognition, features two new tasks, one dedicated to isolated symbol recognition including a reject option for invalid symbol hypotheses, and the second concerns recognizing expressions that contain matrices.
An integrated grammar-based approach for mathematical expression recognition
TL;DR: The statistical framework of a model based on two-dimensional grammars and its associated parsing algorithm is defined and a system that implements this approach is developed and results on the large public dataset of the CROHME international competition are reported.
114
A retrospective study on handwritten mathematical symbols and expressions: Classification and recognition
Sakshi,Vinay Kukreja +1 more
TL;DR: In this paper, the authors performed an extensive state-of-the-art on the techniques and methods used for recognizing and classifying HMSE, and brought out all significant findings in sub-processes, representation models, algorithms, tools, datasets, and comparative analysis of the accuracy of the recognition models.
75
SRD: A Tree Structure Based Decoder for Online Handwritten Mathematical Expression Recognition
TL;DR: This paper proposes a novel sequential relation decoder (SRD) that aims to decode expressions into tree structures for online handwritten mathematical expression recognition and demonstrates how the proposed SRD outperforms state-of-the-art string decoders through a set of experiments on CROHME database.
38
CoMER: Modeling Coverage for Transformer-Based Handwritten Mathematical Expression Recognition
TL;DR: CoMER as discussed by the authors proposes a novel attention refinement module to refine the attention weights with past alignment information without hurting its parallelism, which improves the ExpRate by 0.61%/2.09%/1.59% compared to the current state-of-the-art model and reaches 59.33%/59.81%/62.97% on the CROHME 2014/2016/2019 test sets.
28
References
A survey of table recognition: Models, observations, transformations, and inferences
TL;DR: This presentation clarifies both the decisions made by a table recognizer and the assumptions and inferencing techniques that underlie these decisions.
Recognition and retrieval of mathematical expressions
TL;DR: This paper surveys the state of the art in recognition and retrieval of mathematical expressions, organized around four key problems in math retrieval (query construction, normalization, indexing, and relevance feedback), and four key problem in math recognition (detecting expressions, detecting and classifying symbols, analyzing symbol layout, and constructing a representation of meaning).
Error detection, error correction and performance evaluation in on-line mathematical expression recognition
Kam Fai Chan,Dit-Yan Yeung +1 more
TL;DR: This paper incorporates an error detection and correction mechanism into a parser developed previously by us based on de"nite clause grammar (DCG), and proposes a performance evaluation scheme which can be used to demonstrate the e!ectiveness of both the symbol recognition and structural analysis stages by a single measure.
104
Evaluating the performance of table processing algorithms
TL;DR: An intuitive, easy-to-implement evaluation schemes for the related problems of table detection and table structure recognition are introduced and a new paradigm, “graph probing,” is described for comparing the results returned by the recognition system and the representation created during ground-truthing.
97
CROHME2011: Competition on Recognition of Online Handwritten Mathematical Expressions
Harold Mouchère,Christian Viard-Gaudin,Dae Hwan Kim,Jin Hyung Kim,Utpal Garain +4 more
- 18 Sep 2011
TL;DR: The best expression level recognition accuracy shown by the competing systems is 19.83% whereas a baseline system developed by one of the organizing groups reports an accuracy 22.41% on the same data set.