Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy
Jonathan Krause,Varun Gulshan,Ehsan Rahimy,Peter Karth,Kasumi Widner,Greg S. Corrado,Lily Peng,Dale R. Webster +7 more
TL;DR: Adjudication reduces the errors in DR grading by using a small number of adjudicated consensus grades as a tuning dataset and higher-resolution images as input, and to train an improved automated algorithm for DR grading.
read more
About: This article is published in Ophthalmology. The article was published on 12 Mar 2018. and is currently open access.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Table 4. Comparison of ophthalmologist grades versus adjudicated grades from retina specialists on the validation dataset. Confusion matrix for diabetic retinopathy and DME between the grade determined by majority decision of the ophthalmologists and the adjudicated consensus of retinal specialists. 
Table 5. Agreement between ophthalmologists’ grades with the adjudicated reference standard on the validation dataset. Sensitivity and specificity metrics are for moderate or worse DR and referable DME for each grader. Agreement between the adjudicated grade and the 5-point scale is also measured by the quadratic-weighted kappa. 
Table 3. Agreement between each retina specialist and the adjudicated reference standard on the validation dataset. Retina specialists correspond to those who contributed to the final adjudicated reference standard. Sensitivity and specificity metrics reported are for moderate or worse DR. Agreement between the preadjudication 5-point DR grade and the final adjudicated grade is also measured by the quadratic-weighted kappa. 
Table 2. Comparison of retinal specialist grades before and after adjudication on the validation dataset. Confusion matrix for diabetic retinopathy between the grade determined by majority decision and adjudicated consensus. 
Fig. 1. Grader agreement based on the adjudicated consensus grade for referable diabetic retinopathy (DR) and diabetic macular edema (DME). Independent grading of all 3 retinal specialists and all 3 ophthalmologists are included in this analysis. 
Fig. 2. Image resolution input to model versus area under the curve (AUC) for mild and above DR. Left: Using majority decision of retinal specialists as the reference standard. Right: Using the adjudicated consensus grade of retinal specialists as a reference standard. Shaded areas represent a 95% confidence interval as measured via bootstrapping.
Citations
An interpretable multiple-instance approach for the detection of referable diabetic retinopathy in fundus images.
TL;DR: In this article, a machine learning system was proposed for the detection of referable diabetic retinopathy in fundus images, which is based on the paradigm of multiple-instance learning.
Multimodal image encoding pre-training for diabetic retinopathy grading
TL;DR: In this article , a self-supervised pre-training approach is proposed to learn the common characteristics between modalities as well as the characteristics that are exclusive to the input modality.
24
Predicting optical coherence tomography-derived diabetic macular edema grades from fundus photographs using deep learning
Avinash V. Varadarajan,Pinal Bavishi,Paisan Raumviboonsuk,Peranut Chotcomwongse,Subhashini Venugopalan,Arunachalam Narayanaswamy,Jorge Cuadros,Kuniyoshi Kanai,George H. Bresnick,Mongkol Tadarati,Sukhum Silpa-archa,Jirawut Limwattanayingyong,Variya Nganthavee,Joseph R. Ledsam,Pearse A. Keane,Greg S. Corrado,Lily Peng,Dale R. Webster +17 more
TL;DR: In this article, a deep learning model was used to predict center-involved diabetic macular edema (ci-DME) using color fundus photographs and achieved an ROC-AUC of 0.89 (95% CI: 0.87-0.91).
Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Mike Schaekermann,Carrie J. Cai,Abigail E. Huang,Rory Sayres +3 more
- 21 Apr 2020
TL;DR: This work suggests that image adjudication may provide benefits beyond developing trusted consensus labels, and that exposure to specialist discussions can be an effective training intervention for medical diagnosis.
23
References
Deep learning
TL;DR: Deep learning is making major advances in solving problems that have resisted the best attempts of the artificial intelligence community for many years, and will have many more successes in the near future because it requires very little engineering by hand and can easily take advantage of increases in the amount of available computation and data.
67K
Gradient-based learning applied to document recognition
Yann LeCun,Léon Bottou,Léon Bottou,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio,Patrick Haffner +6 more
- 01 Jan 1998
TL;DR: In this article, a graph transformer network (GTN) is proposed for handwritten character recognition, which can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters.
53.5K
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky,Jia Deng,Hao Su,Jonathan Krause,Sanjeev Satheesh,Sean Ma,Zhiheng Huang,Andrej Karpathy,Aditya Khosla,Michael S. Bernstein,Alexander C. Berg,Li Fei-Fei +11 more
TL;DR: The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) as mentioned in this paper is a benchmark in object category classification and detection on hundreds of object categories and millions of images, which has been run annually from 2010 to present, attracting participation from more than fifty institutions.
A Coefficient of agreement for nominal Scales
TL;DR: In this article, the authors present a procedure for having two or more judges independently categorize a sample of units and determine the degree, significance, and significance of the units. But they do not discuss the extent to which these judgments are reproducible, i.e., reliable.
Gradient-based learning applied to document recognition
Yann LeCun,Léon Bottou,Léon Bottou,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio,Patrick Haffner,Patrick Haffner +7 more
- 01 Jan 2001
TL;DR: This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
32.7K
Related Papers (5)
Daniel Shu Wei Ting,Daniel Shu Wei Ting,Carol Y. Cheung,Carol Y. Cheung,Gilbert Lim,Gavin Tan,Gavin Tan,Nguyen Duc Quang,Alfred Tau Liang Gan,Haslina Hamzah,Renata Garcia-Franco,Ian Yew San Yeo,Ian Yew San Yeo,Shu Yen Lee,Shu Yen Lee,Edmund Yick Mun Wong,Edmund Yick Mun Wong,Charumathi Sabanayagam,Charumathi Sabanayagam,Mani Baskaran,Mani Baskaran,Farah Nur Ilyana Mohd Ibrahim,Ngiap Chuan Tan,Ngiap Chuan Tan,Eric A. Finkelstein,Ecosse L. Lamoureux,Ecosse L. Lamoureux,Ian Y. H. Wong,Neil M. Bressler,Sobha Sivaprasad,Rohit Varma,Jost B. Jonas,Mingguang He,Ching-Yu Cheng,Ching-Yu Cheng,Gemmy Cheung,Gemmy Cheung,Tin Aung,Tin Aung,Wynne Hsu,Mong Li Lee,Tien Yin Wong,Tien Yin Wong +42 more
Jeffrey De Fauw,Joseph R. Ledsam,Bernardino Romera-Paredes,Stanislav Nikolov,Nenad Tomasev,Sam Blackwell,Harry Askham,Xavier Glorot,Brendan O'Donoghue,Daniel Visentin,George van den Driessche,Balaji Lakshminarayanan,Clemens Meyer,Faith Mackinder,Simon Bouton,Kareem Ayoub,Reena Chopra,Dominic King,Alan Karthikesalingam,Cian Hughes,Rosalind Raine,Julian Hughes,Dawn A Sim,Catherine A Egan,Adnan Tufail,Hugh Montgomery,Demis Hassabis,Geraint Rees,Trevor Back,Peng T. Khaw,Mustafa Suleyman,Julien Cornebise,Pearse A. Keane,Olaf Ronneberger +33 more
[...]