Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy
Jonathan Krause,Varun Gulshan,Ehsan Rahimy,Peter Karth,Kasumi Widner,Greg S. Corrado,Lily Peng,Dale R. Webster +7 more
TL;DR: Adjudication reduces the errors in DR grading by using a small number of adjudicated consensus grades as a tuning dataset and higher-resolution images as input, and to train an improved automated algorithm for DR grading.
read more
About: This article is published in Ophthalmology. The article was published on 12 Mar 2018. and is currently open access.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Table 4. Comparison of ophthalmologist grades versus adjudicated grades from retina specialists on the validation dataset. Confusion matrix for diabetic retinopathy and DME between the grade determined by majority decision of the ophthalmologists and the adjudicated consensus of retinal specialists. 
Table 5. Agreement between ophthalmologists’ grades with the adjudicated reference standard on the validation dataset. Sensitivity and specificity metrics are for moderate or worse DR and referable DME for each grader. Agreement between the adjudicated grade and the 5-point scale is also measured by the quadratic-weighted kappa. 
Table 3. Agreement between each retina specialist and the adjudicated reference standard on the validation dataset. Retina specialists correspond to those who contributed to the final adjudicated reference standard. Sensitivity and specificity metrics reported are for moderate or worse DR. Agreement between the preadjudication 5-point DR grade and the final adjudicated grade is also measured by the quadratic-weighted kappa. 
Table 2. Comparison of retinal specialist grades before and after adjudication on the validation dataset. Confusion matrix for diabetic retinopathy between the grade determined by majority decision and adjudicated consensus. 
Fig. 1. Grader agreement based on the adjudicated consensus grade for referable diabetic retinopathy (DR) and diabetic macular edema (DME). Independent grading of all 3 retinal specialists and all 3 ophthalmologists are included in this analysis. 
Fig. 2. Image resolution input to model versus area under the curve (AUC) for mild and above DR. Left: Using majority decision of retinal specialists as the reference standard. Right: Using the adjudicated consensus grade of retinal specialists as a reference standard. Shaded areas represent a 95% confidence interval as measured via bootstrapping.
Citations
Grading of diabetic retinopathy using a pre‐segmenting deep learning classification model: Validation of an automated algorithm
Dyllan Edson Similié,J. Andersen,Sebastian Dinesen,Thiusius Rajeeth Savarimuthu,Jakob Grauslund +4 more
TL;DR: A deep learning algorithm for diabetic retinopathy grading achieved comparable performance to human graders in a high-risk population, with 92% negative predictive value, suggesting its potential for autonomous identification of non-diabetic retinopathy patients in real-world settings.
Expert-validated estimation of diagnostic uncertainty for deep neural networks in diabetic retinopathy detection.
Murat Seckin Ayhan,Laura Kühlewein,Gulnar Aliyeva,Werner Inhoffen,Focke Ziemssen,Philipp Berens +5 more
TL;DR: This work describes an intuitive framework based on test-time data augmentation for quantifying the diagnostic uncertainty of a state-of-the-art DNN for diagnosing diabetic retinopathy and shows that the derived measure of uncertainty is well-calibrated and paves the way for an integrated treatment of uncertainty in DNN-based diagnostic systems.
Automated detection of retinal exudates and drusen in ultra-widefield fundus images based on deep learning.
Zhongwen Li,Chong Guo,Danyao Nie,Duoru Lin,Tingxin Cui,Yi Zhu,Chuan Chen,Lanqin Zhao,Xulin Zhang,Meimei Dongye,Dongni Wang,Fabao Xu,Chenjin Jin,Ping Zhang,Yu Han,Pisong Yan,Haotian Lin +16 more
TL;DR: Zhang et al. as mentioned in this paper developed and assessed a deep learning system for automated detection of RED using ultra-widefield fundus (UWF) images, which achieved areas under the receiver operating characteristic curve of 0.994 (95% confidence interval [CI]: 0.991-0.996), 0.972 ( 95% CI: 0.983-0,0.984), and 0.
Explainable Artificial Intelligence (XAI) with IoHT for Smart Healthcare: A Review
TL;DR: In this article , the authors discuss the use of artificial intelligence (AI) in healthcare, explainability is a highly contentious topic, because the majority of existing AI systems are incomprehensible and opaque, it is unlikely that AI technologies will be properly exploited and incorporated into standard clinical practice.
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).
Been Kim,Martin Wattenberg,Justin Gilmer,Carrie Cai,James Wexler,Fernanda Viegas,Rory Sayres +6 more
TL;DR: Researchers introduce Concept Activation Vectors (CAVs) to interpret deep learning models, enabling quantitative testing of concept importance through directional derivatives, and demonstrate its application in image classification and medical domains for hypothesis exploration and insight generation.
References
Deep learning
TL;DR: Deep learning is making major advances in solving problems that have resisted the best attempts of the artificial intelligence community for many years, and will have many more successes in the near future because it requires very little engineering by hand and can easily take advantage of increases in the amount of available computation and data.
67K
Gradient-based learning applied to document recognition
Yann LeCun,Léon Bottou,Léon Bottou,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio,Patrick Haffner +6 more
- 01 Jan 1998
TL;DR: In this article, a graph transformer network (GTN) is proposed for handwritten character recognition, which can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters.
53.5K
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky,Jia Deng,Hao Su,Jonathan Krause,Sanjeev Satheesh,Sean Ma,Zhiheng Huang,Andrej Karpathy,Aditya Khosla,Michael S. Bernstein,Alexander C. Berg,Li Fei-Fei +11 more
TL;DR: The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) as mentioned in this paper is a benchmark in object category classification and detection on hundreds of object categories and millions of images, which has been run annually from 2010 to present, attracting participation from more than fifty institutions.
A Coefficient of agreement for nominal Scales
TL;DR: In this article, the authors present a procedure for having two or more judges independently categorize a sample of units and determine the degree, significance, and significance of the units. But they do not discuss the extent to which these judgments are reproducible, i.e., reliable.
Gradient-based learning applied to document recognition
Yann LeCun,Léon Bottou,Léon Bottou,Yoshua Bengio,Yoshua Bengio,Yoshua Bengio,Patrick Haffner,Patrick Haffner +7 more
- 01 Jan 2001
TL;DR: This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
32.7K
Related Papers (5)
Daniel Shu Wei Ting,Daniel Shu Wei Ting,Carol Y. Cheung,Carol Y. Cheung,Gilbert Lim,Gavin Tan,Gavin Tan,Nguyen Duc Quang,Alfred Tau Liang Gan,Haslina Hamzah,Renata Garcia-Franco,Ian Yew San Yeo,Ian Yew San Yeo,Shu Yen Lee,Shu Yen Lee,Edmund Yick Mun Wong,Edmund Yick Mun Wong,Charumathi Sabanayagam,Charumathi Sabanayagam,Mani Baskaran,Mani Baskaran,Farah Nur Ilyana Mohd Ibrahim,Ngiap Chuan Tan,Ngiap Chuan Tan,Eric A. Finkelstein,Ecosse L. Lamoureux,Ecosse L. Lamoureux,Ian Y. H. Wong,Neil M. Bressler,Sobha Sivaprasad,Rohit Varma,Jost B. Jonas,Mingguang He,Ching-Yu Cheng,Ching-Yu Cheng,Gemmy Cheung,Gemmy Cheung,Tin Aung,Tin Aung,Wynne Hsu,Mong Li Lee,Tien Yin Wong,Tien Yin Wong +42 more
Jeffrey De Fauw,Joseph R. Ledsam,Bernardino Romera-Paredes,Stanislav Nikolov,Nenad Tomasev,Sam Blackwell,Harry Askham,Xavier Glorot,Brendan O'Donoghue,Daniel Visentin,George van den Driessche,Balaji Lakshminarayanan,Clemens Meyer,Faith Mackinder,Simon Bouton,Kareem Ayoub,Reena Chopra,Dominic King,Alan Karthikesalingam,Cian Hughes,Rosalind Raine,Julian Hughes,Dawn A Sim,Catherine A Egan,Adnan Tufail,Hugh Montgomery,Demis Hassabis,Geraint Rees,Trevor Back,Peng T. Khaw,Mustafa Suleyman,Julien Cornebise,Pearse A. Keane,Olaf Ronneberger +33 more
[...]