Abstract There is interest in comparing the output, principally the likelihood ratio, from the two probabilistic genotyping software EuroForMix (EFM) and STRmix™. Many of these comparison studies are descriptive and make little or no effort to diagnose the cause of difference. There are fundamental differences between EFM and STRmix™ that are causative of the largest set of likelihood ratio differences. This set of differences is for false donors where there are many instances of LR s just above or below 1 for EFM that give much lower LR s in STRmix™. This is caused by the separate estimation of parameters such as allele height variance and mixture proportion using MLE under H p and H a for EFM. This can result in very different estimations of these parameters under H p and H a . It results in a departure from calibration for EFM in the region of LR s just above and below 1. 

A diagnosis of the primary difference between <scp>EuroForMix</scp> and <scp>STRmix</scp>™

Journal of Forensic SciencesEarly View LETTER TO THE EDITOR Commentary on: Hahn M, Anslinger K, Eckert M, Fimmers R, Grethe S, Hohoff C, et al. [Joint recommendations of the project group "Biostatistical DNA Calculations" and the Trace Commission on the Biostatistical Evaluation of Forensic DNA Analytical Findings with Fully Continuous Models (FCM)]. Rechtsmedizin (Berl). 2023; 33(1):3–12. doi: 10.1007/s00194-022-00599-5 Charles E. H. Berger PhD, Charles E. H. Berger PhD Netherlands Forensic Institute, The Hague, The Netherlands Institute of Criminal Law and Criminology, Leiden University, Leiden, The NetherlandsSearch for more papers by this authorMaarten Kruijver PhD, Maarten Kruijver PhD Institute of Environmental Science and Research Limited, Auckland, New ZealandSearch for more papers by this authorTacha Hicks PhD, Tacha Hicks PhD Forensic Genetics Unit, University Center of Legal Medicine, Lausanne—Geneva, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland Fondation pour la Formation Continue Universitaire Lausannoise (UNIL-EPFL) & School of Criminal Justice, Lausanne, SwitzerlandSearch for more papers by this authorChristophe Champod PhD, Christophe Champod PhD orcid.org/0000-0002-4035-2698 Faculty of Law, Criminal Justice and Public Administration, School of Criminal Justice, University of Lausanne, Lausanne, SwitzerlandSearch for more papers by this authorDuncan Taylor PhD, Duncan Taylor PhD Forensic Science SA, Adelaide, South Australia, Australia School of Biological Sciences, Flinders University, Adelaide, South Australia, AustraliaSearch for more papers by this authorJohn Buckleton DSc, Corresponding Author John Buckleton DSc [email protected] Institute of Environmental Science and Research Limited, Auckland, New Zealand Department of Statistics, University of Auckland, Auckland, New Zealand Correspondence John Buckleton, Department of Statistics, University of Auckland, Private Bag 92019, Auckland 1142, New Zealand. Email: [email protected]Search for more papers by this author Charles E. H. Berger PhD, Charles E. H. Berger PhD Netherlands Forensic Institute, The Hague, The Netherlands Institute of Criminal Law and Criminology, Leiden University, Leiden, The NetherlandsSearch for more papers by this authorMaarten Kruijver PhD, Maarten Kruijver PhD Institute of Environmental Science and Research Limited, Auckland, New ZealandSearch for more papers by this authorTacha Hicks PhD, Tacha Hicks PhD Forensic Genetics Unit, University Center of Legal Medicine, Lausanne—Geneva, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland Fondation pour la Formation Continue Universitaire Lausannoise (UNIL-EPFL) & School of Criminal Justice, Lausanne, SwitzerlandSearch for more papers by this authorChristophe Champod PhD, Christophe Champod PhD orcid.org/0000-0002-4035-2698 Faculty of Law, Criminal Justice and Public Administration, School of Criminal Justice, University of Lausanne, Lausanne, SwitzerlandSearch for more papers by this authorDuncan Taylor PhD, Duncan Taylor PhD Forensic Science SA, Adelaide, South Australia, Australia School of Biological Sciences, Flinders University, Adelaide, South Australia, AustraliaSearch for more papers by this authorJohn Buckleton DSc, Corresponding Author John Buckleton DSc [email protected] Institute of Environmental Science and Research Limited, Auckland, New Zealand Department of Statistics, University of Auckland, Auckland, New Zealand Correspondence John Buckleton, Department of Statistics, University of Auckland, Private Bag 92019, Auckland 1142, New Zealand. Email: [email protected]Search for more papers by this author First published: 20 November 2023 https://doi.org/10.1111/1556-4029.15424 See Original Article here. See Authors' response here. Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinkedInRedditWechat REFERENCES 1Hahn M, Anslinger K, Eckert M, Fimmers R, Grethe S, Hohoff C, et al. Gemeinsame Empfehlungen der Projektgruppe "Biostatistische DNA-Berechnungen" und der Spurenkommission zur biostatistischen Bewertung forensischer DNA-analytischer Befunde mit vollkontinuierlichen Modellen (VKM) [Joint recommendations of the project group "biostatistical DNA calculations" and the trace commission on the biostatistical evaluation of forensic DNA analytical findings with fully continuous models (FCM)]. Rechtsmedizin (Berl). Rechtsmedizin. 2023; 33(1): 3–12. https://doi.org/10.1007/s00194-022-00599-5 10.1007/s00194-022-00599-5 Google Scholar 2Templin M, Zimmermann P, Kranz S, Eckert M, Leuker C, Razbin S, et al. Einsatz vollkontinuierlicher modelle zur biostatistischen bewertung forensischer DNA-analytischer befunde [Use of fully continuous models for the biostatistical evaluation of forensic DNA analysis findings]. Dent Rec. 2023; 33(1): 13–29. https://doi.org/10.1007/s00194-022-00600-1 10.1007/s00194?022?00600?1 Google Scholar 3Riman S, Iyer H, Vallone PM. Examining performance and likelihood ratios for two likelihood ratio systems using the PROVEDIt dataset. PLoS One. 2021; 16(9):e0256714. https://doi.org/10.1371/journal.pone.0256714 10.1371/journal.pone.0256714 CASPubMedWeb of Science®Google Scholar 4Costa C, Figueiredo C, Amorim A, Costa S, Ferreira PM, Pinto N. Quantification of forensic genetic evidence: comparison of results obtained by qualitative and quantitative software for real casework samples. Forensic Sci Int Genetics. 2022; 59:102715. https://doi.org/10.1016/j.fsigen.2022.102715 10.1016/j.fsigen.2022.102715 CASPubMedWeb of Science®Google Scholar 5Buckleton J, Bright J-A, Taylor D, Wivell R, Bleka Ø, Gill P, et al. Re: Riman et al. Examining performance and likelihood ratios for two likelihood ratio systems using the PROVEDIt Dataset. Forensic Sci Int Genetics. 2022; 59:102709. https://doi.org/10.1016/j.fsigen.2022.102709 10.1016/j.fsigen.2022.102709 CASPubMedWeb of Science®Google Scholar 6Susik M, Sbalzarini IF. Analysis of the Hamiltonian Monte Carlo genotyping algorithm on PROVEDIt mixtures including a novel precision benchmark. Forensic Sci Int Genetics. 2023; 64:102840. https://doi.org/10.1016/j.fsigen.2023.102840 10.1016/j.fsigen.2023.102840 CASPubMedWeb of Science®Google Scholar 7Cheng K, Bleka O, Gill P, Curran J, Bright J-A, Taylor D, et al. A comparison of likelihood ratios obtained from EuroForMix and STRmix™. J Forensic Sci. 2021; 66(6): 2138–2155. https://doi.org/10.1111/1556-4029.14886 10.1111/1556-4029.14886 CASPubMedWeb of Science®Google Scholar 8Berger CEH, Slooten K. The LR does not exist. Sci Justice. 2016; 56(5): 388–391. https://doi.org/10.1016/j.scijus.2016.06.005 10.1016/j.scijus.2016.06.005 PubMedWeb of Science®Google Scholar 9Ramos D, Gonzalez-Rodriguez J. Reliable support: measuring calibration of likelihood ratios. Forensic Sci Int. 2013; 230(1–3): 156–169. https://doi.org/10.1016/j.forsciint.2013.04.014 10.1016/j.forsciint.2013.04.014 PubMedWeb of Science®Google Scholar 10Ramos D. On the calibration of likelihood ratios. 2011. Accessed from: http://arantxa.ii.uam.es/~dramos/files/2011_02_08_WIC_Ramos_calibrtionLRValues_v2.pdf. Accessed 24 Oct 2023. Google Scholar 11Ramos D, Meuwly D, Haraksim R, Berger CEH. Validation of forensic automatic likelihood ratio methods. In: D Banks, K Kafadar, D Kaye, M Tackett, editors. Handbook of forensic statistics. Boca Raton, FL: Chapman & Hall/CRC; 2020. p. 143–164. 10.1201/9780367527709-7 Google Scholar 12Robertson B, Vignaux GA, Berger CEH. Interpreting evidence: evaluating forensic science in the courtroom. 2nd ed. Hoboken, NJ: John Wiley & Sons, Ltd; 2016. 10.1002/9781118492475 Google Scholar 13Buckleton JS, Kruijver M, Curran J, Bright J-A. Calibration of STRmix LRs following the method of Hannig et al. Wellington, New Zealand: Institute of Environmental Science and Research; 2020. https://doi.org/10.26091/ESRNZ.12324011.v1 Google Scholar 14Bright J-A, Jones Dukes M, Pugh SN, Evett IW, Buckleton JS. Applying calibration to LRs produced by a DNA interpretation software. Aust J Forensic Sci. 2019; 53(2): 147–153. https://doi.org/10.1080/00450618.2019.1682668 10.1080/00450618.2019.1682668 Web of Science®Google Scholar 15Stoney DA. What made us ever think we could individualize using statistics? J Forensic Sci Soc. 1991; 31(2): 197–199. https://doi.org/10.1016/s0015-7368(91)7368(91)73138-1 10.1016/S0015-7368(91)73138-1 CASPubMedWeb of Science®Google Scholar 16Champod C, Biedermann A. Overview and meaning of identification/individualization. In: MM Houck, editor. Encyclopedia of forensic sciences. 3rd ed. Oxford, U.K.: Elsevier Ltd.; 2023. p. 53–62. 10.1016/B978-0-12-823677-2.00152-5 Google Scholar Early ViewOnline Version of Record before inclusion in an issue ReferencesRelatedInformation 

Commentary on: Hahn M, Anslinger K, Eckert M, Fimmers R, Grethe S, Hohoff C, et al. [Joint recommendations of the project group “Biostatistical <scp>DNA</scp> Calculations” and the Trace Commission on the Biostatistical Evaluation of Forensic <scp>DNA</scp> Analytical Findings with Fully Continuous Models (<scp>FCM</scp>)]. Rechtsmedizin (Berl). 2023; 33(1):3–12. doi: 10.1007/s00194‐022‐00599‐5

There is increasing support for reporting evidential strength as a likelihood ratio (LR) and increasing interest in (semi-)automated LR systems. The log-likelihood ratio cost (Cllr) is a popular metric for such systems, penalizing misleading LRs further from 1 more. Cllr = 0 indicates perfection while Cllr = 1 indicates an uninformative system. However, beyond this, what constitutes a "good" Cllr is unclear. Aiming to provide handles on when a Cllr is "good", we studied 136 publications on (semi-)automated LR systems. Results show Cllr use heavily depends on the field, e.g., being absent in DNA analysis. Despite more publications on automated LR systems over time, the proportion reporting Cllr remains stable. Noticeably, Cllr values lack clear patterns and depend on the area, analysis and dataset. As LR systems become more prevalent, comparing them becomes crucial. This is hampered by different studies using different datasets. We advocate using public benchmark datasets to advance the field. 

An overview of log likelihood ratio cost in forensic science – Where is it used and what values can we expect?

Abstract The validity of a probabilistic genotyping (PG) system is typically demonstrated by following international guidelines for the developmental and internal validation of PG software. These guidelines mainly focus on discriminatory power. Very few studies have reported with metrics that depend on calibration of likelihood ratio (LR) systems. In this study, discriminatory power as well as various calibration metrics, such as Empirical Cross-Entropy (ECE) plots, pool adjacent violator (PAV) plots, log likelihood ratio cost (Cllr and Cllr cal ), fiducial calibration discrepancy plots, and Turing’ expectation were examined using the publicly-available PROVEDIt dataset. The aim was to gain deeper insight into the performance of a variety of PG software in the ‘lower’ LR ranges (∼LR 1-10,000), with focus on DNAStatistX and EuroForMix which use maximum likelihood estimation (MLE). This may be a driving force for the end users to reconsider current LR thresholds for reporting. In previous studies, overstated ‘low’ LRs were observed for these PG software. However, applying (arbitrarily) high LR thresholds for reporting wastes relevant evidential value. This study demonstrates, based on calibration performance, that previously reported LR thresholds can be lowered or even discarded. Considering LRs &gt;1, there was no evidence for miscalibration performance above LR ∼1,000 when using Fst 0.01. Below this LR value, miscalibration was observed. Calibration performance generally improved with the use of Fst 0.03, but the extent of this was dependent on the dataset: results ranged from miscalibration up to LR ∼100 to no evidence of miscalibration alike PG software using different methods to model peak height, HMC and STRmix. This study demonstrates that practitioners using MLE-based models should be careful when low LR ranges are reported, though applying arbitrarily high LR thresholds is discouraged. This study also highlights various calibration metrics that are useful in understanding the performance of a PG system. Highlights Discriminatory power and calibration performance of PG software are evaluated. The utility of various calibration metrics are explored in ‘low’ LR ranges. Focus was on DNAStatistX and EuroForMix software using the MLE method. Calibration performance was dependent on Fst value and dataset size. Results suggest reconsideration of lower LR thresholds and cautious reporting of ‘low’ LRs. 

pdf/low-lrs-obtained-from-dna-mixtures-on-calibration-and-lxn78494zg.pdf

‘Low’ LRs obtained from DNA mixtures: On calibration and discrimination performance of probabilistic genotyping software

We provide an internal validation study of a recently published precise DNA mixture algorithm based on Hamiltonian Monte Carlo sampling (Susik et al., 2022). We provide results for all 428 mixtures analysed by Riman et al. (2021) and compare the results with two state-of-the-art software products: STRmix™ v2.6 and Euroformix v3.4.0. The comparison shows that the Hamiltonian Monte Carlo method provides reliable values of likelihood ratios (LRs) close to the other methods. We further propose a novel large-scale precision benchmark and quantify the precision of the Hamiltonian Monte Carlo method, indicating its improvements over existing solutions. Finally, we analyse the influence of the factors discussed by Buckleton et al. (2022). 

Analysis of the Hamiltonian Monte Carlo genotyping algorithm on PROVEDIt mixtures including a novel precision benchmark

We investigate a class of DNA mixture deconvolution algorithms based on variational inference, and we show that this can significantly reduce computational runtimes with little or no effect on the accuracy and precision of the result. In particular, we consider Stein Variational Gradient Descent (SVGD) and Variational Inference (VI) with an evidence lower-bound objective. Both provide alternatives to the commonly used Markov-Chain Monte-Carlo methods for estimating the model posterior in Bayesian probabilistic genotyping. We demonstrate that both SVGD and VI significantly reduce computational costs over the current state of the art. Importantly, VI does so without sacrificing precision or accuracy, presenting an overall improvement over previously published methods. 

Variational inference accelerates accurate DNA mixture deconvolution

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Ann Cartwright

Author Tools

Chat about Author

Papers

Analysis of the Hamiltonian Monte Carlo genotyping algorithm on PROVEDIt mixtures including a novel precision benchmark

Variational inference accelerates accurate DNA mixture deconvolution