Back to Presentations and Publications Main Page

Pathology Informatics Summit 2021

20210506_PIsummit-Gallas-share.pdf (2 MB, uploaded by Brandon D. Gallas 2 years 11 months ago)

Background

Artificial intelligence algorithms in digital pathology have enormous potential to increase diagnostic speed and accuracy. However, the performance of these algorithms must be validated against a reference standard before deployment in clinical practice. In this work, pathologists are considered as the reference standard. We studied interobserver variability in pathologists who evaluate stromal tumor-infiltrating lymphocytes (sTILs) in hematoxylin and eosin stained breast cancer biopsy specimens. Our ultimate goal is to create a validation dataset that is fit for a regulatory purpose.

Methods

Following an IRB exempt determination protocol, we obtained informed consent of volunteer pathologist annotators prior to completing data collection tasks via two modalities: an optical light microscope and two digital platforms (slides were scanned at 40X). Pathologists were trained on the clinical task of sTIL density estimation before annotating pre-specified regions of interest (ROIs) across multiple platforms. The ROI selection protocol sampled ROIs in the tumor, tumor boundary, and elsewhere. Inter-pathologist agreement was characterized with the root mean-squared difference, which is analogous to the root mean-squared error but doesn’t require ground truth.

Results

The pilot study accumulated 6,257 sTIL density estimates from 34 pathologists evaluating 64 cases, with 10 ROIs per case. The variability of sTIL density estimates in an ROI increases with the mean; the square roots of the reader-averaged mean-squared differences were 8.3, 17.7, and 40.4 as the sTIL density reference score increased from 0-10%, 10-40%, and 40-100%, respectively. We also found that the root mean-squared differences for some pathologists were considerably larger than others (as much as 120% larger than the next largest root mean-squared difference).

Conclusions

Slides, images, and annotations were successfully provided by volunteer collaborators and participants, which created an innovative and thorough method for data collection and truthing. This pilot study will inform the development of statistical methods, simulation models, and sizing analyses for pivotal studies. The development and results of this validation dataset and analysis tools will be made publicly available to serve as an instructive tool for algorithm developers and researchers. Furthermore, the methods used to analyze pathologist agreement between density estimates are applicable to other quantitative biomarkers.

Katherine Elfer, PhD, MPH
- FDA/CDRH/OSEL/DIDSR
Mohamed Amgad, MD
- Department of Pathology, Northwestern University
Weijie Chen, PhD
- FDA/CDRH/OSEL/DIDSR
Sarah Dudgeon, MPH
- CORE Center for Computational Health Yale-New Haven Hospital
Rajarsi Gupta, MD/PhD
- Stony Brook Medicine Dept of Biomedical Informatics
Matthew Hanna, MD
- Memorial Sloan Kettering Cancer Center
Steven Hart, PhD
- Department of Health Sciences Research, Mayo Clinic
Richard Huang, MD
- Massachusetts General Hospital/Harvard Medical School
Evangelos Hytopoulos, PhD
- iRhythmTechnologies Inc
Denis Larsimont, MD
- Department of Pathology, InstitutJules Bordet
Xiaoxian Li, MD/PhD
- Emory University School of Medicine
Anant Madabhushi, PhD
- Case Western Reserve University
Hetal Marble, PhD
- Massachusetts General Hospital/Harvard Medical School
Roberto Salgado, PhD
- Division of Research, Peter Mac CallumCancer Centre, Melbourne, Australia; Department of Pathology, GZA-ZNA Hospitals
Joel Saltz, MD/PhD
- Stony Brook Medicine Dept of Biomedical Informatics
Manasi Sheth, PhD
- FDA/CDRH/OPQE/Division of Biostatistics
Rajendra Singh, MD
- Northwell health and Zucker School of Medicine
Evan Szu, PhD
- Arrive Bio
Darick Tong, MS
- Arrive Bio
Si Wen, PhD
- FDA/CDRH/OSEL/DIDSR
Bruce Werness, MD
- Arrive Bio

Created on 12 May 2021, Last modified on 12 May 2021