Editors: Liron Pantanowitz, MD, PhD, MHA, chair of the Department of Pathology and professor of pathology, University of Pittsburgh Medical Center, and Matthew G. Hanna, MD, vice chair of pathology informatics and associate professor, Department of Pathology, University of Pittsburgh Medical Center.
Recommendations for evaluating performance of machine learning in pathology
July 2024—The integration of machine learning models into pathology has revolutionized the field, offering new functionalities and workflows. Numerous machine learning (ML) models are commercially available, and organizations with computational pathology resources can develop their own. These models, whether or not they are imaging based, are intended to enhance clinical practice. However, no formal guidelines pertaining to verifying or validating such systems are available. Therefore, the authors proposed recommendations for evaluating ML systems that are based on evidence and literature that address, among other factors, the scope, strengths, and limitations of the technology. Machine learning augments human intelligence in pathology, assisting pathologists in patient testing and reporting by enhancing cognitive performance and decision-making. However, it is necessary to rigorously verify and validate the performance of ML models using metrics and local data before applying the technology in the clinical setting. Properly documenting training and competency is also crucial before clinical use. Continuous monitoring for performance defects, including shifts and drifts, is vital. Recommendations for evaluating the performance of ML-based clinical decision-support systems in pathology include clarifying the scope of evaluation, ensuring compliance with regulatory standards for verification and validation, and conducting a comprehensive risk assessment to identify sources of variability and error. It is also essential that users understand the variation in domain-specific terminology and consider predeployment factors, such as model properties, data compatibility, bias, and ethics. Furthermore, performance evaluation should use diverse real-world data sources and appropriate reference standards. It too is necessary to verify unmodified regulator-authorized ML models and validate modified ML models or laboratory-developed tests. The recommendations also emphasize using evaluation metrics and sample size calculations, conducting precision studies, and ensuring proper documentation and change management. Continuous monitoring to mitigate performance shifts, human factor evaluation to ensure model explainability and trust, and personnel training and competency evaluation planning for all involved personnel are integral parts of the recommendations. The authors concluded that these guidelines provide a comprehensive framework for integrating ML into pathology clinical practice, ensuring reliability and enhancing patient care.
Hanna MG, Olson NH, Zarella M, et al. Recommendations for performance evaluation of machine learning in pathology: A concept paper from the College of American Pathologists. Arch Pathol Lab Med. 2023. doi:10.5858/arpa.2023-0042-CP
Correspondence: Dr. Matthew G. Hanna at [email protected]
ASC recommendations for rapid on-site evaluation using telecytology
The use of telecytology to support rapid on-site evaluation of cytology samples has gained enormous popularity globally. This is largely in response to the rising demand for cytology teams to evaluate small biopsies using ancillary testing in the face of a diminishing workforce. Several medical centers have published their experiences with rapid on-site evaluation (ROSE) using telecytology, in which they shared their varying validation procedures, challenges, and deployment results. However, no pathology society or pathology organization had published a consensus or established guideline on best practices for this telecytology use case. Therefore, the American Society of Cytopathology (ASC) established a 13-member task force in 2023, comprising cytopathologists and cytologists from various medical centers who had extensive experience using telecytology for ROSE, to provide recommendations regarding validation of the technology for that use. The intent of the guidelines are to ensure safe practices and compliance with regulations. The experts based their recommendations on an extensive literature review. The recently published recommendations are broad and cover telecytology platforms (static and dynamic), personnel requirements, minimum number and types of cases to be used in validation, facilities to include for validation, acceptable diagnostic concordance rates (that is, at least 90 percent between telecytology and manual microscopic review), development of a quality management program for telecytology for such purposes as tracking errors and rectifying discrepancy trends, and billing. The recommendations endorse testing all hardware and software components separately and together. They also discuss the need to train end users and regularly evaluate their competency. The guidelines are intended to be a resource for pathology laboratories considering implementing telecytology or that already use this technology for ROSE. They should maximize the benefits of the technology for patients.
Lin O, Alperstein S, Barkan GA, et al. American Society of Cytopathology telecytology validation recommendations for rapid on-site evaluation (ROSE). J Am Soc Cytopathol. 2024;13(2):111–121.
Correspondence: Dr. Oscar Lin at [email protected]