Editors: Liron Pantanowitz, MD, PhD, MHA, chair of the Department of Pathology and professor of pathology, University of Pittsburgh Medical Center, and Matthew G. Hanna, MD, vice chair of pathology informatics and associate professor, Department of Pathology, University of Pittsburgh Medical Center.
Validation, implementation, and impact of an AI model for pathologic diagnosis of prostate cancer
October 2026—Artificial intelligence studies in the realm of pathology increasingly demonstrate the impressive diagnostic performance of AI, but it is also worthwhile to understand what happens when these tools are incorporated into routine clinical practice. To this end, the authors evaluated the validation, implementation, and real-world impact of an AI–assisted decision-support tool for prostate needle biopsies within a fully digital pathology practice. Rather than focusing solely on algorithm accuracy, they examined whether AI integration could meaningfully affect practical measures of laboratory performance, including diagnostic turnaround time and use of IHC. The authors first validated the tool in their institution using routine clinical cases and pathologist-rendered diagnoses as the reference standard. The AI tool demonstrated strong diagnostic performance, with sensitivity of 91 to 100 percent, specificity of 99 percent, positive predictive value of 98 percent, negative predictive value of 96 percent, and an area under the receiver operating characteristic curve of 0.97. Following validation, the AI tool was incorporated into routine prostate biopsy sign-out. The authors then compared cases from the three-month periods before and after implementation, excluding a transition month. Importantly, turnaround time was measured from completion of whole slide scanning to final pathologist sign-out, allowing the analysis to focus on the interpretive portion of the digital pathology workflow. The study found that, following implementation of the AI tool, diagnostic turnaround time decreased by approximately 30 percent and utilization by 38 percent. Together, these findings suggest that AI-assisted review may help pathologists reach diagnostic decisions more efficiently and reduce the need for ancillary testing in selected cases. The study also illustrated why local validation remains critical. Even high-performing algorithms must be evaluated within the local environment in which they will be used and mirror the intended uses, including for specimen preparation, scanning infrastructure, and pathologist workflow. The AI system in this study functioned as a decision-support tool rather than an autonomous diagnostic system, with the pathologist retaining responsibility for interpretation and final diagnosis. Perhaps the most important contribution of this study is that it begins to address the next phase of computational pathology by demonstrating the measurable value of AI after clinical deployment. For laboratories considering AI, performance metrics such as sensitivity, specificity, and area under the curve remain necessary, but they may no longer be sufficient alone. Turnaround time, ancillary test utilization, diagnostic quality, resource utilization, and, ultimately, cost and patient outcomes will increasingly be important measures of successful implementation of the technology. As digital pathology becomes more widely adopted, studies such as this one shift the conversation from whether pathology AI works to how it should be validated, integrated, monitored, and evaluated in clinical practice. The transition from algorithm performance to demonstrable clinical and operational value may determine whether AI becomes an experimental add-on or a sustainable component of the pathology laboratory.
Udoh AI, Eyzaguirre E, Muthukumarana V, et al. Validation, implementation, and impact of an AI model in routine practice for pathologic diagnosis of prostate cancer in an academic medical center. J Pathol Inform. 2026. doi.org/10.1016/j.jpi.2026.100675
Correspondence: Dr. Agnes I. Udoh at aiudoh@utmb.edu or Dr. Eduardo Eyzaguirre at ejeyzagu@utmb.edu
Assessing HER2 expression in breast cancer whole slide images: agreement across 10 AI models
Artificial intelligence-based analysis for HER2 using whole slide imaging provides precise and reproducible assessments compared to human interpretation, which is variable, even for HER2 low cases. While many AI models have been developed for such analysis, the extent of their measurement variability is unclear. The authors conducted a multi-stakeholder study to evaluate the consistency of 10 independently developed AI models for scoring HER2 IHC in breast cancer whole slide images (WSI). They analyzed 1,124 HER2-stained WSI scanned at 40 times magnification (0.26 μm/pixel) using the Aperio GT 450 DX scanner. The images were obtained from 733 breast cancer patients via an accredited pathology laboratory in Antwerp, Belgium. The 10 AI models that were compared included nine research-use-only tools and one regulatory-cleared tool. The models varied substantially in architecture; training data sets; use of paired H&E or HER2-stained images, or both; and requirements for human intervention. Importantly, all models were “locked” and applied independently to the same image data set without modification or pathologist review, allowing direct comparison of algorithmic outputs. The primary finding was that overall agreement between AI models was moderate and remarkably similar to agreement among pathologists. Across the seven models that provided standard American Society of Clinical Oncology/College of American Pathologists HER2 categorical scores (0, 1+, 2+, 3+), the median overall percent agreement (OPA) was 65.1 percent, with a Cohen’s kappa of 0.51, which is moderate agreement and usually considered inadequate for clinical application. The pathologists demonstrated somewhat higher agreement (OPA, 70.4 percent; kappa, 0.57) but also demonstrated substantial variability. Interestingly, agreement between AI models and pathologists was essentially equivalent to model-to-model agreement, suggesting that AI systems are likely encountering many of the same interpretive challenges as human observers. The study found that concordance varied considerably by HER2 category. Agreement was highest for unequivocally positive HER2 3+ cases. When dichotomized as HER2 3+ versus not 3+, the AI models achieved a median agreement of 97.3 percent, with a kappa of 0.86, indicating excellent reproducibility. In contrast, agreement was substantially lower for HER2-low and -intermediate categories. The lowest concordance was observed for categories 1+ and 2+, which are becoming increasingly important for therapeutic decision-making. These study findings mirror previously reported challenges among pathologists and reinforce the difficulty of reliably classifying low-level HER2 expression. The authors further explored the reasons for discordance by examining the percentage of tumor cells displaying each HER2 staining pattern. They found that both AI models and pathologists achieved the greatest consistency when tumors showed predominantly no staining (HER2 0) or strong complete membrane staining (HER2 3+). Cases dominated by faint or weak staining patterns, characteristic of HER2-low disease, produced substantially lower agreement. Reliability metrics showed that AI models were very consistent in recognizing strong complete membrane staining but much less reliable in assessing the weaker staining patterns associated with HER2 1+ and 2+ categories. The study also distinguished between tumor detection and staining interpretation. Several AI models reported tumor cell counts and counts of stained cells. Agreement for detecting invasive carcinoma cells was high, with strong concordance correlation coefficients for total and stained tumor cells. This suggests that differences in HER2 classification are driven primarily by interpretation of staining intensity and completeness of membrane staining rather than an inability to recognize tumor cells. In other words, the major challenge likely lies in biomarker interpretation rather than tumor segmentation. The authors also analyzed continuous quantitative measures, such as H-score, which is a semi-quantitative method used in immunohistochemistry to assess biomarker expression by multiplying the percentage of positive tumor cells by their staining intensity. While H-scores showed strong correlation across models, variability increased in the low and intermediate ranges, again paralleling the findings for categorical HER2 scoring. AI-based H-scores may ultimately offer more nuanced quantification than traditional categorical scoring, particularly for HER2-low disease, but additional validation is needed before such metrics can be routinely incorporated into clinical decision-making. A detailed review of highly discordant cases by the authors revealed several recurrent causes of disagreement. These involved tissue artifacts, poor staining quality, out-of-focus scans, sparse tumor cellularity, stromal-rich tumors, benign cells with HER2 staining, ductal carcinoma in situ showing positive staining, and heterogeneous tumors containing regions with both HER2 0 and HER2 3+ expression. Weak or incomplete membrane staining was also a particularly frequent source of disagreement. Many of these challenges affect both AI models and pathologists, emphasizing that discordance often reflects biologic and technical complexity rather than algorithmic failure alone. The authors concluded that this study has important implications for computational pathology as it demonstrates that AI systems can achieve pathologist-level reproducibility for HER2 assessment; identifies HER2-low classification as the major remaining challenge in such assessment; and highlights the need for standardized reference data sets, benchmarking frameworks, and regulatory alignment to improve consistency across AI tools.
McKelvey B, Torres-Saavedra PA, Li J, et al. Agreement across 10 artificial intelligence models in assessing human epidermal growth factor receptor 2 (HER2) expression in breast cancer whole-slide images. Mod Pathol. 2026. doi.org/10.1016/j.modpat.2025.100944
Correspondence: Dr. Mark Stewart at mstewart@focr.org