Amy Carpenter
November 2024—In the clinical parasitology laboratory at Mayo Clinic, job satisfaction is up and ergonomic strain is down. Work difficulty is down, too, as is the pressure of time.
All that and much more was measured one year after bringing artificial intelligence into the laboratory to screen out the negatives.
“It has been a game changer for us,” Bobbi S. Pritt, MD, MSc, DTMH, chair of the Mayo Division of Clinical Microbiology and director of the parasitology laboratory in Rochester, Minn., said of the lab’s use of an AI algorithm developed by Techcyte of Orem, Utah.
To detect parasites, “morphologic examination of clinical specimens by light microscopy remains the gold standard, and microscopy is very good if you have skilled, trained microscopists,” said Dr. Pritt, who is professor of laboratory medicine and pathology, Mayo Clinic College of Medicine and Science. But detecting ova and parasites is a manual, subjective exam that requires a high level of training and skill, and in nonendemic settings where parasites are seldom seen, many specimens are negative. “This leads to low staff satisfaction, challenges in maintaining competency, and, if you have high volumes like we do, ergonomic issues and the potential to get distracted and miss the positives.”
That’s where digitalization and artificial intelligence come in, she said, in a July webinar sponsored by Techcyte.
“Screening out negatives was our primary interest when we implemented Techcyte AI, and it definitely has fulfilled that promise,” Dr. Pritt said. “If you screen out negatives, which in our case is 90 to 95 percent of all of our tests for the stool parasitology exam, that allows our highly trained personnel to focus on what’s interesting—the positive or suspicious findings.”
“A good AI system,” she said, also has the potential to shorten turnaround time and increase sensitivity. And teleparasitology becomes possible, putting diagnostic expertise in areas where local expertise is unavailable. “It’s great for teaching,” she said, “and maybe someday it will allow our technologists to work from home, although we’re not encouraging that at this time.”
A slide scanner and software are needed, and the system has to be validated as a laboratory-developed test because in the United States no system at this time is FDA approved or cleared for in vitro diagnostic use.
“And you never want to automate or apply AI to a poor process, so you may want to take the opportunity to see if you can improve your existing processes” before implementing a digitalized and AI workflow.
Dr. Pritt used stool parasitology as an example to explain how the AI works but noted it can be applied to other types of preparations. “We’re using Techcyte AI in our laboratory for bacteriology Gram stain exams as well, and it’s going very well,” she said. “It is finding bacteria that had been missed by our technologists, so it is increasing our sensitivity.”
Techcyte’s other AI systems are for cervical cytology, urine cytology, and automated blood differential. “New algorithms are being developed continually both in the clinical pathology space as well as for anatomic pathology,” says Troy Bankhead, VP of marketing, Techcyte Europe.
The conventional O&P exam comprises two parts, and CAP-accredited laboratories must do both: a concentrated wet mount examination for helminth eggs and protozoa and a permanently stained preparation for protozoa (typically made from unconcentrated stool and stained with trichrome, as at Mayo, or another stain).
It was a group at ARUP Laboratories that developed with Techcyte the algorithm used in the Techcyte system for intestinal protozoa (Mathison BA, et al. J Clin Microbiol. 2020;58[6]:e02053-19). “They had helped design with Techcyte this deep convolutional neural network for detecting protozoa within trichrome-stained stool specimens,” Dr. Pritt said. The algorithm detects and identifies multiple types of protozoa and was optimized for high sensitivity. “It may overcall potential parasites, but then the technologist uses it as a tool to determine if there truly is something there. So the negative predictive value is very high, which is important—you don’t want to miss any potential parasites.”
The algorithm was trained on between 1,394 and 23,566 exemplars per class of organism, and the positive and negative slide-level agreement with conventional microscopy was 98 to 99 percent. Analytical sensitivity using serial fivefold dilutions was greater than manual examination.
After reading their 2020 article, Dr. Pritt decided to do a full evaluation in her laboratory, after which she and colleagues proceeded to implementation.
The workflow in the laboratory begins with slide preparation, staining, coverslipping, and scanning. “That’s the digitalization component,” Dr. Pritt said. “And then the digitalized images are analyzed by the
AI software system to assist the technologist in their interpretation.”
Techcyte’s AI software searches through the matrix of material to look for objects of interest it is trained to find. “They’re grouped into suggested categories so it’s helpful and easy to use,” Dr. Pritt said, and they’re presented to the technologist for analysis and review. The technologist can click on a box if they agree with the interpretation, and the result is transferred into the laboratory information system.
“Negative specimens can be reviewed digitally and signed out if the technologist feels comfortable doing so, but all positive or questionable results are manually reviewed. That’s how we’re doing it in my laboratory and how it’s been widely adopted,” Dr. Pritt said.
After selecting the Techcyte AI system of the two commercially available systems she and her laboratory staff evaluated, they purchased the Hamamatsu Nanozoomer S360 digital slide scanner, which accommodates high volumes. “It’s a large, freestanding scanner, so you need to have space in your laboratory.” It captures images at 40× mode dry but digital zooms to 1,000× mode, “so you get that nice, high resolution,” Dr. Pritt said. It reads barcodes and communicates bidirectionally with the LIS. “Having that transfer of the information with barcode reading was important to us in a high-volume laboratory.”
The scanner self-adjusts focus points. The other AI/scanner system they evaluated required the technologist to refocus “every so many slides, and that wouldn’t be amenable to our high workflow,” she said. Up to 360 slides can be loaded for scanning at a time. “This could be done the night before so when the technologists come in the next morning all the slides are ready and available to read.”
Dr. Pritt and colleagues had to adjust their workflow to accommodate the scanning time.
They also had to look at how their slides were made. “We had to make the slides thin and have a monolayer. The slides on the Hamamatsu are scanned in x- and y-axes, but there’s no z-plane focus” for an up-and-down focus. “It’s possible, but it takes up a lot of memory and is time-consuming. And the system is trained on just x- and y-axes,” and that means having a thin monolayer. “Yet by making the specimen relatively thin, there was a concern we would dilute the specimen too much and lose sensitivity,” Dr. Pritt said, “so we went with using the concentrate for our permanently stained preparations, which worked well.” All laboratory assistants were trained to make the monolayers in a consistent manner.
They had to change their barcode labels, too. “Our existing barcodes were a little large—they hung over the slide,” and would get stuck in the scanner. “And we had to use the right type of barcode so it could be read by the barcode reader with the scanner.”
They also had to permanently mount the slides. “We used to put a drop of mineral oil on the slide, put a coverslip on it, and go, but that meant the coverslip could move around while being scanned. So we needed to use a permanent mounting medium,” Dr. Pritt said.
Having to coverslip all slides up front to allow for scanning was “a perfect opportunity for automation,” she said. Staff evaluated two automatic coverslippers and selected the Leica CV5030. The slides go through the stainer and are transferred to the coverslipper and automatically coverslipped—“a nice workflow change,” she noted.
With so many workflow adjustments, she and colleagues decided to reexamine everything and simplify, including consolidating to a single version of the trichrome stain. They had been using Ecofix with Ecostain and decided to see if the Ecostain would also work well with the two-vial fixative polyvinyl alcohol and sodium acetate-acetic acid-formalin. “And it worked well,” she said. The laboratory now uses only the Ecostain, a modified version of the trichrome stain. They also decided to limit the types of acceptable preservatives and no longer accepted PVA with heavy metals. “That was a good step for us,” she said.
All changes—coverslip, mounting medium, single stain—required individual validations. “We also had to compare the unconcentrated thick specimen to the concentrated thin specimen to make sure we didn’t lose sensitivity, especially if the slides had to be read manually if the scanner went down. And we found we gained sensitivity with the thin, concentrated method.”
Validating the algorithm was the next step.
“As a laboratory-developed test, we validated it completely, including evaluation of accuracy, precision, and limit of detection,” Dr. Pritt said. They started with accuracy. “We had 142 stool specimens that included positives representing all the classes identifiable with the Techcyte algorithm, and we had negatives.”
All specimens were reviewed by the Techcyte algorithm and manually in a blinded fashion by multiple trained and competent technologists, in what Dr. Pritt describes as “a pretty big undertaking.”
Of the manual results, 82 were positives. By AI, 83 were found to be positive. “So the AI found an additional positive that had been missed by manual review with light microscopy,” she said. “We went back and rereviewed that one that was not identified initially, and it was a true positive.”
Of the manual results, 60 were negative. By AI, 59 were negative. “That’s understandable.” The AI algorithm detected more white blood cells (29) and red blood cells (12) than the manual review (24 WBC, eight RBC). “These were all confirmed positives. Our technologists in the Techcyte process review all positives to confirm the result, and that’s our workflow to this day.”
Analytical sensitivity came next. They had three positive specimens that contained a range of commonly seen parasites. “We did twofold dilutions—1:1 all the way down to 1:128.” Six slides were made from each dilution (144 total slides), and all were reviewed in a blinded manner by both conventional light microscopy and AI-assisted microscopy. “And the AI algorithm provided equal or improved sensitivity for each specimen,” Dr. Pritt said.
“In each case, we were able to go down multiple dilutions with AI and still find parasites long after manual microscopy ceased to identify parasites. So the AI algorithm is more sensitive than manual microscopy.”
She and colleagues validated the AI platform as a laboratory-developed test or method, “but it was completed and marketed by Techcyte, so it’s not something we created in the laboratory. And it is part of our ova-and-parasite exam, so it’s a tool, not a standalone test.” Laboratories that want to develop their own software systems would have to perform significantly more studies.
To measure the impact of AI, they tracked, among other things, turnaround time.
“That was going up after the pandemic. Some labs stopped doing ova-and-parasite exams and started sending them to us. Unfortunately, as our volumes went up, our turnaround time also went up, and it got to a point”—about two weeks—“that we considered unreasonable. This was demoralizing to our staff.”
After implementing the Techcyte algorithm in May 2023 with only two “super users,” there was an immediate small drop in turnaround time. “And then we slowly rolled it out to the rest of the laboratory, and there was a dramatic drop in turnaround time,” though volumes remained high. “Our turnaround time is now down to about the two-day mark, which is where we want it to be. We have since followed up, and it remains around two days, so this is sustained improvement.”
They tracked the positivity rate, too, measured in periods pre- and post-implementation of the algorithm. From Aug. 1, 2021 to April 30, 2022, positivity was 3.07 percent. From Aug. 1, 2022 to April 30, 2023, it was 3.97 percent, and from Aug. 1, 2023 to April 30, 2024, it was 5.28 percent. In the summer this year, the positivity rate was closer to about six percent.
Overall, “our positives went up,” Dr. Pritt said, noting they are confirmed positives. “This is improving patient care.”
Staff satisfaction rose too. Measurements taken between pre-implementation in October 2022 and post-implementation one year later showed declines in the levels of physical demand, time pressure, and work difficulty. Job satisfaction went up, and ergonomic strain went down. “The satisfaction of just ‘performance’ almost doubled” (see “Parasitology staff satisfaction,” above).
Mental demand increased slightly, from less than 60 percent to about 65 percent during the early period when technologists were learning the new system.
Capacity is the final measure of success. Technologist review time of the trichrome-stained slides went from four minutes on average to 1.5 minutes on average—down 62.5 percent. Review of clearly negative specimens, which make up about 90 percent of the cases, declined from five minutes to 30 seconds. “If you can just review and quickly sign those out as negative, that’s a huge win. That was a 90 percent reduction.” The laboratory was short-staffed, so there was never a plan to let people go. “There was more than enough work to go around.”
Laboratory staff were part of the process, she said, from evaluation to validation to implementation. “They’ve presented their results to multiple groups, and I think they have a lot of pride in the work they did. And it’s been very well accepted.”
Dr. Pritt added the following finer points about integrating the Techcyte system with the LIS, performing quality control on the AI system, and more:
- Integrating the AI system with the LIS isn’t required to use the Techcyte system, but the Mayo lab worked with its IT staff and Techcyte to do so. “We decided it was important due to our volumes, and it was helpful to our technologists to be able to click the button on the parasites they agreed with and then transfer those results into the LIS.”
- For quality control, four slides serving different functions are run daily. The two positive and one negative controls go through the full staining and coverslipping process to test the entire system. Additionally, a pre-stained drift control is scanned and analyzed each day. “It has a certain quantity of parasites on it; we picked something like Giardia that’s straightforward. We expect a certain quantity of Giardia parasites to be detected. We would worry if, for example, only five parasites were detected one day and 100 the next. That would be outside our acceptance range.” This indicates the scanner is functioning properly and that the algorithm “is getting the same number of organisms in about the same range.”
Not a control but part of the lab’s quality system is a periodic pull of negative specimens for review to ensure positives are not missed. “They’re always negative. The system has never missed any to our knowledge.” - For slides on which multiple parasites are seen, Dr. Pritt said: “It’s excellent at looking for all the different types of parasites because it analyzes for each of them separately. But there are some categories it clumps together because there are parasites that look similar. That’s understandable. Human technologists have challenges with some of these parasites as well.” That’s why every positive is pulled for manual review.
- The lab does its own internal proficiency testing and has external quality assessment as well. “We enroll in the CAP’s proficiency test. We cannot at this time run it on the Techcyte algorithm because the slides are too thick and not made with concentrated stool. They are not the same system, which is why we also do our own internal evaluation.” They’re testing two things separately, she said: “our manual microscopy skills and our digitalization AI system.”
The success her laboratory has seen in bacteriology with Gram stains, for multiple different specimen sources, has also been seen by other users of the Techcyte systems, Dr. Pritt said. “So I think that’s going to be what we see across the board for the reasons I mentioned earlier. AI doesn’t get tired, it doesn’t get distracted, it doesn’t pause to pick up the phone and forget where it is in the screening process.”
Amy Carpenter is CAP TODAY senior editor.