Anne Paxton
July 2024—A Mayo Clinic project is “democratizing artificial intelligence,” its leaders say, by enabling pathologists with varying levels of AI expertise—most of whom had never before participated in an AI project—to create and implement algorithms.
“If we’re going to implement AI in real practice, we need to broaden people’s knowledge of it,” says Thomas Flotte, MD, a consultant in Mayo Clinic’s Division of Anatomic Pathology and professor in the Department of Laboratory Medicine and Pathology, who led the work and coauthored an article about it (Flotte TJ, et al. Arch Pathol Lab Med. Published online April 23, 2024. doi:10.5858/arpa.2023-0205-OA).
“If you get to play with it and understand a little about how it works,” he says of AI, “then it’s going to make it much easier for people to adopt it. I wanted people to understand what the excitement is about and how we think it’s going to help people.”
Dr. Flotte has been engaged in AI projects for eight of the past 20 years of his pathology practice. His view is that experiencing AI without having to write code will help pathologists who do want to implement AI. At Mayo Clinic, he says, “many of our pathologists are subject matter experts,” interested in developing algorithms but not in learning the details of the linear algebra underlying machine learning and AI algorithms.
In the project, the first step was determining whether Mayo’s pathologists could be recruited to participate. The project team sent an email to attending pathologists to gauge their interest, assuring that project ideas would not be disseminated to leadership or other faculty members, which gave pathologists a secure pathway to relay ideas they were interested or invested in. More than 200 project ideas from Mayo Clinic’s three locations were proposed.
A small group investigated whether the platform should be a vended product or developed in-house and determined that the latter would require too much time and personnel. A request for proposal was developed consisting of six key criteria, among them flexibility and scalability. After reviewing vended (commercial) products that would support development of the algorithms and their deployment in the clinical workflow, the project team chose Aiforia Technologies Plc (Helsinki, Finland), and installed the company’s platform on Mayo Clinic’s cloud environment.
The initial clinical application deployed was Aiforia’s Ki-67 algorithm for breast cancer. Before taking on this project, “We had a small amount of AI with our digital image analysis, which was AI driven,” Dr. Flotte says. “That helped us quantify things,” particularly Ki-67.
The project team accepted 31 of the 45 submitted proposals, with the largest number of projects in hematopathology and pulmonary, breast, and gastrointestinal pathology. Segmentation, classification, and rare event detection were the most common machine learning algorithms chosen for development.
The project team recruited two cohorts from among members of the Department of Laboratory Medicine and Pathology across all three Mayo Clinic sites. Eighty-four users participated in the first cohort, from 2022 to 2023. “The vast majority had never participated in an AI project,” Dr. Flotte says. All users completed training, and every proposal team had its data set digitized. Then they progressed through the model development process of annotating, training, and evaluation.
A strong indicator of success was that, based on their finished AI projects, users in the first cohort submitted 15 abstracts—13 to the USCAP 2024 annual meeting and two to specialty society meetings. “They had one year to do it, to go from not having ever done an AI project to having peer-reviewed, publishable data,” Dr. Flotte says. “So half the projects had gotten to the point where they can publish an abstract. The remarkable thing is how fast they went from zero to 60.”
For the second cohort, which got underway in 2023 and is still in progress, 26 of the submitted proposals were accepted. “We received as many applications, but we had a little less money,” Dr. Flotte says of the smaller number. “But we’re excited.” This round, “the proposals are a little more distributed among the institutions”—12 in Minnesota, nine in Florida, and five in Arizona.
Some of what is being done in the second cohort is an extension of work done in the first cohort, says Lucas Stetzik, PhD, senior scientist with Aiforia and a coauthor of the article published in Archives. “So maybe they started in the first round with, ‘We want to make tissue and these types of cells and this type of tumor.’ Then they said, ‘Let’s see if we can elaborate on that. Instead of just detecting cells in the tumor, let’s see if we can grade the tumor first and then detect the cells.’ So there are a lot of cool projects happening.”
In one of the projects, for example, a deep learning algorithm was developed to predict the risk of endometrioid adenocarcinoma from biopsies showing atypical endometrial cells. The authors write in the Archives article: “There is significant interobserver variation in the interpretation of endometrial biopsies that ranges from benign to malignant. The goal of the project was to develop an algorithm to predict the risk of concurrent and future endometrial endometrioid adenocarcinomas from biopsies showing atypical endometrial hyperplasia.”
Case selection consisted of 122 patients with endometrial biopsy and subsequent hysterectomy. Fourteen cases were used for training, 82 for validation. Initial biopsies were used and the algorithmic approach had three layers: the first to identify endometrium versus other tissue, the second to identity crowded glands in the endometrium, and the third to differentiate crowded glands that were associated with carcinoma in the subsequent hysterectomy versus crowded glands that were associated with atypical endometrial hyperplasia.
At the time of publication of the Archives article, the algorithm had a positive predictive value of 91 percent, negative predictive value of 57 percent, sensitivity of 62 percent, and specificity of 90 percent. As the article illustrates (see images), the trained model did not identify any of the normal endometrium as crowded glands, while the areas of crowded glands that the trained model identified perfectly matches the ground truth annotation of areas of crowded glands. “You can see the program you annotated in the left-hand picture and the right-hand picture showing where the computer version is correct,” Dr. Flotte says.
The project leaders plan to enlarge the training set and add quantitative analysis of certain features to increase the model’s accuracy.
Two tools from Aiforia make algorithm development possible, Dr. Stetzik says: Aiforia Create and Aiforia Clinical Suite.
“Aiforia Create is the tool that allows the pathologists to make their own models and test and evaluate them, so they are put in contact with me and I help them onboard onto the platform.” Aiforia Clinical Suite offers models that have been developed in-house or in collaboration with pathologists at Mayo Clinic.
“Aiforia Create allowed us to get very specific morphological quantifications of microglia on a much larger scale than before,” Dr. Stetzik explains. In his postdoctoral work, before he joined Aiforia, “we made very select, small samples from our images and analyzed them with a custom-made MATLAB script or something adjacent, but we knew there was room for improvement.”
“There was always an issue with sampling by hand,” he continues. “We wanted to minimize any potentially biased results, but because we always chose examples that worked with our script, some bias was unavoidable. With Aiforia Create, we were able to quantify everything in the region that we were interested in and get a much more biologically relevant result.”
“The problem with AI,” Dr. Flotte says, “is that it’s really different from what people have done before. This algorithm is something that pathologists can feel they are creating uniquely with their own skills. It allows people to be both the subject matter experts and the data scientists.” Mayo Clinic could be alone among academic medical institutions, in his view, “in trying to get everybody involved,” as it has done with this development project.
Aiforia’s AI differs from Chat GPT. “Fundamentally, Chat GPT is trying to predict what the next word is going to be,” Dr. Flotte says. “The underlying technology of ChatGPT is natural language processing.” Aiforia’s program is a no-code visual interface, he says. Convolutional neural networks (CNNs) are “the underlying fundamental technology at work in the use of AI to develop algorithms. They are what many image analysis software algorithms are based on.”
CNNs are central to developing algorithms from annotations, Dr. Stetzik says. “CNNs are what we are using to train the models. They’re all computer vision algorithms that are able to extract features from the images using different types of sampling. So it’s a much more abstract process than quantifying based on how dark a tissue is here versus here.”
Having an annotated data set is 80 percent or 90 percent of the work of the AI, Dr. Flotte says. So for participating pathologists facing a computer screen, “the majority of what they’re doing is annotating the images they’ve uploaded. That’s the dull work that goes into it before you can do the fun part”—watching what happens when the model is trained on the annotations.
The algorithms that pathologists develop using Aiforia’s Create and Clinical Suite are their own intellectual property, Dr. Flotte says. “There’s no restriction because the company doesn’t own any of the intellectual property rights in our contracts.” Could those algorithms be sold in the marketplace? Yes, he says, but at this time the algorithm would be limited to the Aiforia platform and could not be used with the platforms of other companies. “Within the informatics community, there’s a whole group of people trying to figure out how to make algorithms interchangeable between different systems,” Dr. Flotte says.
The more people who do AI, the better it will get, he says. “There are going to be more and more of these algorithms out there, and then one of two things is going to happen. Either somebody’s going to develop that algorithm that everybody wants, or you’ll have what I think is more likely: a suite of algorithms looking at infectious disease, not a single algorithm. You’ll have one that can look at genomes, one that can look at AFB stains. The combination of all those algorithms will become what people actually want.”
It’s Dr. Flotte’s hope that algorithms in the future will be more independent of the commercial platform because of the appeal of such combinations. “If you have many of the medical centers across the country developing 10, 20, 50, 100 algorithms within their own institution, there will be enough of a groundswell of this sort of data that people are going to want that to happen.”
What the democratization project leaders tried to do was to create a road map, Dr. Flotte says. “But ultimately you’d prefer to have algorithms be developed based on patient outcomes.”
Potentially, others can compensate the pathologists developing each algorithm for use of their product, Dr. Stetzik says. For now, the algorithms are tied to the Aiforia platform. “I know there are groups at Mayo who are looking for ways to make their models more accessible, but that requires extra backflips on the data side where they’re not really using our platform anymore. A lot of what is being developed is staying within the Aiforia platform, so in that way it is still vended.”
But Dr. Stetzik doesn’t see that as necessarily limiting pathologists’ future development of algorithms. “The pathologists who are participating are providing the deep learning. In some cases they’re just providing that ground truth. But in all cases they are working the platform, and once they’re oriented they can do it all on their own. They may want our expertise and our input along the way or they may just go for it themselves. We do see both happening.”
Having a computer science background, Dr. Stetzik has found, may not be that helpful for users of Aiforia’s platform. “Sometimes the people who have a computer science background struggle because there are discrepancies between the theory they’ve learned or other methods they’ve used and the practice of building models in our platform.” With a clean slate, “you are kind of fresh, no bad habits.”
Of AI in general, he says, “There have always been skeptics. There’s a serious point of view that AI is concerning and possibly an existential threat to us as a society.” He does not wish to dismiss that belief but says, “In any talks I give now, I’m trying to get people to understand that’s not the direction, and the most ‘dangerous’ aspect of AI is that it is genuinely empowering.”
AI is “a tool that you can do something with”—one that will significantly advance pathologists’ scope, Dr. Stetzik says. “It requires a bit of learning ahead of time. And I think that is the part that makes it a little unpredictable and the part that, if anything, should be taken very seriously.”
Anne Paxton is a writer and attorney in Seattle.