Sherrie Rice
September 2026—Technology and regulation have always co-evolved, and artificial intelligence should be no exception.
“The tech industry would have Americans believe there should be no regulation of anything that has AI in its title until such time as the technology is fully matured and ready to be regulated,” says Brian R. Jackson, MD, MS, professor of pathology, director of clinical pathology, and CLIA director of laboratories, University of Maryland Medical Center.
He calls it “complete gaslighting.”
A better approach, Dr. Jackson says, is for the core technology and regulatory scheme to be developed in parallel.
Driving is an example. Automobiles have been around for about 130 years, and laws around driving, traffic, and roads have evolved over the same period. “The interplay between the two sides creates a much better, much safer situation than if you let the technology run wild and then try to come up with a regulatory scheme to bring it under control.”
“The more risky or dangerous the technology is, or the larger its consequences,” he says, “the greater the variety of regulatory approaches we need to have in place because we need it to work for us properly in society.”
With AI, too, he says, the technology and the regulation need to co-evolve. “But Pandora’s box has already been opened, and it’s hard to rein things in.”
Dr. Jackson, a member of the CAP Artificial Intelligence Committee, was speaking last year in a CAP25 session, where he acknowledged that conversations about AI tend to center on its exciting possibilities. His focus, though, was one of caution: He described a hypothetical but realistic scenario about AI, privacy, and pathology practice to illustrate how AI can pull things in the wrong direction.
In the scenario, CancerNostics.com, a fictional venture-backed startup, approaches a pathology group to pitch a deal: The startup founders get exclusive access to the practice’s de-identified slides and pathology reports. Their plan is to digitize all slides and create AI diagnostic software. The pathology group partners would get an up-front payment plus stock options.
“I know these deals get pitched from time to time,” Dr. Jackson said, adding, “I’ve talked to pathologists who have had these kinds of pitches. And the pitch is that in exchange for digitized copies of all of your slides along with your reports, they will give you money and/or equity in their company.”
The CancerNostics founders making the offer assure the pathologists all patient identifiers will be removed, that only the histology and diagnosis are needed to create the AI software. In addition, they say an NDA will prohibit disclosure about the arrangement.
The pathologists talk about the deal, consult their attorney, decide it’s legitimate, and proceed with the deal.
What might happen: The pathologists discover down the road that CancerNostics has purchased de-identified medical records from all hospitals in their region. “So now they [the founders] have lab, medication, and diagnostic data, maybe some other things—all de-identified, and this is legal under HIPAA,” Dr. Jackson says.
The same company has also purchased de-identified claims data from the major health insurers in the area. “With the magic of AI, it turns out you put those data sets together and the AI can figure out 99 or so percent of the cases to re-identify them, to tie those data sets together.” What might they do with it? Use it for marketing purposes or sell it to a pharmaceutical company, for example, that wants it for direct-to-patient marketing, leaving a patient who has received an advertisement in the mail wondering how a company learned they had a specific type of cancer. “Again, this could be easily done,” Dr. Jackson said.
The scenario is not only a plausible one but also a real one, citing a May 23, 2022 story in STAT titled “How a Complex Web of Businesses Turned Private Health Records from GE into a Lucrative Portrait of Patients.”
“It was with one of the ambulatory care electronic health record companies that happened to be owned by GE at the time,” Dr. Jackson explained. The EHR company had sold de-identified medical records from many practices to a third party and then discovered that same third party had gone out to a different source and acquired claims data and used it to re-identify everything. GE executives found out this happened and feared they would be held liable for a HIPAA violation.
“Their attorney said, ‘GE is totally fine here,’” Dr. Jackson said.
It’s one of the “big loopholes in HIPAA,” which was written long before the big data and processing power were available, and says, “If you strip out identifiers, what’s left is not subject to HIPAA. Therefore, it’s salable. You can give it away,” he said. “You can do anything you want with it, with no further repercussions, even if someone down the road who’s not covered by HIPAA happens to re-identify it all.”
Those in hospitals and health systems, academic centers, and research enterprises have long operated under the assumption that if data is de-identified, “we’re okay,” Dr. Jackson said, “and IRBs have been going from that perspective as well for many years—but without considering what could happen downstream.”
He returned to the hypothetical CancerNostics scenario. In addition to the re-identified data being used for marketing, there is another possible use for the data: employment recruitment. CancerNostics could sell the data to the hypothetical RobotRecruiters.com, which could use it to screen out candidates with a history of cancer or other serious illness or of high medical expenditures—a family member with cancer, for example. In these circumstances, a patient who wonders why they are not getting responses to job applications won’t know it’s because AI uncovered they had cancer.
Taking the scenario further: The hypothetical company could sell the data to a company in the medical malpractice space, the hypothetical Torts-R-Us.com. It’s a data services provider to the legal and insurance industries that will use the data set to identify pathologists who might have a higher error rate than that of their peers. “Now they’re using the practice’s pathology data to try to identify candidates for malpractice lawsuits by looking at pathology accuracy patterns,” Dr. Jackson said, and the puzzled “patient” in this scenario is a pathologist wondering why his or her malpractice insurance just rose despite no claims having been filed.
Dr. Jackson said data re-identification is not only possible but common, used widely in marketing. “It’s essentially the same technology that makes you wonder, Why am I getting these Facebook ads or these ads on Google that feel so creepy? The technology is there for the companies to know an enormous amount about you just by piecing different data sources together, even if no one data source has the complete picture on you or could positively identify you. And you can do the exact same thing with de-identified medical data.”
The U.S. has weak privacy laws, he said, and he has little confidence it will change in the near future. From an ethical standpoint, then, his advice is this: If you’re in a position to make decisions about sharing your practice’s data with third parties, at a minimum spend time coming up with contractual terms to ensure that downstream re-identification, reuse, and re-sale are prohibited. In addition, there should be a contractual term that says data will be destroyed at the conclusion of a project or if the company is acquired or bankrupt. “And it should be auditable and enforceable.”
Dr. Jackson’s second case, using the same hypothetical CancerNostics, was a scenario related to bias, errors, and patient safety. The company does what it said it would do: build a pathology image analysis app by training on biopsy slides and reports from the pathology practice. The company selected a high-volume area, GI biopsies, and developed software that could assign diagnoses based on histology. It obtains Food and Drug Administration approval as a diagnostic device for tubular adenoma. Approval is not sought for other diagnoses or conditions.
A hypothetical private equity firm, BlackHunk Partners, purchases the pathology practice unexpectedly. It owns other pathology, gastroenterology, and dermatology practices across the U.S. The firm promises the pathology group when it purchases the practice it will not interfere with medical decisions. It promises to prioritize care quality and preserve professional autonomy while improving back-office operations. “In some cases,” Dr. Jackson noted, “this is a matter of law because some states still have corporate practice of medicine laws that say the owner can’t interfere with medical decision-making.”
BlackHunk might have equity in CancerNostics, so the new owners install the image analysis software on the desktop computers of the pathologists and encourage them to use the app on all GI cases, Dr. Jackson said. When the pathologists ask how the app was validated, the answer might be that it’s FDA approved so validation is unnecessary. Another, or second, answer: “Trust us. We’re monitoring performance and are confident because the software company told us it’s going to be accurate,” Dr. Jackson said.
If any type of pathology AI app will play a role in the pathologist’s diagnostic process, in his view, it is the pathologist who should oversee its quality and accuracy. Historically, however, when it comes to software, it is the IT department that oversees it. “This is the way electronic health records work,” he noted. “They’re clinical in nature, but who’s monitoring the quality? The IT department.” But when it comes to software that will be used in generating diagnoses, “I think pathologists need to be officially and practically in charge” of its oversight.
Dr. Jackson and others coauthored a perspective piece on how CLIA’s framework for regulating medical technologies with local oversight can be extended to clinical AI (Jackson BR, et al. J Am Med Inform Assoc. 2025;32[2]:404–407). They explain how the CLIA model could be adapted in terms of licensure, risk stratification, personnel requirements, local validation, proficiency testing, and continuous monitoring and improvement, among other things. “By extending a CLIA model to AI,” they write, “we can achieve substantial improvements in quality, safety, economic efficiency, equity, and speed of AI adoption.”
A laboratory test and AI software used for diagnosis are both highly complex tools that can go wrong in multiple ways if they’re not carefully managed and quality controlled, Dr. Jackson said in his talk.
“So in the same way that we have CLIA, which says every laboratory needs a doctoral-level medical director who is officially responsible for the quality management system, we need this in health care IT. And it doesn’t exist today.” Many hospitals have chief medical information officers, but most CMIOs don’t have the same signoff responsibility on quality that a laboratory director has for a clinical laboratory, he said.
“They tend to be some combination of physician liaison, technical advisory, maybe a business kind of role. But in terms of an official quality role that the CMIO has to sign off on the software having been properly quality managed, which we have to do in our laboratories, it doesn’t exist. It doesn’t exist in radiology. It doesn’t exist anywhere in medicine other than clinical labs.”
Those who are familiar with the way CLIA works should advocate for an analogous regulatory approach to be developed, in his view. “It will probably require legislation, but my personal opinion is that this is the direction we ought to go with AI in health care, and certainly with AI in pathology.”
Back to BlackHunk. “AI can enable all kinds of creative ways to do clinical practice,” Dr. Jackson noted, and he offered more. BlackHunk could begin to supply the CancerNostics software to its own GI practices and encourage the gastroenterologists to use it themselves to interpret their patients’ biopsies, bypassing the pathologists. BlackHunk could also urge its pathologists to process more cases more quickly by using the AI software, putting accuracy at risk.
To sum up, he said AI is not a purely technical topic. “AI is also a business and organizational and quality topic, and these things are interconnected in ways that can be hard to tease out. And all of these things are integral to the practice of medicine.”
AI is a powerful tool. For that reason, it needs comprehensive quality management, Dr. Jackson said, and yet “the field of medical AI is really, really immature right now.” It has taken decades for the world of pathology and laboratory medicine to figure out how best to quality control all of what it does, he noted, and “we’re only a couple of years into the medical AI world, so none of that is in place.
“It’s an opportunity, yes,” he continued, “but it’s also an area for a lot of caution in the meantime.”
A final warning: The corporations involved shouldn’t be trusted to do what is needed to protect the interests of patients and pathologists, he said. Take the stock market, where many of the largest companies are AI in one form or another. “There’s a lot of money behind the technology development, and these companies, if there’s not a strong professional counterbalance, will take things in whatever directions are to their benefit.
“So if we allow corporations to do anything that would erode the professional role of pathologists,” he said, “that is toothpaste that can’t be put back into the tube.”
Sherrie Rice is editor of CAP TODAY.