How Pap test automation is playing out

October 2006
Cover Story

Karen Titus

For decades, Pap tests were done like factory piecework, using conventional smears on a slide that were then read manually. Now they aren’t, thanks to a long line of entertainingly spelled companies and products that have transformed the field.

Neopath, Neuromedical Systems, CompuCyte, AccuMed, AutoCyte, TriPath, Pathfinder, PAPNET, AutoPap, ThinPrep, Cyto-Rich, TracCell, AcCell, FocalPoint, SurePath, PrepStain, Cytyc... It’s not unusual to hear pathologists reel off several of these in rapid succession before landing on the correct name of their current instrument. Sometimes they even mix in an entirely different name: Digene. Digene, of course, is the maker of the Hybrid Capture HPV test, and there’s good reason for labs to include it in their discussions of Pap testing.

Such mix-ups reflect the ongoing problems and questions in the field. Ironically, as the industry looks to automation to standardize matters, lab responses have been anything but cookie cutter.

Gene F. Pawlick, MD, mixes the past with the future in his present role as medical director of Permanente Medical Group Regional Laboratories, Berkeley and Richmond, Calif. At Kaiser, Pap tests are done using conventional smears that are then read on the FocalPoint automated screener; at the same time, women 30 or older receive HPV testing. This blend of technology appears a little odd at first, like filling a Toyota Prius with diesel fuel. But it’s what works for Dr. Pawlick and his colleagues as well as their patients.

Kaiser began looking at changes in Pap testing when liquid-based cytology entered the marketplace. The lab looked first at Cytyc’s setup, doing two fairly lengthy studies of the company’s ThinPrep slides on Kaiser patients. Dr. Pawlick says he and his colleagues were quite enamored of the quality of the smears and the perceived increased benefit to health plan members.

As they were wrapping up their studies, they learned of an automated instrument for screening Pap smears, NeoPath’s AutoPap system (now TriPath’s FocalPoint). Following another lengthy study, Kaiser chose the latter instrument, which can handle both conventional smears as well as liquid-based samples (SurePath, formerly known as AutoCyte).

In its original study, says Dr. Pawlick, the lab picked up 53 additional cases of HSILs or carcinoma in situ in one year. "It was a significant increase," he says. "Let me put it another way: If we hadn’t been using that [the automated instrument], these are cases that probably would have been missed. I’m not trying to throw a stone at anybody. I’m just saying that the technology is really good."

Kaiser is one of the few holdouts, and possibly the largest, to stick with conventional smears. Early on that earned the lab the same kind of scorn normally accorded members of the Flat Earth Society. "I want to say one thing," Dr. Pawlick says. "We have been lambasted and plastered and painted yellow, and chicken, and red, had all kinds of epithets hurled at us because we did not get on the bandwagon with liquid-based cytology."

Kaiser also had to deal with expectations arising from direct-to-consumer marketing, which meant Dr. Pawlick had to explain himself to plenty of patients. "There was a period when I was getting a call maybe twice a day," he says. "And I have a lot of filters—this is a very large laboratory, so there’s about 600 people between me and the docs and the public. By the time it filters through, if Im getting two calls, that means other people are getting 20 complaints a day."

Such calls have died down, and Dr. Pawlick thinks that’s significant. Not only are the clinicians happy with the setup, he says, but also his highly educated, medically savvy patients. Or, as he puts it, "This is the Bay Area, and Silicon Valley, and our patients are aware of it, and thinking about it, and probably enjoying a cocktail as they discuss it," he says with a laugh. "And I don’t hear the questions I was getting two years ago. This whole thing is just kind of humming along for us, like a well-run Swiss watch."

He and his colleagues did revisit the issue at one point, however, because of the acrimony that arose from their decision to stick with conventional smears. "For us, there was no statistical difference in the pickup rate using Cytyc over our cytotechs.

"That’s a strong statement," he continues. "And it’s only applicable to our laboratory—I’m not making that claim for any other laboratory. But for us, it made no sense to switch." Though he and his colleagues liked the quality of the Cytyc slides, the liquid-based samples didn’t significantly improve health, reduce cost, or improve productivity. "Cytyc doesn’t like me to say that, but that’s the case," he says. "We fought a lot of wars to get where we are, and we feel pretty strongly about what we’re doing. And I think we’ve got very good evidence that it has paid off in terms of women’s health. And that’s the bottom line. I sleep very well at night, if this is the question I have to ponder."

They must be doing something right. David Wilbur, MD, director of cytopathology at Massachusetts General Hospital, Boston, and head of the CAP Cytopathology Committee, speaks admiringly of Kaiser’s approach to managing cervical health, noting that with the triad of conventional smear, FocalPoint screening, and HPV testing, "I bet they’re probably at as high a standard of care as anyone in the country."

Kaiser’s unusual approach sets the tone for other labs that are looking at some form of Pap automation. As the various technologies have moved into labs across the country, one might expect an answer to emerge about what the best system is; in fact, the best answer may be: There are many best answers. Each lab needs to find out for itself what works best, as Dr. Pawlick’s lab did.

If no consensus is emerging, a number of individual stories are. Hype is slowly giving way to everyday experiences, told with a peculiar mix of reticence and bluntness usually found only in small towns and certain families. That’s why you’ll hear someone like Dr. Pawlick saying his lab found no statistical difference in sensitivity rates between conventional smears and liquid-based cytology. That’s also why you’ll hear any number of pathologists say they simply don’t know if reality has caught up to all the promises, no matter how good the technology may be.

Dr. Pawlick finds a number of advantages with Kaiser’s setup. The FocalPoint system, which detects morphologic changes, ranks slides according to their likelihood of being abnormal. That means a certain percentage of cases (at Kaiser, it’s around 20 percent) can be declared negative and archived for "no further review," without the need for cytotechnologist review. Given the ongoing cytotechnologist shortage, "this has been a godsend," he says.

"Another advantage of TriPath FocalPoint Neopath, whatever you’re going to call it," he continues, "is that it picks up significantly more cases than the cytotechs alone can pick up." In addition, he says, the system can be used for directed rescreening of negatives to comply with CLIA QC requirements. Doing this the old-fashioned way—at Kaiser, the laboratory information system pulled slides randomly—meant most rescreened cases were negative. With the FocalPoint system, Kaiser can use a prescreened ranking system to pull negative slides at higher risk. "Statistically that’s where the danger zone is, and that’s where we find more missed cases."

Dr. Wilbur’s lab uses both the Cytyc and the FocalPoint systems—not as odd as it sounds, given its setting in a large academic center—which should put him in a position to compare the two. But that’s not going to happen anytime soon, in part because the two systems are quite different.

Cytyc’s ThinPrep Imaging System, like FocalPoint, is a computer-assisted morphology-based system. Unlike FocalPoint, it also uses a proprietary DNA quantitation stain to evaluate DNA content of nuclei; in addition, it has received FDA approval for its location-guided screening device, which directs readers to 22 areas of greatest interest on the slide.

TriPath’s location-guided screening has yet to be approved by the FDA—the company withdrew its PMA for this application last fall and is redoing its clinical trial, say several people familiar with the situation. One estimated the company would resubmit its PMA by the end of the year; another noted, "It’s not hard to imagine that sooner is better for this company. Everybody would like to see this happen, or at least get decided in a timely fashion." Becton Dickinson is set to purchase TriPath Imaging by year’s end, however, and it’s not clear how or if that will affect the situation. Until—or if—the FocalPoint LGS receives approval, it’s impossible to talk about its performance characteristics.

That leaves someone like Dr. Wilbur to compare a location-guided instrument to a triage instrument. Both, he says, have their plusses and minuses.

Dr. Wilbur is a fan of location-guided screening, saying it improves productivity and potentially increases sensitivity. Not that he’s seeing those benefits maximized in his lab now. His high-risk population and approach to practice has led to lab protocols that require manual QC rescreening on a significant number of slides, which diminishes the efficiencies generated by the Cytyc Imager. "We’re thinking about whether we want to change the population that’s actually going through the Cytyc Imager at the present time, from one of our more higher risk clinics to one of our lower risk clinics."

At this point, the Cytyc instrument has been in place only a short while, which also makes it difficult for Dr. Wilbur to comment on whether it has reduced his lab’s false-negative rates.

In some ways it’s easier for him to talk about the FocalPoint system, since they’ve used it longer—four years. Echoing Dr. Pawlick, he notes that a fifth of his lab’s cases don’t get seen by a human being as a matter of course. "This leads not only to improved productivity but also improved specificity," he says. "You can’t have a false-positive if a human does not look at the slide." In this extremely low-risk population, "false-positives are the errors that occur if you actually look at the slides, which can lead to overtreatment." Those that are reviewed by a human being, Dr. Wilbur says, are enriched with abnormal cases, improving vigilance and leading to better screening.

In the Escherian logistics that seem to characterize much of Pap testing, however, labs can’t actually use FocalPoint for high-risk cases.

No one wants to talk about this, really, at least not for attribution. That’s because no one wants a surprise visit from the FDA. Here’s where Pap test talk veers from small-town candor and starts to resemble Peyton Place.

When FocalPoint received its original approval for use with conventional smears, high-risk slides were not part of the equation. The labeling that accompanies approval for SurePath’s use on FocalPoint, however, says that high-risk slides were included in the clinical study to extend FocalPoint approval. And that, say some users of FocalPoint, logically means high-risk SurePath slides can indeed be run on the instrument. At one lab, high-risk slides are run on FocalPoint, then undergo a manual screen, as a sort of internal QC. "We think the advantage of having the FocalPoint tell us, even in the high-risk population, that it’s the highest quintile, is a really good way to more sensitively screen the slides," says a pathologist at the lab.

Another lab loads its high-risk slides into the FocalPoint because risk status isn’t known until late in the screening process; once the high-risk information from the LIS catches up with the samples, those samples are removed from the machine—a step that doubtless lowers efficiency. It definitely reduces the percentage of "no further review" cases. The instrument is set up to accept up to 25 percent of such cases (though in Europe, many labs push that to as high as 50 percent); in this lab, the rate would normally be 23 to 24 percent. Losing the ability to run high-risk cases brings the number down to about 18 percent.

In this lab’s view, running high-risk cases on the FocalPoint makes nothing but sense. The technologists would know not only that the slides are high risk but also their status when they’re screened. That leads to an extra level of review, because all high-risk cases are automatically rescreened. "But it’s also not exactly the way the instrument was originally approved for use," concedes a pathologist at the lab.

Not running high-risk slides on the system is counterintuitive, say those who disagree with the FDA’s apparent limitations on screening high-risk slides. If automated screening is an improvement, why would the highest risk cases be shunted to a less-effective method?

"Those slides are getting the short end of the stick," says one observer. "The high-risk patient deserves to have the most sensitive procedure." The FDA has argued that including high-risk cases dilutes the population, pushing all other positives down lower in the slide rankings, he says, but at his lab, at least, "We don’t have any evidence that that’s what happens." For this to play out, "the positive rate in any given FocalPoint run would have to be much higher than we normally see, a situation that is theoretically possible but practically very unlikely," he says. In addition, a common misunderstanding is that this situation might cause some abnormal cases to be pushed down into the no-further-review, or NFR, category and thus not be screened manually. This doesn’t happen because once a slide scores above the fixed threshold for the NFR population, it can never fall into that group. "The NFR number in that run would just be reduced in size," he says.

Oddly, there’s also a certain amount of reluctance to talk about increased screening accuracy, though ostensibly that’s the whole point of automation. "It can get pretty incendiary," says Dr. Pawlick.

Anecdotally, users say they like both systems—and that neither affords huge improvements. The reason, they say, is obvious—there wasn’t much room for improvement to begin with. Lesser labs might show bigger gains, but are probably reluctant to talk about them. Big improvements are grounds for suspicion, not celebration.

"If I were a person who was having my Pap done in a laboratory that noted a huge increase in HSIL detection with the use of automation, I would be really concerned about how many Pap smears had been misread in the past," says Dr. Wilbur.

At labs where cytotechs have historically been pressured to look at more slides, says R. Marshall Austin, MD, PhD, "What we’re hearing is that they’re seeing very dramatic increases in the detection of abnormality, to the extent that now it’s hard to get them to even talk about it."

This makes it hard to evaluate the systems’ effectiveness, though not impossible.

Dr. Austin is professor of pathology and director of cytopathology at Magee-Women’s Hospital of the University of Pittsburgh Medical Center, which uses the Cytyc Imager. "We have basically seen the Imager lift all the cytotechnologists into categories where their false-negative rates are in the lowest ranges. It’s kind of like Lake Wobegon, where all children are above average," he jokes.

Data from Heather Mitchell, MD, in Australia, among others, show that if a slide does not have more than 50 abnormal cells on it, it’s 25 times more likely to be called falsely negative than if it has more than 50 abnormal cells, Dr. Austin says. "What that tells you is that with conventional screening techniques, you need to have 50 abnormal cells on the slide before one abnormal cell is reliably detected. And that’s kind of scary." Computer-assisted screening "is definitely a powerful tool," Dr. Austin says. "I think if it was really understood what a valuable tool it is, it would be pretty much required." In his own lab, the most recent six-month false-negative fraction was 2.5 percent—the lowest it’s ever been.

"Given the rapid advances occurring in modern computer technology, it seems intuitive to me that computer technology developed in the last few years would have a significant advantage over computer technology developed in the mid-1990s," Dr. Austin says.

A solid literature might provide more answers, but "the literature," says William Tench, MD, "is a little disappointing."

Like everyone else, though, Dr. Tench, who is associate director, Palomar Medical Center Laboratory, Escondido, Calif., says he’d be suspicious of any study that showed mind-boggling improvements, such as a doubling of HSIL pickup rates. "I wonder, if they are finding a hundred percent more high-grade SILs than they did before they started using the instrument, what happened to all those patients?" More common, he says, are higher rates of LSIL pickups, which are less clinically significant.

Adds Dr. Austin, "If you have a huge increment in abnormal detection rates, it’s probably a reflection of missing significantly more on the front end. I’d be reluctant to brag that we had the largest increase in abnormalities. But we have seen some." Some of these are cases moving from an indeterminate abnormal category to a more specific diagnostic group; others are cases with very few abnormal cells that previously were undetected.

Even if a large literature were available, however, it would be a challenge to parse.

In the mid- to late-’80s and early ’90s, Pap collection devices moved from cotton swabs and spatulas to endocervical broom and endocervical brush/spatula combinations. Much of the data collected on conventional Paps—data against which newer technologies as well as HPV tests have been compared—are based on samples gathered via older collection devices, which are not nearly as good as the newer ones for detecting transformation zone lesions. Dr. Wilbur suggests that any historical study showing a huge increase in detection be taken with a grain of salt. "When you start throwing that variable in, I can’t even begin to tell you whether the early data on the transition between conventional and liquid-based has everything to do with the liquid-based method. It’s very possible that a significant component of any improvement is sampling device."

That being said, he acknowledges that most literature shows that moving from conventional to liquid-based improves detection of HSILs and LSILs. "That is the average experience"—the caveat being, he reiterates, that in labs that used optimized (that is, ethanol liquid immersion fixation) conventional smears, the impact is less than dramatic.

Dr. Wilbur has the added experience not only of moving from conventional to liquid-based samples, but from one liquid-based type (ThinPrep) to the other (SurePath). In his view, there’s not much difference between the two liquid-based methods, though it doesn’t take much effort to find users of each who’ll persuasively argue their choice.

Head-to-head comparisons are nearly impossible because neither instrument is set up to accept samples from the other. Dr. Tench reports there are methods for converting specimens from one type to the other—much like enabling software to run on both Mac and Windows—but "there’s a lot of discomfort with that. It’s off-label with the FDA," he explains.

"Some of us know quite well that the conversion process of making a satisfactory SurePath preparation from a ThinPrep vial is really a very simple process," Dr. Tench continues. "And it works quite well." But adoption would require hefty validation efforts. "And given the medical-legal environment association with bad outcomes on Pap smear interpretations, people are going to be uncomfortable going off in that direction anyway."

That leaves the vendors to duke it out for new customers and to persuade others to make a switch, though the latter may be less likely. A ThinPrep user will find it attractive to stick with that technology and to use the Imager for automated screening. SurePath users will find the FocalPoint a more convenient choice. And if both are better than conventional methods, who’s to say either decision is a bad one?

What’s needed is one of the very large commercial cytology labs to say, "’OK, now we’ve done a million smears with the ThinPrep Imager, here’s what we found,’" says Dr. Wilbur. You don’t need a degree in statistical probability to figure out the odds of that happening. "The last thing such a lab might want to do is say, ’Hey, we got 98 percent of our stuff right!’" Dr. Wilbur says. "Because you’re going to have trial lawyers saying, ’OK, tell us which two percent you got wrong, and we’ll contact the patients.’"

Academic centers such as Dr. Wilbur’s are left to fill the gap. In the meantime, labs are relying on the clinical trial studies and, perhaps, marketing literature and an eager sales force.

"Our understanding of all this is still evolving," says Edmund Cibas, MD, director of cytopathology, Brigham and Women’s Hospital, Boston, and associate professor of pathology, Harvard Medical School. "More people have to use it, and there has to be more peer-reviewed publications before people will have a good grasp of how much more accurate the computer-assisted screening is compared to manual screening.

"We’re still just beginning to publish on our experiences," Dr. Cibas adds. So far, he notes, the peer-reviewed literature on the Cytyc Imager has been favorable. "It’s still early in the game, and it’s always a little uncertain when you can’t point to a lot of peer-reviewed articles supporting either the productivity enhancement and/or improvement in sensitivity." The FDA based its approval of the Imager on what Dr. Cibas calls a "compelling" multicenter clinical trial. "But, for many labs, you’re still relying in large part on the sales force to tell you, ’This is a better instrument,’" he says.

Pathology & Cytology Labs, Lexington, Ky., a user of the Cytyc instrument, recently completed a six-month study (now being extended, for a total of two years), which saw a 48 percent increase in low-grade detection and 36 percent in high-grade detection.

Richard Lozano, MD, director of cytology, knows he’d be fielding plenty of complaints if the lab were reporting high-grade cases that weren’t confirmed on biopsy. But that hasn’t been the case. "They’re seeing more disease, and it’s true disease, and so they just go about their business. We have great biopsy correlation data."

Dr. Lozano hasn’t seen large improvements in productivity, which he says surprised him. He has noted more consistency in workflow, however. "We’d be behind a couple days, then catch up. With the Imager, we seem to stay caught up all the time. That’s huge in the eyes of our clinicians."

If Dr. Pawlick’s lab represents one unexpected destination for Pap testing, Dr. Tench’s may soon demonstrate another.

Palomar Medical Center Laboratory adopted the Neopath system after its initial approval, for QC screening, and it has continued to use the instrument as it became approved for liquid-based samples and primary screening. Early on, says Dr. Tench, his lab’s annual Pap volume was between 35,000 and 38,000 cases. Today, it’s a push to make 6,000 or 7,000, a drop that he says is due to contractual issues with local health maintenance organizations.

At his current volume, he would be reluctant to bring an automated system onboard. The instrument, brought into the lab on what amounted to a lease-to-own contract, now belongs to the lab. But the annual maintenance contract is not cheap, and at lowered volumes, paying for it "is an iffy thing," Dr. Tench says. If Pap volumes drop further, "then quite possibly we would have to say, ’Hmmmm, we don’t know if we can afford to do this.’ And we’re fairly close to that right now." Worse-case scenario: "We might just say, ’We’re not going to do Pap smears at all.’ Because I’m not willing to give up the technology. I believe the technology has a lot to offer, and if we cannot make use of it, then I think that we will probably get out of the business altogether."

This Pap-less scenario may play out in other labs if they can’t afford automation. It’s not clear, however, what volumes make the most fiscal sense.

Owensboro (Ky.) Medical Health System, with three staff pathologists and two cytotechnologists, became the first lab in its region to install the Cytyc Imaging System, says Chris Ross, SCT (ASCP), lead technologist in cytology. Current annual volume is less than 18,000, which he acknowledges is low. But the lab has already picked up one new account, which should boost monthly volume by about 250 cases, and more may be in the offing.

It’s also possible several small labs could combine forces and share a system, partnering on the technical component but maintaining their own professional billing. While such arrangements may not sit well with independently minded labs, the alternative is hardly better.

How bad would it be to stop doing Paps? Says Dr. Wilbur: "I can tell you, in a training environment, we certainly want to keep doing Pap smears, although every now and then I wonder whether our administrator’s going to come down and say, ’Hey, we’re sending all our Pap smears to Laboratory X starting next week, so we don’t need you anymore.’" He laughs before sounding a more serious note. "But it’s really part of the core of anatomic pathology. We teach all of cytology based on the methods and extensive experience we, as a field, have in gynecologic cytology." Also, in a best practice, a lab would be able to directly correlate its cytology and biopsy results, and keep slides in-house for QA review. "If your Pap smears are read in Dallas, and your biopsies are read in Tacoma, that’s not optimal."

The other, down-the-road possibility is that the traditional role of Pap tests will go the way of the mastodon—important, interesting, and ultimately unsustainable. In this scenario, HPV will evolve as the primary player. Even the Pap manufacturers are hedging their bets, looking for their own HPV molecular markers.

But useful HPV data are scarce. "We have 50-plus years of data using the conventional smear," says Dr. Lozano. "And when scientists or policy makers talk about sensitivity and disease detection, a lot of them still use the figures or the data for the conventional Pap smear. What they need to do is recalibrate their thinking to take into account imaged-assisted, or computer-assisted, cytology."

Dr. Cibas recently finished a study looking at the frequency of HPV positivity in 1,000 women 30 or older whose Pap tests were negative using the ThinPrep Imaging System. The rate, as it turns out, was quite low: 6.7 percent in women ages 30-35, 3.0 percent in women 36-40, and 2.6 percent in women 41-45. The abstract was presented at the ASCCP meeting in Las Vegas in March.

The rate might even be lower, Dr. Cibas speculates. The overall 3.9 percent rate was based on the Hybrid Capture test; when those positive cases were examined again, by PCR, some turned out to be false-positives, and others were determined to be low-risk rather than high-risk. However, those data are still preliminary, Dr. Cibas cautions.

It’s one study, but it helps fill a big gap. "I was surprised to discover there really hadn’t been a study to look at how often an HPV test is positive in a woman who has a negative Pap smear," says Dr. Cibas. "If you’re going to advocate for doing the HPV test with the Pap smear, there should be some added value to that HPV test."

Dr. Tench says he’s somewhat surprised that vendors of the automated screeners have devoted the amount of resources they have to develop their instruments, though the reasons for doing so are understandable. At the time, molecular HPV testing and the HPV vaccine must have seemed like the faintest of promises.

That’s not to say an investment in automated screening isn’t justified. It’s quite possible, given the companies’ interests in molecular markers, that the screeners will be used for molecular tests on the liquid-based samples. "But I really don’t think that in five to 10 years, pathologists in this country will be screening Paps in the same way we’re used to," Dr. Tench says. "I think there’s going to be a huge migration, and molecular will be the end result. If I were going into cytology now, I would not be investing a huge amount of my professional life in learning GYN cytology—I would be learning all the molecular stuff that’s going to be coming down the road."

No one is holding up a sign that reads, "The end is near." But as labs have shown, the lines between cytology’s past, present, and future are closely intertwined, which could make the end difficult to recognize even if it does arrive.