Feature Story

Beefed-up QC gives rise to ruckus

January 2001
Anne Paxton

In 1994, when the emergency room at Methodist Hospital of Indiana was pushing to have a stat laboratory, the pathology department looked at history and refused to go along. "The ER had a stat lab before. One of the reasons it was taken out was because they didn’t document quality control," says Catherine Cox, MD, who was then a pathologist at the Indianapolis hospital.

So when the laboratory eventually agreed to look at point-of-care devices for electrolytes and blood gases in the ER, it insisted on requiring electronic documentation of results. "We already knew they weren’t good at writing down the date, time done, patient name, result, and QC details," Dr. Cox explains.

As at Methodist, hospitals using point-of-care testing throughout the country find that meeting quality assurance and quality control standards continues to be a challenge. But possible regulatory changes affecting waived POC testing have been drawing sharp criticism in recent months. The changes suggest that although there can never be too much excellence, when it comes to QC there can easily be too much of the wrong kind of standards.

"It’s an evolving field," says Edgard Delvin, PhD, head of the clinical biochemistry department at St. Justine Hospital, Montreal, Quebec, and coordinator for the American Association of Clinical Chemistry listserv on POC testing. "It started out with self-administered glucose instruments and went on to more elaborate laboratory testing. The concern is that the quality control and quality assurance programs are not there to really comply with CLIA [the Clinical Laboratory Improvement Amendments of 1988]."

In fact, in the regulatory framework now governing point-of-care testing, a fine line may separate theoretical quality from absurdity.

That, at least, reflects the experience of many members of the AACC’s POC testing listserv. In recent months they have posted a barrage of complaints and alarms about QC requirements for waived tests, charging that the Food and Drug Administration is leaning on manufacturers of waived tests to beef up their written QC guidelines, which users must follow if they are to use the waived tests at the point of care.

Steven Gutman, MD, MBA, director of the FDA’s Division of Clinical Laboratory Devices, says his agency has not sent any correspondence to manufacturers about QC for waived tests. But the Centers for Disease Control and Prevention, which was responsible for test categorization including waiver approval until Jan. 31, 2000, did send such letters to some manufacturers in September 1999.

Sharon Granade, MT(ASCP), a health scientist with the CDC’s Division of Laboratory Systems, explained the background of the letters. In 1994 guidelines, CDC clarified the waiver criteria in the CLIA law and created a process for waiving tests, which specified that the tests must have a fail-safe mechanism to ensure quality testing and reduce the risk of an erroneous result. "In order to satisfy the fail-safe requirement, some manufacturers specified testing positive and negative controls when opening a new test kit, and with each additional operator who performs testing using that kit," Granade says.

Since the kits for some qualitative tests like group A streptococcus and Helicobacter pylori result in a color change or line showing up, as opposed to a readout on a device, "There’s no way to build a fail-safe mechanism into these qualitative kits, so you’d get no answer if the test is performed incorrectly," Granade says. "That’s why we required that external QC be performed when it is used as the fail-safe mechanism."

But the requirement is supposed to be specified on the package insert. "Some of the product inserts stated that performance of positive and negative external controls was ’recommended’ and that each operator ’should’ test a positive and negative external control once with each 25-test kit," Granade told CAP TODAY. "In reviewing waiver files, CDC determined that the wording was unclear and could be misinterpreted by laboratories."

"In September 1999, CDC sent letters to applicable manufacturers, notifying them that, to maintain the waived status of the test systems referenced in the letters, the wording in the product inserts needed to be changed to reflect the intention of the mandatory fail-safe requirement. We emphasize—this letter only applied to those test systems of those manufacturers notified, and not to waived tests in general."

The FDA is reviewing QC requirements for products that are under review, Dr. Gutman says, but is reluctant to go back and re-review approved products’ package inserts. "There has been some dissatisfaction with the fact that the FDA has not weighed in on the issue," he adds.

Cheryl Cook, POCT coordinator for Grant/Riverside Methodist Hospital, Columbus, Ohio, said that new package inserts for strep and influenza kits mean that each operator must do external controls in each kit of reagents, not just each lot number or shipment. "This is a significant change in frequency of controls, especially when you have many different operators doing testing," she contends.

Practically speaking, to perform group A strep as a POC test, says Cathy Bartholomew, POC testing coordinator for Baystate Medical Center, Springfield, Mass., "Our options are to limit the number of people performing the testing to one or two, which is my recommendation, or have each person have their own kit, which is expensive."

Some manufacturers’ package inserts also recommend that positive and negative controls on the strep test be run every 25 tests, twice per kit, as well as when changing operators. Ironically, says Cook, "In practice, by the time all the nurses would run their initial QC, we would be doing well to have 25 patient tests left in a kit, so I guess worrying about running QC again after 25 patients would be a moot issue."

Roseanne Dolega, MT(ASCP), POC testing coordinator at Henry Ford Hospital, Detroit, points to another quirk of the regulations affecting QC.

"The strep kits we use are moderately complex, and at our institution, there’s not an issue with strep. It almost seems like the standards for QC are less stringent or simpler on moderately complex tests than on waived tests." It’s already difficult enough trying to explain to nonlaboratory people why they have to do so much QC, she adds. "We have a mix of CAP and Joint Commission accreditation right now, and to try to get on the same page of the songbook, we’ve tried to have everyone do the same level of QC. But sometimes a nurse will go, ’I think we’d be more compliant if we didn’t have to do QC that often.’"

"People seem really upset that, with neither advance notice nor solicitation of comments, the CDC appears to have pressured manufacturers to put in their procedures recommendations that separate controls be run by each operator who uses a device from a given kit," says Petrie Rainey, MD, PhD, professor of laboratory medicine at the University of Washington School of Medicine, Seattle. In requesting that manufacturers change their recommendations on running controls, the CDC was "effectively changing the rules without having the commentary period that would be required by the standard regulatory process," he says. "A commentary period is important because it gives end-users an opportunity to suggest changes in requirements that may be unnecessarily burdensome or unlikely to produce significant improvements in quality. It also allows one time to prepare for potential changes."

Another major issue, he says, "is that it’s extraordinarily difficult to comply with these requirements in a multi-user environment. You can simplify things by designating only a few people to do the testing, but in many instances, many nurses have already undergone training and competency testing to perform these tests."

Some institutions, he adds, are assigning each nurse his or her own box of devices, "which may increase the likelihood of getting a bad result, because the box may be sitting around for a long time rather than being used up in one or two days."

"The CDC hasn’t been realistic at all," says Dr. Delvin. "Once an instrument is let out on the market, if it’s a self-administered test, it’s very difficult to have people comply with QC and QA unless you have a very strict policy. And in the context of hospitals and HMOs, I think this is not feasible."

Dr. Cox, now director of medical affairs at BioStar Inc., a Boulder, Colo., medical device firm, says the new QC -re-quire-ments for waived tests may make it economically unfeasible to provide rapid POC testing in some settings. "To meet the CDC mandatory fail-safe requirement, operators would need to increase the frequency of QC testing," she says. "One way to keep the QC/patient result ratio low and remain cost-effective would be for every operator to have their own box, but in some settings there are 10 or 15 or more nurses. This is a lot of test kits to have in use for a potentially low-volume test." With test cost averaging $15 to $25 each and reimbursement similar to the cost, the additional QC requirement could put the cost of testing out of reach for most point-of-care sites. "It’s hard to justify performing the tests for better patient care if they can’t be done economically," Dr. Cox says.

Not all POC testing programs say they are experiencing problems. When Teresa McDonough, MT(ASCP), CLS (NCA), took over a little over a year ago as the first full-time POC testing coordinator for Mercy Medical Center, Des Moines, Iowa, the laboratory had only recently taken charge of POC glucose testing and was grappling with several deficiencies found during an inspection. The most recent inspection turned up no deficiencies in POC testing.

"QC is not an issue with our glucose tests," she says. The glucose meters have a QC lockout function, which requires two levels of QC to be performed every 24 hours with results in range, or the meter won’t function. Mercy has about 1,200 operators for glucose and does not require QC with each new operator. "For some of the group A strep tests, I have heard about that, but I’m not really affected because we don’t offer that as a POC test," she says.

In general, training is key to instilling a strong attitude in support of QC and QA, Dr. Delvin notes. "We must realize that once these instruments go out of the lab, a number of people on the wards will be handling the instruments and doing tests, and for those people the dimension of quality assurance and QC is not really the one we have in our labs," he says. "We have to insist on having a good QA and QC attitude, and we do it by having dedicated people in the laboratory to look after it. But it’s rather expensive, and this type of activity is not accounted for in any of the productivity programs."

Quality control is better used for checking up on the POC testing products, not on the people using them, Dr. Cox believes. "QC to me is a complex issue. In the traditional old-time world, QC was done to check the kit or the instrument. In a big laboratory you ran QC in the morning because you were putting in bottles of reagent and checking to make sure everything was going right. You -didn’t redo it every time a different operator came up and pushed the button.

"But then QC took on another connotation: You were supposed to be ’QC-ing’ the operator. Can the operator say this is pink and this is red, this is positive and this is negative? CLIA addresses this issue in its training and competency requirements. With the new waived requirements, competency and QC are now blurred into a single action. Performing QC is establishing the competency of the operator for each kit or day of use. This is not the same thing as performing QC to confirm the integrity of the reagents or instrument."

Indeed, several reports confirm that operator error is a frequent source of problems. At Johns Hopkins Medical Institutions, Baltimore, complaints constitute one of four areas that are monitored routinely to catch problems, reported James H. Nichols, PhD, a faculty member of the Department of Pathology, at an AACC roundtable this year on practical management of POC testing. A glucose incident resolution report over an eight-month period at Hopkins identified 19 percent of the complaints related to the device and 58.6 percent to operator technique, with 22.4 percent found to have no actual problem. Only 8.3 percent of complaints in a hemoglobin incident resolution report for the same period were blamed on the device, while 78.8 percent were owed to operator technique and 12.9 percent to no actual problem.

Hopkins found that financial incentives can help stem these kinds of human error. Because purchasing is the responsibility of the department that performs the test, the nursing/clinical units own the POC testing devices and kits, provide QC and maintain the devices, conduct training and re-educating, hold QC competency records, and maintain backup devices. The pathology department, responsible for ensuring compliance, discovered an effective way to remind operators to date urine dipstick vials upon opening: During systematic inspection of all units, it removed and disposed of all undated vials. Those vials came out of the nursing budget, so nursing management had a financial incentive to ensure proper dating procedures were used, and the problem did not recur, Dr. Nichols reported.

Quality relates to the quantity of POC testing performed, many POC coordinators indicate, but there may be an inverse correlation. After Baystate brought i-Stat testing to nine sites about five or six years ago, the usage was high, says POC coordinator Bartholomew. "But over the past year or so, the number of tests has actually dropped 30 percent, even though the same sites are doing testing and the proficiency has improved. So we’re not wasting cartridges, and people are more confident in the results. When they know they have an instrument available any time of day, they’re less anxious about patient status and don’t feel they have to test as much."

At Little Company of Mary Hospital in Torrance, Calif., the amount of POC testing is not changing significantly, says laboratory supervisor Patt Hess, but in some cases nursing has asked to stop certain POC testing. "They may find out it’s more work than they want.

"It’s mostly the competency testing they don’t like, and also the daily checking and sending down the paperwork. They’d prefer not to do it if they had the choice; it’s the clinicians who want results as fast as possible."

"We did stop doing occult blood on the units," she adds. "The reason was it was really hard to keep up with all the paperwork and competency requirements when the units hardly ever had the test. It’s an enormous number of people to keep trained, and they preferred not to go through all that."

Efforts to contain costs have kept use of the i-Stat in her emergency department at less than once a day, even though the ER sees 120 to 150 patients a day, says Frances Montoya, POC coordinator at Presbyterian Hospital, Albuquerque, NM. "The criteria state that the i-Stat should only be used if only the i-Stat analytes, or the i-Stat analytes and a CBC, are ordered. The rationale here is that if any other lab chemistries are ordered, the ER docs would have to wait for those results, so it would be more cost--effective to send everything to the lab." But, she points out, the low usage level "raises a separate issue for me, in ensuring continued competency, time involved in training new staff, and so on."

Partly in response to the Institute of Medicine error study issued a year ago, the FDA’s device section is under pressure to generically reduce errors across the board in medicine, says Gerald J. Kost, MD, PhD, director of the Point-of-Care Testing Center for Teaching Research at the University of California at Davis. In a recent study at an academic health center, accuracy was not a leading source of higher satisfaction observed for POC testing versus central laboratory testing. Timeliness, convenience, labor conservation, and improved patient care score the strong-est relationship with satisfaction. But this is no reason to become complacent about inaccurate POC testing, Dr. Kost argues.

After surveying some three dozen experts last spring, Dr. Kost designed criteria for prevention of errors in POC testing to include standards in four areas: security, validation, performance safeguards, and connectivity. In written critiques he received from experts at several conferences related to POC testing, "every single one insisted that all POC operators should be validated and certified before using a device."

They also stressed the need to set performance safeguards according to the way local institutions are designed, tailoring QC to specific clinical areas, he adds. "You don’t want retinal identification for something used for home-testing glucose. On the other hand, you don’t want the test kit floating around the hospital being used by anybody without QC."

Not surprisingly, connectivity is a top priority for many. Dr. Kost and others pin their hope on the POC Connectivity Industry Consortium’s effort to create a connectivity standard and users insisting that vendors’ POC devices comply. "The endpoint pressure on industry will get the problem fixed," he says. Data management in POC testing falls far short of what it should be. "Lack of connectivity is one of the biggest problems POC coordinators face," Dr. Kost says.

Among the technological shortcomings of POC testing, one of the chief concerns Dr. Delvin has is the reliability of instruments in critical zones of results. "Most of the validation of instruments has been done with samples which are almost normal in terms of the analytes measured," he says. But POC testing in the context of the hospital is really done to look at either very low or very high values at the extremes.

Unfortunately, "What we are facing is that the instruments are probably reliable within concentrations of analytes for which we don’t really need POC testing."

When operators get aberrant results, they will often discard them, deciding that they don’t really fit and that another test is in order, he says. "But the usual cause is either the user improperly putting the sample on the instrument, or the fact that we are dealing with samples which are not in the middle portion of normality. They’re outside normal ranges, and the discrepancy or variation of results is higher at both ends of the spectrum."

Similarly, the instruments may not work well with certain kinds of patients. St. Justine’s, which is a pediatric hospital, has a large pediatric ICU and neonatal ICU, but "we have a hard time adding reliable data through POC testing because the high viscosity of blood and the high level of hematocrits all render the instruments less reliable," Dr. Delvin says, noting that the device industry has not really studied this problem because there is not much money in pediatrics compared with other clinical areas that use POC testing.

McDonough says her main challenge at Mercy Medical Center has been developing a good relationship with nursing, which she started on by doing face-to-face interviews with nurses in every unit. "Nurses think differently from lab people. You can’t just dictate to them; you have to make it a team effort," she says. "I think POC testing has helped nurses see a little more of why things happen in the lab and why tests are rejected, but I wouldn’t go so far as to say they really understand. Their main focus is their personal interaction with patients. They view that as their No. 1 mission, so they still have a lot of the mentality of ’I’ll put the sample in the box and it will give a result, regardless of QC or whether the box was dropped or the room was 85? with 100 percent humidity.’"

With outcomes like these, POC testing can be a means of getting at larger quality issues, Dr. Cox believes. At Methodist Hospital of Indiana, the main benefit of the POC testing program was that "nurses had a greater understanding of laboratory issues and the laboratory had a better understanding of nursing issues. That wasn’t there before.

"Before, the nurse blindly sent a sample to the lab. Particularly with neonates, we’d often reject the sample because it had clots, and they’d say, ’It wasn’t clotted when I sent it.’ When the nurses started using the samples they collected for POC testing, they’d see the little clots come out of the tube." In that way, the nursing department acquired an immediate appreciation of the importance of specimen collection and handling, she says.

"They never understood why we were rejecting their samples. Now they see the lab was not being rigid and insensitive to neonates, and they are much more careful in collection and in QC."

Anne Paxton is a freelance writer in Seattle.