Accuracy-Based Programs for ‘bigger dividends,’ better care

“If you’re using a manufacturer’s FDA-cleared assay and run it the way the manufacturer has directed, the burden of making sure that assay is good is on the manufacturer,” he notes. “You have fewer validation studies to do than when you develop your own test. In the case of LDTs—and almost all the LC-MS assays are LDTs—each one is a little different. They use different columns, different transitions, different reagents. You don’t have a peer group. So the only way you can know your test is accurate is to use commutable samples and compare results to a reference method. Those using LDTs, in particular, should be using Accuracy-Based Surveys.”

Harmonization is a related issue of concern to the committee. In the absence of a reference method, “it’s impossible to know what the true value is,” which is needed to establish accuracy, Dr. Horowitz said in a CAP podcast on the Accuracy-Based Programs. But with commutable matrix-matched specimens, whether different methods get the same results can be determined, “and if all the results are the same, or harmonized, that’s a good thing. And once a reference method and reference materials are developed, we can then assess accuracy. If different methods are not harmonized using commutable specimens, we would hope that the reference intervals are accordingly different,” he told listeners of the podcast.

“Accuracy-Based Programs can, at a minimum, provide insight into whether various test methods are harmonized and get comparable results on genuine human specimens,” he said in an interview.

TSH is one of many analytes for which no reference method is available. “We know there are differences in TSH measurements between methods,” Dr. Horowitz says. “They’re not harmonized; we do not know what the true value is. We’re hoping the reference intervals are different to reflect those differences.” Cystatin C, too, was a problem. “But based on a commutable specimen, one of the peer groups that was different has recalibrated, and now cystatin C is harmonized. We don’t know what the true value is, but at least all the major methods are getting the same value.”

D-dimer is under discussion between the Accuracy-Based Programs Committee and the CAP’s Hemostasis and Thrombosis Committee. “We don’t know if everybody’s getting the same results,” Dr. Horowitz says. “What we do know is that for D-dimer in particular, two different sets of units are being used across the country,” with some using a cutoff of 500 and others using 250 because of the differing units. “At the very least we want to get everybody on the same units and to see whether these assays are harmonized by using commutable materials.” The goal is to test the tests to make sure they’re accurate. “Let’s make sure doctors can use them interchangeably,” he says, “because people are much more mobile than they used to be.”

Dr. Hoofnagle agrees there is a need for greater industry standardization that might be achieved through efforts such as Accuracy-Based Programs and harmonization. He illustrates the point by offering examples of problems discovered in his own laboratory.

One assay he was running for LDL and HDL cholesterol on a specific platform was greatly skewed. “Even though there’s a standardization program at the CDC, LDL and HDL cholesterol numbers remain a disaster. It’s really bad,” Dr. Hoofnagle says. “One instrument that I have gives patients better numbers in terms of cardiac risk because the HDL results we get are higher than every other peer group. It’s possible that a patient would decide against taking a statin if their specimens were measured in our laboratory on that platform. If they drive half a mile away and go to a different laboratory, a different platform will give them a different number and their HDL will be lower and their LDL will be higher.”

Another disturbing situation he encountered involved testosterone testing. “I used an immunoassay by a manufacturer for testosterone for many years,” Dr. Hoofnagle says. “Our providers kept saying, ‘Andy, you are not getting the right result; your answers are wrong.’ I said, ‘That’s okay because I’ve changed my reference range. My reference range is shifted lower because the results are lower than what we see on average across the industry.’ And they said, ‘That doesn’t matter. There’s a cutoff now in the literature. All of the patients are looking at that cutoff, so when an older man walks into my office and they get a test on your platform, they may fall below that number.’ And I say to them, ‘It’s okay, you’re in the reference range,’ but they point to that number on the paper and say, ‘No, I’m below that number. You have to give me testosterone.’ It makes the patient-provider conversation more complicated than it needs to be.”

Dr. Hoofnagle says he heard this from his providers repeatedly over the years. Then the Accuracy-Based Program for testosterone became available. “Now, looking over seven years, we see we have had this bias of about 25 to 30 percent every year compared with the reference measurement procedure. That’s where this accuracy-based effort is so amazing. This wasn’t concocted material. These were actual human specimens and every single one of them was wrong. I brought this to the manufacturer’s attention on three different phone calls. And I said, ‘This is bad. You need to fix this.’”

If laboratories don’t use the Accuracy-Based Programs it could be because they know they meet the requirements of accreditation with traditional proficiency testing Surveys or because fewer analytes are in the accuracy-based specimens, Dr. Horowitz says. “For the regular proficiency test, you could have 40 or 50 different analytes in it, but it is not commutable. So it’s a very small number of tests that are in the Accuracy-Based Surveys. Ideally, we would do this for all proficiency testing, but it’s prohibitively expensive and very hard to get commutable samples that would achieve everything.”

Dr. Horowitz is hopeful that the CAP will get a representative sample of laboratories from each peer group to use Accuracy-Based Programs. “All we need is 10 labs from each peer group to participate. Then we will be able to say, ‘Peer group one uses XYZ method for testosterone, and when they participated in the testosterone Accuracy-Based Survey their values were right on target with the reference method.’ Then anyone else that uses that specific FDA-cleared method, running it the way the manufacturer directs, can say, ‘This is great. I’m on the mean for my peer group and I know my peer group is accurate based on the Accuracy-Based Survey.’ So if you’re close to the mean value for your peer group and your peer group shows good performance in the Accuracy-Based Programs, you know you’re good.”

Participation in Accuracy-Based Programs helps the entire field take care of patients, Dr. Hoofnagle says. “It’s saying, ‘I want the entire field to get the right answer; therefore I am going to be in ABP.’ It is our job as the CAP to help labs give patients the right answers every time. I would plead, if I could, with every lab director to consider ABP as a way to improve health care.”

Identifying manufacturer methods that behave differently from other methods in the field is itself a compelling reason to participate, Dr. Harry says. The most important reason, he says, is to drive better patient care.

“Most physicians look at a number on a paper or in a computer and never question that number or what it means or how it came about. They take it at face value,” Dr. Harry says. “So it’s the clinical pathologist’s job to make sure that number is the most accurate number we can provide.” Doing so is how physicians and others practicing laboratory medicine treat their patients. “We don’t see our patients in the clinic,” he says. “We have to make sure they get good care because we provide their doctors with good data.”

Valerie Neff Newitt is a writer in Audubon, Pa.