Accuracy-Based Programs for ‘bigger dividends,’ better care

Valerie Neff Newitt

July 2021—Proficiency testing is the bedrock of good laboratory performance, but Accuracy-Based Programs are equally important, which is why members of a CAP committee are hoping for 10 labs from each peer group to participate.

“If we can get enough people to step up to the plate, we can make sure that the methods that are FDA cleared in the United States are accurate. And then traditional proficiency testing will pay even bigger dividends,” says Gary L. Horowitz, MD, chair of the CAP Accuracy-Based Programs Committee and professor of pathology, Tufts University School of Medicine, and chief of clinical pathology, Tufts Medical Center.

The CAP’s Accuracy-Based Programs do what proficiency tests can’t: verify the accuracy of test results against a gold standard. Proficiency testing provides a check on procedural methodology and results reliability as compared with peer laboratories. “While it is reassuring to know you match your peers, sometimes an entire peer group could be getting inaccurate results,” Dr. Horowitz says.

“That’s the whole point,” says committee vice chair Andrew N. Hoofnagle, MD, PhD, professor of laboratory medicine and head of the Division of Clinical Chemistry, University of Washington. “The whole peer group can be skewed, different than either the rest of the peer groups or, more importantly, the reference measurement procedure, the gold standard, if there is one.”

It is difficult to find human specimens with the range of concentrations needed to assess all relevant concentrations of all tests. Proficiency testing providers therefore use materials that simulate human specimens, adding materials to obtain the range of concentrations needed, Dr. Horowitz notes.

“When PT material is made, it begins with basic plasma or serum, but that starting material is dialyzed, preservatives are added, and then it’s spiked with calcium or cystatin C or transaminases and/or other materials to get the concentrations desired. At each of those steps the material becomes less like genuine human specimens,” he explains. “There are occasions when it is okay, but most of the time it doesn’t work. These specimens do not react exactly like genuine human specimens, due to a phenomenon referred to as matrix effect.”

Matrix effects are caused by something nonspecific in the sample that changes the value of whatever the laboratory is trying to measure, says Brian Harry, MD, PhD, a member of the Accuracy-Based Programs Committee and assistant professor of pathology and medical director of special chemistry, University of Colorado School of Medicine. “We don’t know specifically what they are, but we do know they exist and can disrupt measurements.”

Two peer groups using a given test might get exactly the same results on human specimens but different results on the proficiency material. “Vitamin D testing provided a good example of this,” Dr. Horowitz says. “Using regular PT Survey materials, there were twofold differences in responses. We’d get a value of 50 by one method in one peer group, and 120 on the same specimen using another method in another peer group. But when we sent real human serum, the numbers agreed between the two groups. So even with tests that have FDA clearance and existing documentation indicating they are good tests, labs were getting values that appeared to be wildly different on traditional proficiency testing material when in reality the tests provided comparable results on real human specimens.”

“When you agree with your peer group and the peer group is inaccurate compared to the reference method, it’s not a mark against you as a laboratory. It’s a mark against the manufacturer,” he says.

Accuracy-Based Programs run only on genuine human samples that exhibit essentially no matrix effects. Commutability and reference measurement procedure are the two terms to keep in mind, Dr. Hoofnagle says. “Commutability means samples behave like actual patient samples in each assay. Reference measurement procedure means we have some gauge on the truth—the actual value of the concentration of the analyte in the sample. That’s what sets Accuracy-Based Programs apart from traditional PT, which uses samples that I call concocted, pretend. They do not represent actual human biology.”

The CAP obtains specimens for the Accuracy-Based Programs in several ways. For some tests, such as testosterone, cortisol, and A1C, Dr. Horowitz says, “it’s relatively easy to get specimens from individuals with a range of concentrations—men versus women, morning versus evening, individuals whose diabetes is controlled to varying degrees. In other cases, like creatinine, we have proved we can add creatinine to normal serum to get high concentrations without introducing matrix effects.” These matrix-effect–free specimens are commutable and can be used to compare different methods to one another and to reference methods, he says. “Then we can assess whether the method used in a lab is generating truly accurate results.”

These assessments drive industrywide improvement. In general, he says, “things work pretty well.” But when a problem is detected, the CAP shares the data with manufacturers, and they can then improve the affected assays accordingly.

“For the CAP, the great-grandparent of Accuracy-Based Programs is hemoglobin A1C,” Dr. Horowitz says. “For many years it has used commutable material and results have been compared to a reference-based measurement procedure. A1C is a huge success story because when data became available from hundreds, if not thousands, of labs, manufacturers with problems fixed their methods.” The methods used today are far more accurate and precise, he says. “We wouldn’t be where we are today without that Accuracy-Based Survey. Now everyone can reliably use the same cut-points for making a diagnosis of diabetes and for determining what is considered good or bad control. It made such a dent in the field when people actually saw performance using commutable material and the reference values for it.”

A1C was unique, he adds, in that it was easy to find patients with normal values and various degrees of elevation because diabetes is prevalent. “It wasn’t hard to find materials to cover the range of values we needed. So the CAP was able to send out real commutable materials for that Survey. Newer Surveys are the grandchildren of A1C, but clearly it has been a lot more difficult to get commutable materials and the ranges we want.”

The Accuracy-Based Programs available now are as follows: Accuracy-Based Glucose, Insulin, and C-Peptide; Accuracy-Based Testosterone, Estradiol; Accuracy-Based Lipids; Accuracy-Based Vitamin D; Accuracy-Based Urine; Harmonized Thyroid; Hemoglobin A1c GH5 (five challenge), GH2 (three challenge); Hemoglobin A1c Accuracy Calibration Verification/Linearity; and Creatinine Accuracy Calibration Verification/Linearity.

At a meeting in March, members of the Accuracy-Based Programs Committee discussed what other analytes should be measured in the Accuracy-Based Programs. “ABP doesn’t develop material for everything that’s measured in the lab,” Dr. Harry says, “but it does focus on things that are critical, whether it be because technologies produce different results, because an analyte is particularly hard to measure because of its chemistry, or because it is something that affects many patients.”

Dr. Horowitz makes a strong case for the use of Accuracy-Based Programs for labs using laboratory-developed tests rather than FDA-cleared assays.