ASR issues up in the air—
  but for how long?

March 2004
Cover Story

William Check, PhD

Like winter weather, the Food and Drug Administrations intention to revisit its 1997 rule defining and regulating analyte-specific reagents just seems to hang around and hang around. It’s March now, and people living in the North are tired of the cold, windy days and dreary gray skies. Dan Farkas, PhD, HCLD, feels the same way about perpetually pending proposals to alter the ASR rule.

"I can’t believe another article is going to be written about ASRs," exclaims Dr. Farkas, associate professor of pathology at Baylor College of Medicine and director of molecular pathology at The Methodist Hospital, Houston. "This issue just won’t go away. It seems like nothing changes. A few in vitro diagnostics manufacturers abused ASRs, FDA took notice, and now FDA suggests that ASRs need to be revisited. The system isn’t broken; it just needs to be tweaked."

Like a winter-weary person longing for sunshine, Dr. Farkas wants some light thrown on the much-discussed but still unsettled changes.

Despite all the buzz, says Debra Leonard, MD, PhD, director of molecular pathology at the University of Pennsylvania, Philadelphia, "FDA hasn’t done anything to re-look at ASR regulations that any one of us can determine. They keep saying that they are going to be re-looking at ASR regulations, so there is uncertainty about what might happen. At this point, we just have to operate under current regulations until FDA makes some definitive statement."

Paradoxically, says Dr. Farkas, who was president of the Association for Molecular Pathology in 2003, the reagents themselves are working perfectly well. "What frustrates me is that the suggestion for change flies in the face of how well ASRs work in the clinical laboratory," he says. Probably every IVD company is selling ASRs, Dr. Farkas says, and working within the rule. Speaking of the Roche/ Affymetrix AmpliChip CYP450 that caused the furor last year, Dr. Farkas says: "That was ludicrous. A 40,000-site chip is not an ASR by definition. But that was a rare exception."

Steve Gutman, MD, director of the FDA’s Office of In Vitro Diagnostic Device Evaluation and Safety, confirms that one motivation to revisit the ASR rule was that "some companies appeared to be interpreting the ASR rule a bit broadly, and there were some instances where at least it appeared to us that products that look like kits were being marketed as though they were ASRs." Dr. Gutman declines to confirm examples other than the Roche/ Affymetrix incident. "I’m reluctant to discuss nonpublic or pending actions," he says. About that incident, he says only this: "It was certainly interesting enough that FDA used the opportunity to make public its thoughts about this kind of marketing practice."

However, Dr. Gutman says, there is a second major impetus to revisit the ASR rule: to determine whether the rule and the regulations provided by the FDA and CLIA adequately handle all types of in-house-developed (homebrew) tests.

"The idea of genetic exceptionalism may not be wide enough," Dr. Gutman says. "Genetic tests may not be the only tests that could pose high risk to patients."

Two infectious disease diagnostics that he believes are not good fits with the ASR rule are tests for the SARS coronavirus and for smallpox. He says SARS and smallpox are like HIV and Mycobacterium tuberculosis, which were classified as high-risk tests in the original rule: All are contagious, with powerful public health impact, and have interventions that may change the outcome. Risk-benefit balance is most important in these cases.

"A SARS test that isn’t exceptionally accurate but is fast might fill the bill," he says. "What makes this both art and science is that it is not accuracy per se; it is accuracy and access that makes it of interest."

Dr. Gutman calls the high-risk categorization of HIV and TB tests "an existing pillar for the risk-based approach." The FDA’s proposal for revising the ASR rule envisioned expanding this classification scheme, but, Dr. Gutman acknowledges, "crafting a well-grounded risk-based approach has proved challenging to SACGT [the Secretary’s Advisory Committee on Genetic Testing] and to FDA, and to the professional groups that with good intent have tried to provide FDA with solicited advice." It has been harder than he would have predicted. "The agency is still interested in moving forward in this area," he says, "but I would be reluctant to predict how it would move forward." Or in what time frame.

At the heart of this discussion are questions about what constitutes adequate analytical and clinical validation and how they can best be ensured. Molecular laboratory directors believe that these needs are already sufficiently addressed. Dr. Leonard was the chair of the College’s Molecular Test Validation Project Work Group, which revised the molecular pathology test validation checklist. "The previous Secretary’s Advisory Committee was concerned that laboratories were developing tests that had no premarket review," Dr. Leonard says. SACGT recommended that the FDA develop a program to review these tests. "The College felt strongly that premarket review could be done within the existing laboratory inspection process," Dr. Leonard says. The work group expanded some questions and wrote new ones. The revised checklist was implemented in December 2003.

"CAP’s checklist requires both analytical and clinical validation," says Wayne Grody, MD, PhD, professor of pathology and laboratory medicine and director of the molecular pathology laboratory at the University of California, Los Angeles, who was a member of the work group. "That is enough of a safeguard as long as you have honest laboratory directors who will not introduce a test that is completely bogus. FDA has less knowledge and experience in this field than we do. Why do they need to do secondary review of something that we have already reviewed?"

Ultimately, reliable test results depend on responsible laboratory directors, agrees David Hillyard, MD, associate professor of pathology at the University of Utah School of Medicine and director of molecular infectious disease testing at ARUP Laboratories, Salt Lake City. "For me an assay must not only contain GMP-manufactured reagents, but also be put together well and clinically validated. Just because a manufacturer has a GMP certificate on the wall doesn’t ensure that a reagent or kit performs reliably in clinical testing. For molecular ASRs that I deal with," he says, "my assumption is that they require the same level of careful scrutiny and extensive validation that an in-house-developed test would receive. We have examined molecular ASR kits that we would not consider to be adequate."

Validation may become less burdensome as more laboratorians describe their experience with ASRs and clinical validation studies in the literature, Dr. Hillyard suggests. "But for molecular ASRs, which are just coming out, we treat them as if we had developed that test ourselves."

Everyone agrees that analyte-specific reagents provide major benefits for molecular laboratories. "ASRs support and enable homebrew testing," Dr. Farkas says. "They add a level of confidence to homebrew testing that I think has moved the community forward." ASRs also promote standardization, says Karen Kaul, MD, PhD, director of the molecular diagnostics laboratory at Evanston (Ill.) Northwestern Healthcare. "For some assay types, virtually every laboratory has a unique assay," Dr. Kaul says. "Moving toward ASRs is a step toward standardization." Dr. Kaul explains that ASRs can shorten the analytical development and validation phase of new test development. "With these reagents we can assume some of the homework is done in terms of target choice," she says.

ASR-based assays also act as bridges to FDA-approved tests, Dr. Hillyard says. He cites molecular hepatitis assays, which came online as non-FDA-cleared tests and then migrated to FDA-cleared status. "These tests filled an essential role in their pre-FDA-approved stage," he says. More recent examples are tests for CNS and respiratory infections. "There is no FDA-approved molecular test for HSV encephalitis," Dr. Hillyard says, "but the consensus is overwhelming that such testing contributes enormously to patient care." Because molecular testing moves so quickly, there will always be such cases.

Though beneficial, ASRs are not cure-alls. They can be more expensive than in-house assays, and they are not superior for all applications. Dr. Kaul has used some in-house assays for years and feels no compulsion to switch to an ASR or FDA-cleared assay. Recently, she put her veteran in-house assay for TB directly on a real-time PCR instrument. "We know how it performs and what it measures, so we feel no need to switch," she says. Similarly, Dr. Hillyard found some ASR assays for encephalitis not sensitive enough compared with his extensively validated in-house-developed assay. Dr. Leonard has developed certain tests in-house because there are no ASRs for them. One example is a test for spinal muscular atrophy, which is the second most common lethal recessive disease after cystic fibrosis.

"Industry focuses on infectious disease tests because they are higher volume," Dr. Leonard says. "They are done repeatedly for the same patient, while a genetic test is done only once. Because of these marketing issues, there will always be a need for laboratory-developed genetic tests."

Still, in many situations ASRs offer major benefits. Without ASRs, "much molecular laboratory test standardization and quality improvement would be held hostage to the exceedingly slow FDA approval process," Dr. Hillyard says. So it is not surprising that laboratory directors are concerned about revisions that might make the validation process more burdensome. "We have to validate any test we bring in, whether it is an FDA-approved kit, an ASR, or homebrew," Dr. Kaul says. "What is different is the magnitude of evaluation and time involved among the three. When starting with ASRs, we can assume we are starting with primer and probe sequences that identify what we are looking for. We still look at reproducibility, sensitivity, and specificity. But I think in general our studies to validate ASRs are a bit shorter than for completely homebrew assays."

However, Dr. Hillyard notes, some ASR-based viral load assays that perform extremely well pose difficulties for validation. "Doing a viral load assay entails generating a standard curve," he explains, "and the manufacturer can’t even tell you what material to use to make that curve or how to interpret its shape with respect to the assay’s software. Calibration also requires fitting to an absolute standard. "For hepatitis C and B, there is WHO standard material," Dr. Hillyard says, "but the process, though it sounds straightforward, can be challenging to perform accurately." Hindering the process is that only small amounts of WHO material are sent to a given laboratory.

In addition, Dr. Kaul says, "we do a lot of comparison with other laboratories since ASRs do not have an assay protocol appended to them. It is up to us to see how they perform in our various laboratories." Such comparisons provide information over and above CAP Surveys. Dr. Kaul is involved now in an exchange of samples initiated by Dr. Hillyard, exploring a question about Roche’s herpes simplex virus assay. "The runs always look good," Dr. Kaul says. "But we are looking more closely at low-end copy number." She notes that molecular laboratory directors often go beyond the minimum standard for QC.

"This may be rooted in our history of designing and developing in-house assays," Dr. Kaul says, "and the fact that we feel compelled to have a deeper understanding of the tests we perform and the results we produce."

Dr. Farkas agrees: "We are continually looking to make sure that reagents are behaving properly. We don’t stop looking at performance of ASRs once they go live."

Andrea Ferreira-Gonzalez, PhD, associate professor of pathology and director of the molecular diagnostics laboratory at Virginia Commonwealth University, related examples of how labs validate ASR-based assays-analytically and clinically. In the oncology area, she recently switched from a single-tube nested RT-PCR procedure to a single-tube real-time RT-PCR for quantitation of BCR/ ABL chimeric mRNA (t[9;22], Philadelphia chromosome) for diagnosing and monitoring of minimal residual disease in patients with chronic myelogenous leukemia. She used a cell line that contains a (9;22) translocation and cell lines without the translocation in addition to residual patient samples for the analytical validation steps. For clinical validation, she looked at samples already characterized for the presence of BCR/ABL chimeric mRNA from patients who had already been diagnosed with CML and/or who had received bone marrow transplants and been monitored for minimal residual disease, looking for clinical diagnosis of relapse or failure of the transplant.

Validating a genetic test for screening for 25 mutations in the CFTR gene was "not as involved because we were able to use commercially available ASRs that we validated in-house," she says. For analytical validation, she used commercially available cell lines characterized for mutations in the CFTR gene and patient specimens. For clinical validation, in addition to consulting the published literature, she ran specimens that were "kindly provided by laboratories" already performing this clinical test. "The major challenge," she says, "was to find enough specimens to test all 25 mutations. Running samples that have been already clinically validated is part of clinical validation, but it relies on another laboratory’s initial work."

Dr. Ferreira-Gonzalez has been developing a quantitative real-time PCR assay for BK virus in urine and plasma of kidney transplant recipients to help in the diagnosis and monitoring of patients at risk for BK-associated nephropathy (BKAN). One of the first steps was to identify the target to amplify: selecting a sequence that doesn’t cross-react with any sequence of commonly associated viruses (in this case, Papova viruses such as JC) or other pathogens that might be present. A next step was to devise an efficient nucleic acid extraction method that eliminates inhibitors as much as possible, followed by establishing amplification conditions that amplify the target of interest without amplifying other viruses or human genetic material. These fundamental early steps critically affect the assay’s performance.

"It takes a lot of trial and error to find optimal conditions for achieving the highest specificity and sensitivity," Dr. Ferreira-Gonzalez says.

Analytical sensitivity and specificity and the linear dynamic range of the assay were tested with purified virus spiked into negative patient specimens.

Dr. Ferreira-Gonzalez went one step further for clinical validation by setting up a clinical study collecting samples from all kidney-transplant patients during routine clinic visits or hospital admission for a period of six months. "Six months later, we performed testing and then went back and performed chart reviews looking for documented diagnosis of BKAN on clinical grounds and/or by clinical laboratory methods," she says. Doing this clinical validation made it possible to set quantitative thresholds to discriminate patients with BKAN disease from those with BK infection, and those who are at increased risk for the disease.

"It was a tremendous amount of work and additional expense to do that clinical validation," Dr. Ferreira-Gonzalez says. Not all laboratories can afford that expense, she points out. Clinical validation can sometimes be done by looking at published literature. "In those two cases [BCR/ ABL and BKAN by real-time PCR], there was not sufficient data published yet," Dr. Ferreira-Gonzalez says. "That is one problem with in-house-developed assays. Even though there is published peer-reviewed literature, you might have developed an assay a bit different from what is in the literature. You can’t assume you can detect the same thing."

(Dr. Ferreira-Gonzalez has coauthored a chapter, "Laboratory Developed Assays in Molecular Diagnostics," in Molecular Diagnostics for the Clinical Laboratorian, Coleman WB, Tsongalis GJ, eds. 2nd ed., in press.)

Dr. Grody recently introduced a new method for detecting Huntington disease mutations. For analytical validation, he says, "we ran our old samples and CAP proficiency samples with the new method in a blinded fashion to make sure we got the same results. That is similar to what we do whenever we validate any new test. Or we might exchange samples with another laboratory to see that our results correlate." In this process, they are looking to ensure that, if a mutation is present, they can find it, and if it is not present, their assay gives a negative result.

Clinical validation is a much bigger undertaking, which requires showing that any mutation detected correlates with disease and determining the degree of risk it poses for the patient. "Often that is something that one laboratory can’t do by itself," Dr. Grody says. "Although some people think we should be gatekeepers for clinical validation, various guidelines, including the CAP checklist, allow for the use of peer-reviewed literature. In genetic diseases especially, clinical validation can require lots of large studies and many years of followup."

Using Huntington disease as an example, Dr. Grody asks, "How could I do clinical validation for something like that? It is usually done as a predictive test on young adults. Even if it is positive, the first symptoms would not appear for up to 30 years. That is how long it would take me to do clinical validation." In such cases, it is essential to be able to rely on existing knowledge.

In some cases even large studies don’t define the exact clinical risk of a mutation. Dr. Grody cites BRCA1 testing as an example where there is still controversy about what a positive result means, because the degree of penetrance is less than unity but uncertain. "People cite this as one of the more difficult tests for clinical validation," he says. Many people who have a BRCA1 mutation will never get breast or ovarian cancer, so test interpretation is more difficult and subtler. "The analytical test can be 100 percent accurate, but the clinical meaning of the result will never be 100 percent certain," Dr. Grody says.

Penetrance can be so low that it affects the validity of the test, he notes. For the Factor V Leiden mutation in a heterozygous state, the lifetime risk of venous thrombosis is only about 10 percent, which is why both CAP and the American College of Medical Genetics discourage predictive FVL testing in asymptomatic individuals. "Some people claim that the hereditary hemochromatosis gene might have penetrance as low as that or even lower," Dr. Grody says. With low-penetrance genetic diseases, uncertainty arises because it takes years to establish clinical correlation, even in multicenter studies.

Dr. Kaul contrasts clinical validation for genetic diseases with infectious diseases. For most quantitative microbial assays, she says, "it is well established that a specific organism causes a disease. We do analytical validation, then show that PCR, for instance, compares well to culture or whatever else we are doing. Then we are done." For genetics, it is a different story.

"That is where most headaches are today," she says. "A locus may be linked to a disease or a predisposition to disease, but it can be extremely difficult to prove that association. That is clearly beyond what molecular diagnostics laboratories can do. For us to validate a test based on an association already accepted in the literature is one thing. But if we are looking at something novel, it will be extremely difficult to do clinical validation ourselves."

To the FDA’s Dr. Gutman, a good way to approach the dilemma about clinical validation is to ask, "What do you actually need to demonstrate to prove that something like CF testing, for example, is a reasonable test to offer?" One legitimate source of proof, he says, is what is already known. When a test has reached a state where it is ready for clinical use, there will be data, either published or unpublished. "CDC, NIH, and industry are sitting on lots of data, both published and unpublished," Dr. Gutman says. In addition to clinical data, he adds, "there should be some assurance that the tool at hand has an analytical link to that data, that the signal being generated in the laboratory does in fact connect to the clinical data." Dr. Gutman cites immunoassays as a class of assay that one must be careful with. "Antibodies can measure very different epitopes," he says. "So they can measure the same analyte but give quite different results." Taking care with these reagents is necessary for manufacturers of devices submitted to the FDA and for laboratories developing in-house assays.

What if clinical evidence for a homebrew assay is lacking?

"For homebrew assays," Dr. Gutman says, "one of the differences between CLIA oversight and FDA oversight is that FDA in a more pointed way appreciates the fact that tests don’t spring to life, but go through an investigational stage. So there are probably tests that may be coming out as homebrew that if FDA would put them into our total life cycle might be considered investigational." Calling them "preliminary" is not a choice for the FDA, he says. "When a test is looking for an intended use and performance characteristics, we have a name for it. That name is investigational device."

Dr. Kaul cites a test that may have been introduced before its intended use was proven. In 1993 and 1994, there were publications saying that it was possible to stage prostate cancer molecularly by measuring circulating tumor cells. "That was quite exciting," Dr. Kaul says, "and it made sense. Such a test could help men decide whether to have radical prostatectomy or just radiation therapy." Amid great publicity, some reference laboratories licensed this assay and started offering it to patients. But after Dr. Kaul and others worked with it further, they found it did not correlate well with stage of disease. For example, there was no statistically significant difference in results in patients with truly organ-confined disease and those with positive margins or lymph node metastasis. Whether results correlate with relapse remains to be seen. "Now 10 years later we still don’t know how or whether we can use this assay," she says. Yet an ASR for this purpose might be offered.

Molecular laboratory directors are not primarily concerned, however, with new requirements for investigational tests. Their concern is more practical-that they could be required to perform more extensive validation for a test whose utility is already established. It was to forestall that possibility that the College’s Laboratory Accreditation Program checklists were expanded.

"The new questions require an inspector to review information about tests brought online since the previous laboratory inspection that were not FDA-approved or -cleared, tests developed and validated by the laboratory," Dr. Leonard says. Inspectors must see whether the new tests were properly validated before use, including whether the newly implemented assay was clinically validated using literature citations or internal study results.

Stephen Sarewitz, MD, staff pathologist at Valley Medical Center, Renton, Wash., is the CAP checklist commissioner. "The new checklist attempts to avoid a heavy-handed regulatory approach to in-house-developed tests," he says. "We were looking for a way to satisfy FDA yet leave laboratories free to develop tests at the forefront of diagnostics." Question ANP.12425, for example, has been expanded to clarify that it applies to all Class I ASRs, not just immunohistochemical studies.

Also, the statement about the use of a disclaimer is more forceful. "There was some confusion about using the disclaimer on reports for ASRs," Dr. Sarewitz says. "We thought it was worthwhile to expand the question so that if people read the note accompanying the question they could understand something about what ASRs are and why FDA requires the disclaimer. There is some question as to whether the disclaimer would be required for an active ingredient synthesized in-house, but the conservative approach would be to use the standard disclaimer." A similarly expanded version of the question is in the molecular pathology checklist.

In both versions, a note says that the laboratory "must establish or verify" the performance characteristics of tests using ASRs. "We felt it was important to show that FDA doesn’t have to come in and regulate these tests, because our laboratories are doing a good job of verifying their laboratory performance," Dr. Sarewitz says. In the molecular pathology checklist, the new section on validation of test performance shows what is required of CAP-accredited laboratories. For instance, a new question (Mol. 31320) asks: "Is the clinical validity of each newly implemented assay documented, using either literature citations or a summary of internal study results?"

Industry, led by Roche Diagnostics, has come up with its own proposal, called the In Vitro Analytical Tests, or IVAT, process. In this new category, which lies somewhere between an ASR and 510(k) clearance, a manufacturer would provide analytical validation and performance parameters for a new test but would not generate clinical data. Response from laboratory directors is mixed. Dr. Farkas notes that the IVAT proposal was "juxtaposed" to the AmpliChip CYP450 incident.

"I think the pathology and laboratory medicine community is cautiously in favor of IVAT," he says, "as long as we don’t need to do any more extensive level of validation under a new IVAT rule. If, however, IVAT gets approved with the implicit understanding that laboratories have to do more than we used to do under CLIA, that we have to do what manufacturers used to do, that would be onerous."

Dr. Leonard sees an advantage of IVAT. "Industry now sells ASRs but cannot provide all the other reagents needed for their use," she says. "To my mind, IVAT would move one step beyond that and allow us to purchase in one box everything we need to do a test, including controls."

Dr. Ferreira-Gonzalez agrees it would be advantageous to be able to purchase a complete set of reagents that have been quality-controlled and certified to perform at a certain level. She also acknowledges the potential problem of having to do clinical validation for new markers. She adds one pro and one con. "Under IVAT, we will have many more laboratories performing a specific test with the same kit, so it would make it easier to compare results among laboratories and better understand the clinical indications and limitations of the test," she says. On the other hand, she asks, "What is the incentive for manufacturers, once they have an IVAT product, to move to an FDA-cleared product?"

Says Dr. Gutman: "The IVAT proposal is based on bringing tests to market using an analytical base. And in fact that is a common mechanism that the agency uses for plebian tests like hemoglobin or rheumatoid factor where there are well-established uses of markers. When a new test isn’t so well established, and the link between analytical and clinical data is missing or weak, I think what the IVAT model would like us to do is to overlook that weakness. And there is some tension between that idea and the basic mission of our work group. We are kind of grappling with that."

In the meantime, molecular laboratory directors need to continue their work under uncertain circumstances. "What are you going to do?" asks Dr. Leonard. "We just have to wait. And changes to the ASR regulation may not even happen. So why worry about it until it happens?" She agrees that the FDA’s goal is focused and specific recategorization of tests and that the agency is unlikely to do anything that would be "horrendous" for laboratories.

ASRs were initially focused, says Dr. Hillyard, on regulating manufacturers to achieve a higher reagent quality. "That is a good thing," he says. "The trouble is that test design and validation are equally important components of quality testing. Also, the field changes so rapidly that regulations may be quickly outdated." For this reason, he adds, "Laboratory directors need to take responsibility and to be very communicative, using dialogue, posters, peer-reviewed articles, communicating about issues back to manufacturers." One idea, in Dr. Hillyard’s view, would be to hold more workshops to exchange information on specific ASRs, such as those that were held on the general subject of CMV and HSV testing at the last Association for Molecular Pathology meeting.

"It comes down to practice of medicine issues," Dr. Grody says. "Laboratory directors make these decisions every day. That is what we are trained to do. We don’t need someone breathing down our necks second-guessing what we do anymore than there needs to be a government person in the operating room guiding the surgeon’s hand and prescribing what kind of incision to make."

At the FDA, says Dr. Gutman, "We’re trying to decide what is going to happen next. We would like to provide some kind of further communication or guidance to clarify our role, though I am not sure what format that will be in. And I am not sure of the timeline." He is sure, however, that whatever they do will "allow for input from professional groups and from industry."

William Check is a medical writer in Wilmette, Ill.