Anne Paxton
January 2013—With the arrival of another flu season—this one early and intense—rapid influenza diagnostic tests (RIDTs) are once again occupying many laboratory directors’ minds. But although laboratories have found RIDTs useful for the last decade, evaluations of the test kits’ performance have been limited to manufacturers’ product inserts and a few small-scale studies. Like swing shift and day shift workers in the hospital, RIDTs have not been brought together for an assessment side by side.
A new study sponsored by the Centers for Disease Control and Prevention and the Biomedical Advanced Research and Development Authority, an agency of the Department of Health and Human Services, fills that gap. Titled “Evaluation of 11 commercially available rapid influenza diagnostic tests—United States, 2011–2012” (MMWR, Nov. 2, 2012), the study is the first to measure the performance of the commonly used RIDTs against a standardized set of representative influenza viruses. “Clinical laboratories now have their first comprehensive evaluation of the majority of the commercially available tests,” says study co-author Daniel B. Jernigan, MD, MPH, deputy director of the CDC’s Influenza Division.
In the study, researchers at the Medical College of Wisconsin tested performance of test kits made by Thermo Fisher Scientific, Becton Dickinson, Meridian Bioscience, Inverness Medical, Response Biomedical, SA Scientific, Quidel, Princeton BioMeditech, and Sekisui Diagnostics. For each of the 11 FDA-cleared RIDTs commercially available for the 2010–11 influenza season, the researchers measured the number of positive samples in progressively increased dilutions of 23 influenza viruses—16 influenza A and seven influenza B. The study used identical viral concentrations for each kit tested and a large collection of recent influenza viruses to allow for a more finely detailed characterization of test performance.
The evaluation of RIDTs was not intended to brand any particular test as good or bad, Dr. Jernigan emphasizes. Rather, the study is part of a three-pronged CDC strategy to improve rapid tests by: 1) working with the FDA and manufacturers to make the tests better, 2) working with organizations to improve testing practices, and 3) getting better information about rapid flu testing to clinicians, including partnering with the Joint Commission to develop a Web-based continuing medical education series. “This study gives us baselines that show how the tests are performing using a standard set of conditions,” Dr. Jernigan says. “And we can use the study design to continue to evaluate RIDTs available in the U.S.”
Similar studies have evaluated rapid tests in the past; however, those studies would usually compare just two or three tests at a time, not everything that was out there, says lead study author Eric Beck, PhD, now a senior technologist in molecular diagnostics at Dynacare Laboratories. Dr. Beck was with the Midwest Respiratory Virus Program of the Department of Pediatrics at Medical College of Wisconsin when he helped lead the study. “In all honesty,” Dr. Beck says, “a lot of the results that you see on sensitivity come straight out of the manufacturer’s product insert, so there are not a lot of studies out there that really compare everything kind of equally.” Moreover, many sites use different flu strains, so the results aren’t always able to be correlated from one study to the next.
The swine flu (H1N1) pandemic in 2009 sparked new attention to the quality of rapid antigen testing, Dr. Beck says. At the time, clinicians, researchers, and regulators were concerned about whether RIDTs could detect the newly emerging virus. “When the H1N1 strain first hit, the assumption was that the rapid tests were not picking it up as readily as they did the previous seasonal flu strains.” That was one of the central purposes of the study, he adds: to see if in fact the rapid tests work better or worse for the strains currently out there, especially with the 2009 H1N1 strains supplanting the seasonal strains that had been prevalent.
To answer that question, the researchers used the same samples for all the tests. “A lot of the strains we’ve used in the past are similar, but depending on where you grow them or who propagated that virus, you can get different results. In one lab, your sensitivity can look very good, whereas if a different lab propagates the virus based on how they prepare the virus stock, the sensitivity may appear lower. The point with this study was to use the same virus stock for all of the tests to make them comparable,” Dr. Beck says.
The researchers tested roughly six viruses of each subtype, then broke down the results by how many of the subtypes tested were positive at a certain concentration. “We were encouraged that at higher concentrations, the tests still picked up the currently circulating viruses for the most part. But some didn’t do that well no matter what the concentration was or what the virus was,” Dr. Jernigan says.
In general, the more positives found for any particular test, the more sensitive the test proved to be, and that’s the most critical criterion the researchers were studying in this evaluation, says Dr. Beck. “The other thing was we wanted to see across different viruses or different subtypes that the test would have similar reactivity. For any given test, you want to see that it is capable of detecting seasonal influenza, the 2009 flu, and others. Some are definitely more sensitive for influenza A than for influenza B. So if you have a season where influenza B is one of the predominant strains that is circulating, then that particular test is less effective.”
A laboratory considering which assay to use might want to look for consistency between results with different subtypes in a given test throughout multiple seasons, Dr. Beck says. “Sensitivity does have some bearing on how well the tests perform. However, the purpose of the study was not to make any claims as to which tests are best or to label them that way. It was more intended to show which tests are consistent and to come up with a way to evaluate tests in the future as to their consistency across different subtypes of influenza, to make sure any new tests are at least as good as what’s currently being offered.” It’s important to keep in mind, he adds, that this was all done analytically. “It doesn’t necessarily reflect what the performance of the tests would be in the clinic.”
Often, doctors used to performing HIV or strep tests have grown to think all the tests perform similarly, but with rapid antigen testing for influenza viruses, that’s not true. There is great variability, Dr. Jernigan points out. “There’s variability from one test to another; if you’re using a rapid test you may actually get different sensitivity than with another brand. In addition, some tests are better at detecting influenza A, while some are better at detecting influenza B.”
Most of the differences in manufacturing methods are differences in antibodies, Dr. Jernigan adds. “Each has a slightly different way of preparing specimens, and what we wanted to show in the evaluation is how these current tests are going to work with currently circulating flu viruses. A lot of the tests were designed in the late 1980s and 1990s using influenza antibodies from very old viruses, some from the 1930s and some of them from the 1960s.” These viruses remain in a lot of the research reserves and manufacturers’ reserves as kind of the “workhorse” viruses, he explains. “They have been extremely well characterized, they’re very well known, and they grow well. So they are not necessarily a bad choice.”
But it’s been hard for researchers to do a comprehensive evaluation under standardized conditions, Dr. Jernigan says, because a person and a swab cannot be replicated multiple times. “The problem is that you can’t take a person and swab them 70 times in order to evaluate all the tests equally. If you swab a person one time you’ll get a certain amount of virus, and the second time you’ll get a different amount of virus. For that reason you have to have a virus stock fully characterized in terms of the concentrations and dilutions of virus.” In this study, one mL of virus stock for one of the tests was exactly the same as another mL of virus stock for another of the tests, thus making comparisons possible. The downside of that approach, however, is that the comparison may not reflect actual performance in clinical settings. In part, that’s because there are other things in respiratory secretions in addition to what’s in the virus stock, such as proteins and cells, Dr. Jernigan points out.
During the H1N1 pandemic, laboratories’ use of the rapid antigen tests increased considerably, Dr. Jernigan says. “That was not a bad thing. It just meant that a lot more doctors became aware of the tests and started using them and that use is continuing to grow.” He believes manufacturers are continuing to make the tests better, and notes that several recent improvements such as the automatic readers were not available when the CDC began the study.
Demand for the tests could become high, as the CDC has already indicated it thinks the 2012–13 flu season could be severe. “I’m sure it will be worse than last year,” says Dr. Beck. “We didn’t really have a major flu season last year. We all got off pretty light, and Milwaukee wasn’t alone in that.” He suspects that the 2009 pandemic may also have prompted a lot of people who hadn’t had a flu shot for 10 years to get immunized, and that may have helped tame later outbreaks. “But generally you assume viruses are going to mutate, and there’s always something new coming around the corner. Once it gets to the point where a virus has mutated enough that people aren’t immune to it anymore, we’ll see what we saw in 2009. You don’t necessarily think every three years there will be a pandemic, but the thought is always there.”
However, the need for the rapid tests has to be balanced with caution in using their results, the CDC has emphasized. “One conclusion was these rapid flu tests should be used cautiously. The specificity of the tests is pretty high, so if you get a positive it should be a true positive and you can probably take it to heart—especially if you are using the information for infection control purposes or trying to figure out whether to increase treatment for a child or do prophylaxis for a grandmother with multiple underlying conditions—things like that,” Dr. Jernigan says. But the sensitivity varies, which means you may be getting a false-negative for some of the tests, especially for specimens that are related illnesses or are poorly collected. “If a pregnant woman comes in with a bad respiratory illness and the rapid test is negative, you do not want the doctor to treat the patient based on that result. We saw clearly with H1N1 that pregnant women were having illness and even death from flu. So for certain patients, you need to use caution.”
Expert opinions vary, but the gold standard for influenza testing is technically still culture, Dr. Beck says. “They have rapid cultures that they can do in a day or two rather than five days, so culture is much faster than it used to be. But still, it is much slower than PCR, or rapid antigen testing for that matter. And most clinicians will say they would order a PCR assay before they’d order a culture.” PCR testing has made great progress in speed and practicality, Dr. Beck notes. “It’s improved quite a bit and a lot of that has to do with greater automation. The new platforms that manufacturers are producing make it possible to run a small number of tests at a time in a cost-effective manner. With some of the more conventional PCR tests, it isn’t that they take all that long necessarily; it is just that to do them in a cost-effective manner, you have to batch the tests.”
Though he strongly favors the use of PCR testing for detecting influenza viruses, Dr. Beck isn’t sure that PCR will inevitably take the place of rapid antigen testing. “As manufacturers continue working on RIDTs, they are getting more and more sensitive, and they will still always be faster than PCR. However, I suspect the manufacturers are going to have to work fairly hard to make sure their sensitivity is closer to PCR if they want to persist, because with a lot of newer PCR tests, you can get results out in maybe two hours now.” He says that might still be too long to expect patients to stay on site. “I guess I would not sit there for two hours.”
Laboratories in the field report that those kinds of turnaround time issues with ambulatory patients continue to be pivotal. In Richmond Heights, Ohio, for example, the flu season was already underway by early December and was running well ahead of last year’s numbers, says Sherri A. Gulich, MT(AMT), laboratory supervisor for University Hospitals Richmond Medical Center, a campus of UH regional hospitals. In the first 12 days of December, the laboratory reported three results positive for influenza A and six results positive for influenza B out of 43 tests, compared with only one positive test over the same 12 days in 2011. But Gulich feels the laboratory is well prepared for the flu season with the BD Veritor system it acquired last June to handle rapid flu test orders, the majority of which come from the hospital’s emergency department.
The Veritor system’s ease of use and a five-minute faster turnaround time are two of the features that attracted Gulich. And an automated reader makes the Veritor more standardized and objective than the laboratory’s previous system, she says. “With the manual test, if something is on the paler side, you might be asking five people ‘Does this look positive or not?’” So while the automated reader costs a little more—about $350 for a reader that is good for 3,000 tests—Gulich feels that the increased sensitivity and standardization make it well worth the additional cost. If the test result is negative, the laboratory does not reflex for verification by PCR or culture unless the doctor requests it, she says.
However, the rapid flu test is not routinely used on patient floors. “They will request a PCR rather than the rapid test because of the specificity of the PCR. It’s what our infection control doctors request, whereas we have a lot of walk-ins in the ED and the rapid test is used more as it would be in a doctor’s office.” To stay prepared, the laboratory tries to keep an extra 100 tests on hand. “It’s not unusual to go through that many tests in less than a week. If there’s a heavy flu season, our ED would get inundated.”
At the Ashtabula (Ohio) County Medical Center, similar reasons drove adoption of rapid testing. “When H1N1 first hit, everyone switched to something that had to be faster,” says Dana Skaggs, MT(ASCP), director of laboratory services. The laboratory now has three BD Veritor devices and is considering acquiring an RSV product. With the 10-minute turnaround time of the Veritor, she says “the nice thing is the technologist can start the device, walk away, come back, and read it. They’re not being prompted to add extraction reagents, and the reader takes away all the subjectivity.” The laboratory will bring in molecular testing in 2013, Skaggs says, but it has no plans yet to evaluate flu testing on a PCR platform. “We’ll keep on the rapid platform for right now and see what happens with the flu season.”
One of the measures recommended in the CDC study is an often overlooked part of flu testing, whether through PCR or rapid antigen testing: proper specimen collection. “If you had a swab and didn’t get enough respiratory secretions in the nose, you’re likely to get a negative result,” Dr. Jernigan points out. “So one way to improve performance is by appropriately collecting the specimen. Sometimes the clinician or nurse may take a swab and just do it lightly so as not to bother the patient. If it’s a kid who is already crying, you just end up getting a little bit of a swab around the inside of the nose. But the best way to collect is a nasopharyngeal swab, to get in as far as you can appropriately and swipe to get the lining of the respiratory tract inside the nose where the virus is growing.”
The point is that if the existing test has a certain performance, and it’s not the optimum, “there are still other things you can do to increase the performance of this test, and one of those is to have a really good specimen collection,” Dr Jernigan says.
Dr. Campbell
Another way to improve RIDT performance is to make sure specimens are collected at a point in time when the virus load is highest in the nose, which is usually about 24 to 48 hours after symptom onset. Timing is even more important in rapid testing than in PCR testing, says Sheldon Mark Campbell, MD, PhD, associate professor of laboratory medicine at Yale University School of Medicine and director of laboratories at VA Connecticut Healthcare System. “If rapid tests are going to be performed at all well, you have to get an awful lot of virus. You can’t take someone who got sick four days ago and still feels bad and do a rapid flu test and have any idea what’s going on with that person.”
“Generally,” Dr. Beck notes, “flu antivirals that get prescribed are most effective when they’re administered within the first 24 hours of symptom onset.” But in addition, all the tests are more sensitive if they are timed correctly, within the 24- to 72-hour time frame. “If you’re at the point where you’re shedding the most virus, you have the most likelihood of getting an accurate positive test, which is earlier in the infection.”
Getting inside that window is sometimes going to mean educating the front desk staff, educating the nurses who answer the call lines physicians use, and stressing that for certain individuals it’s going to be especially important to know if they have flu symptoms, says Dr. Jernigan. “Those are high-risk people with underlying medical conditions such as asthma or chronic obstructive pulmonary disease, or they are recently hospitalized or on antibiotics. Those circumstances can precipitate heart disease if they have a bad flu infection.” Those people should be encouraged to come in for testing earlier, he says.
However, the CDC study cautions, its findings do not reflect performance of the RIDTs in clinical settings. “I think in terms of clinical setting performance, your best bet is to look at the product insert and see what they say,” says Dr. Beck. “The clinical studies are comparing performance to either PCR or culture results, and those are a little bit easier to compare across sites than some of the analytical studies where you’re dealing with stocks of virus.”
“The most important discovery we’ve made over the past few years is that not all tests are created the same, and not all flu seasons are created the same, with respect to any given test,” says Dr. Campbell. “It’s clear from this CDC publication that different antigen tests to detect flu have global differences in sensitivity but also are more—or less—effective depending on the strain. None of them are very good at detecting a lower-titer virus. Many can detect a higher-titer virus pretty well, but it really depends on the strain.”
It’s important for labs, if they’re using these tests, to do several things, Dr. Campbell says. “One is to make some effort to verify the performance of the test each year in each year’s new flu. Another is to make certain that providers understand the limited sensitivity of these tests. Some people think this is a rule-out flu test and that is incorrect; none of these tests are sensitive enough to rule out flu.”
In Dr. Campbell’s view, “RIDTs haven’t really improved by any game-changing amount.” But the relatively high front-end cost of a device to perform PCR, often in the $45,000-plus range, is a barrier for many labs that are considering a switch to molecular. “There’s a wide variety of molecular tests, ranging from comparatively simple to moderately complex and extremely complex. But nothing is waived yet in the molecular world, so if you’re a physician office and you absolutely have to do a flu test, maybe the only option is a waived antigen test,” Dr. Campbell says. “Somebody’s going to get a waiver on one of the PCR platforms one of these days, but we’re still a little ways from that happening.”
With money always tight in the health care system, that there are limited treatments for influenza becomes a factor in deciding which tests should be employed. “Treatment of influenza is of modest value. The drugs work okay but not great. It’s good to rule out other things, but you can’t do that with the rapid tests,” Dr. Campbell says. “There are infection control implications, and we’re getting a little more serious about trying to prevent the spread of influenza from person to person, especially in an immunocompromised population. And in that setting, the information produced by multiplexed molecular tests or broad-range respiratory viral culture does become valuable. It’s not primarily important to know someone has paraflu type 3 because there’s no specific therapy for it. But on the other hand, by knowing that, you might save the person from having to have a lung biopsy.”
False-positives can also be a problem with the RIDTs, Dr. Campbell says. ‘There are some anecdotal and small studies where the antigen tests have false-positives, and in general a lab that is using antigen tests should be able to send early season positives and positives in patients who don’t really seem to have the flu out for molecular tests for confirmation.”
Rapid antigen testing is a technology that is reaching the end of its useful service life, Dr. Campbell says. “RIDTs have had a role, and they still do, because not everybody is going to be doing molecular yet. But I think the thing to do is move forward as much as possible with meaningfully better testing, which means the molecular technology to detect influenza and other respiratory viruses. For those labs that are going to stay with the tests, pull some of the positives and negatives out for molecular confirmation, educate providers on the performance of rapid tests to make sure they understand these are insensitive tests, and potentially provide molecular backup for compromised patients.”
“Even if you have to send the tests out for PCR, it’s probably worth doing, since those people may well be sick for a while, and if you find out it’s viral, then that’s likely to be valuable information.” But, he adds, the molecular platforms are now within the technical reach of most labs. “You still have to make the economic case for the PCR tests, but any lab running CBCs and general chemistries is technically capable of handling these platforms, so it’s time for even smaller labs to start thinking about molecular testing for influenza.”
There continues to be strong feeling that RIDTs are still useful. “I personally think they have a place,” Dr. Beck says. “ I think there is a use for them in outpatient clinics and situations where you might not be able to get PCR results based on how far the sample has to travel to get to the lab. Rapid tests probably make the most sense in a situation where you’re trying to figure out quickly whether someone is going to be on antivirals or not. At least they will give you an idea, if you have a positive result. And for the most part, positive results during the flu season represent true positives.”
But Dr. Beck questions their utility outside of flu season. “I think you certainly want to consider using them a little more sparingly when influenza is not in season. They’re probably not your ‘go-to test’ for that. It really has to be kind of a high prevalence of influenza; otherwise I think their sensitivity in general is significantly lower than in PCR testing. And most PCR testing today you can do in the same day, so it’s not a horrific matter of turnaround time. During flu season, I can certainly see their utility in the ER or in outpatient clinics. But you have to take it with a grain of salt. You can’t get a negative result and assume it’s not flu.”
Rapid flu testing is on its way out, Dr. Campbell believes, but it’s not going to disappear overnight. “As rapid, simple molecular platforms come into use, with markedly superior performance in detecting viruses, these rapid antigen tests are going to go away, but they’re not going away instantly.” For reference, as a comparative benchmark, he says, the last CAP Surveys showed there were still about 100 labs offering chlamydia and gonorrhea antigen testing, despite those tests having been outmoded for about 15 years. “So old technologies die, but they sure do go away slowly and painfully.”
With extensive rapid flu testing continuing for the near future, the CDC’s main goal is to make sure RIDTs continue to get better. Says Dr. Jernigan: “We would like to do this evaluation on a regular basis, maybe annually before each flu season. I feel encouraged with the results being reported from manufacturers with the newer tests that are coming out, and we really hope to see if this kind of regular evaluation is improving performance. That would be a great thing.”
Anne Paxton is a writer in Seattle.
The CDC has collaborated with the Joint Commission to offer a free Web-based course (with continuing education credits), “Strategies for Improving Rapid Influenza Testing in Ambulatory Settings (SIRAS),” which reviews the use of RIDT results in diagnosing and treating influenza, and uses videos, produced by Copan Diagnostics, to demonstrate specimen collection. The course can be accessed at www.jointcommission.org/siras.aspx. The videos can be accessed separately at www.youtube.com/user/TheJointCommission or www.copanusa.com/index.php/education/videos.
Rapid flu tests study—key findings in brief
- Analytical sensitivity varied across 11 rapid influenza diagnostic test kits as well as with different influenza viruses.
- Most RIDTs detected viral antigens in samples with the highest influenza virus concentrations, but test performance for all RIDTs dropped significantly with decreasing virus concentration.
- For highest virus concentrations, it is important to collect respiratory specimens within 24–72 hours of symptom onset.
- Clinicians should follow best practices for specimen collection and timing to improve the clinical utility of RIDTs.
- The evaluation does not represent RIDT performance in clinical settings.
- RIDTs should be used cautiously for diagnostic, treatment, and infection control decisions in clinical settings.
—Anne Paxton