Amy Carpenter Aquino
September 2021—A stem cell transplant patient at Lurie Children’s Hospital in Chicago had a disseminated fungal infection by every clinical criterion, but no conventional method had detected it. The child had been on at least three months of broad antifungal coverage when William J. Muller, MD, PhD, attending physician for pediatric infectious diseases, and colleagues convinced a company that offers metagenomic next-generation sequencing to test a sample from the patient under a research protocol.
The company identified Candida dubliniensis, “which was pretty impressive to our stem cell team who thought nothing would come of it,” Dr. Muller said. “We got buy-in from them that there were some specific situations where this might be a useful test.”
Metagenomic NGS is costly and its clinical utility in some cases is far from clear. It comes down to knowing for whom to order it, say Dr. Muller and others who debated its pros and cons in an AMP session last year.
With Dr. Muller on the panel were Steve Miller, MD, PhD; Debra Lynn Palazzi, MD, MEd; and Stephanie Mitchell, PhD, D(ABMM). Erin Graf, PhD, D(ABMM), of Mayo Clinic, moderated.
Lurie Children’s physicians use mNGS on patients for whom they suspect invasive fungal infection, said Dr. Muller, who is also an associate professor of pediatrics (infectious diseases) at Northwestern University Feinberg School of Medicine. “This could be prolonged fever in the setting of broad empiric antibacterial coverage and probably also some antifungal coverage.” It could be abnormal imaging or a skin rash consistent with a disseminated fungal infection.
“We’ve had some success,” he said. Of about 300 patients tested, an invasive fungal infection—not otherwise diagnosed by conventional testing at the time mNGS was sent—was diagnosed in at least 11.
“Most of those were aspergillosis,” Dr. Muller said. “Some of them were actually multiple infections,” the most impressive example of which was a patient with AML who presented in sepsis and had a skin rash that was culture-positive for Candida. “We sent this test because we were concerned the Candida was not the only thing there,” he said. Aspergillus and Mucor were identified also.
The patient subsequently had a skin biopsy that grew Mucor and a lung biopsy that grew Aspergillus, Dr. Muller said. “This was all diagnosis that was made prior to conventional testing confirming it and allowed us to focus broad coverage and be aggressive about debridement in the skin lesions of this patient.”
Not every test sent will be informative for treatment purposes, Dr. Muller said. When talking about clinical utility, “we have to define what it means and whether a test’s result is clinically useful or not useful because of a problem with the test or because of a problem with why the test was sent.”
The time to think about the clinical utility of mNGS is when the test is being ordered, said Dr. Steve Miller, director of the University of California San Francisco clinical microbiology laboratory, which has validated mNGS during the past several years and has experience with it in the UCSF patient population. With tests sent to reference labs, he said, “that is a key point that can get lost.” At UCSF, the laboratory reviews all mNGS tests before performing them to ensure they meet clinical criteria. Those for whom mNGS is best used are immunocompromised patients, he said. “They have the broadest differential, and they are sometimes the hardest to diagnose because they have these kind of occult hidden sources and they’re not able to clear them well.”
Good sample collection is critical, he said. “Most of our experience is in CSF, and we like to see some evidence of inflammation unless the patient is highly immunocompromised,” said Dr. Miller, who is also a professor of laboratory medicine at UCSF. “In the absence of that, the pretest probability is quite low, and the pretest probability can guide when to order those tests.”
In the majority of cases tested by mNGS, he said, there is no detection. But a result that is negative or undetected by mNGS can have an impact on patient care in the right situation, “because you’re looking for severe infection, often in an immunocompromised patient, and if you’re not able to find that, you can effectively rule out certain classes or types of infections,” he said.
When he and colleagues looked at the cases missed by mNGS, they fell into three categories, one of which was very low viral load in patients. “Usually these were human herpes viruses that are commonly latent infections and probably less clinically significant,” he said.
Another is the case of such high host background that the test was analytically compromised. “If there’s high human background that can affect the ability to detect, we make a note of that,” Dr. Miller said. The number of such missed cases is small, he added, and “usually things like bacterial meningitis with Staphylococcus aureus or another virulent organism.” They end up growing in culture but went undetected by mNGS.
Patients who were diagnosed by serology only is the third category and not really a false-negative, Dr. Miller said. “A common pathogen for this is West Nile. The period of viremia is very short, and so unless you’re able to catch them in that specific window, no method that looks for organism—either nucleic acid by PCR or metagenomics—is going to detect it. And so you end up making the diagnosis by serology.”
But even a negative mNGS result has value. “It will tell you this patient does not have a rip-roaring level of organism in that body site, and so it may be reasonable in the right situation to start the patient on a steroid or immunosuppressive agent,” for an autoimmune or noninfectious condition, “and follow them closely.”
Of positive mNGS results, Dr. Miller said he has seen a number of cases in which organism detection prompted further workup, such as in cases of Brucella and Mycobacterium tuberculosis. In one case of the latter in which there were not enough reads to call it a definitive positive but just enough reads for the laboratory to consult with the providers, “they were actually able to grow TB in that patient”—they were able to get multiple collections and culture about a liter of CSF—“and make the definitive diagnosis. In the meantime, they were able to manage the therapy.”
In another case, Dr. Miller said mNGS testing detected Saint Louis encephalitis—thought to be absent from the Arizona and California regions—in a transplant patient with encephalitis and meningitis symptoms. “You’re able to make these unexpected calls and that does make a major impact in those patients,” he said.
[dropcap]A[/dropcap]t Texas Children’s Hospital in Houston, about 30 to 50 mNGS tests are ordered per month, and the hospital has its own success stories. But Dr. Debra Lynn Palazzi, the hospital’s medical director of antimicrobial stewardship, speaking for the con side of the debate, shared examples of when mNGS testing was “almost a hindrance.” “That’s when we are sending the test where there is clearly not a gap in what conventional testing can offer,” and that goes to what Drs. Muller and Miller said “about being thoughtful about the clinical scenario, the specific patient situation,” she said.
In Dr. Palazzi’s experience, the results of conventional tests, if tests are available, are returned faster than mNGS results. “It’s costly to send it as a confirmatory test for something you already know that fits the clinical situation. It’s not useful to confirm the patient doesn’t have bacteremia, as an example, which we’re seeing happen in our severely immunocompromised patients,” said Dr. Palazzi, professor of pediatrics at Baylor College of Medicine. “For some reason, there’s a question: Can we trust our blood cultures?”—despite there being a long-standing effort at Texas Children’s to ensure blood cultures are drawn properly with appropriate volumes. “So there’s not a gap in blood culture methodology. We’re trying to discourage that practice that we’ve started to see.”
She and colleagues haven’t found mNGS helpful in the “I’ve-tried-everything-else-and-I-don’t-know-what-to-do patient,” she said. She agrees that a negative result can be clinically helpful when a provider is trying to pivot away from infectious diseases, but using mNGS “to prove nothing else is worrisome would not be our recommendation.”
The Texas Children’s laboratory and clinicians have found metagenomic testing useful in situations in which it has rapidly led to information that has had an impact on patient care and outcomes, Dr. Palazzi said. “That’s where I think the focus needs to be for this test: When is it impacting decision-making and clinical outcomes?” That is not yet clear in the literature, she added.
In one case of a host who was not immunocompromised, a mildly premature infant had brain abscesses that were not amenable to drainage. “This wasn’t a micropreemie,” Dr. Palazzi said. The patient had a positive blood culture for Bacillus, but it was almost discounted because the patient had no indwelling catheters. The Karius mNGS assay was performed and in fact there were Bacillus brain abscesses. The result solidified the diagnosis and helped target therapy.
In a recent case of a newly diagnosed patient with leukemia, Dr. Palazzi’s infectious diseases service was called to see if therapy for pneumonia could be stopped after 10 treatment days, with the patient still febrile, so chemotherapy could be provided.
“I can count on one hand the number of times I’ve had a newly diagnosed leukemic patient with a lobar infiltrate that was bacterial that wasn’t responding to care,” Dr. Palazzi said. “And sure enough, we did send a Karius—because this wasn’t a patient anybody was going to biopsy at this time—and that patient had Rhizopus and ended up being rapidly diagnosed.” The patient was already on amphotericin, so the diagnosis did not alter that part of the management. But the patient ended up having a partial lung resection of that area, which the physicians would not have performed without that diagnosis.
Dr. Palazzi said she and colleagues have several similar stories but that in their experience, mNGS has been most helpful when there is a lesion of some type that is not amenable to drainage or in a host in whom a procedure would be problematic.
UPMC Children’s Hospital of Pittsburgh has less experience with mNGS than the other institutions have, but Dr. Stephanie Mitchell, medical director of the clinical microbiology laboratory at the time of the session (she is now director of medical affairs at Cepheid), shared a case from her fellowship days of a patient who was encephalitic. “All the standard-of-care up-front testing was negative, including two independent HSV PCRs on two different taps,” she said.
There was discussion with neurology and infectious disease in the laboratory about the utility of sending the patient’s samples to Dr. Miller’s laboratory at UCSF, and the decision was made to do so. The results were “technically negative,” Dr. Mitchell said, but there was a comment about the UCSF laboratory having detected two areas of the HSV genome that weren’t quite meeting the criteria for positivity. “Given the significance of finding this particular pathogen in CSF, they felt like they needed to let us know it was there.” The ID team was unsure how to interpret this result, especially with two prior HSV PCR tests having been negative.
“Ultimately, they decided they couldn’t ignore it, so they started acyclovir,” Dr. Mitchell said. In the meantime, the laboratory had sent out serologies to test for various arboviruses, and the results were positive for the Powassan virus.
“Anyone who’s tried to send off Powassan knows it takes a long time because it goes to the CDC,” Dr. Mitchell said, and the delay makes it harder to manage the patient’s care. This case highlights a variety of interpretive difficulties and considerations for mNGS, she said, adding, “There is a lot of gray area” when it comes to what has impact.
[dropcap]D[/dropcap]r. Muller and colleagues at Lurie Children’s Hospital have not found plasma mNGS to be particularly helpful in ruling out infection, Dr. Muller said, which supports Dr. Palazzi’s comment about confirming a negative blood culture. “I think people intuitively think, ‘Well, maybe the organism is still there, it’s being treated, we can’t pick it up on a culture that will find DNA.’ And we have not found that to be the case,” he said.
That has not been well reported in the literature, he added, though Karius reported several years ago at an infectious disease conference that for patients with sepsis who had received at least three days of treatment, their pathogens were not able to be detected on metagenomic sequencing even though they knew what the organisms should be.
“I would say that has been our experience as well,” Dr. Muller said, “so I don’t know, Dr. Miller, if you think there’s a difference between CSF and blood or plasma as a matrix in terms of negative predictive value, but we’re really cautious about using it for that purpose.”
It’s common to find low-level signals to HSV, CMV, or HHV-6, and it is difficult to assess them because they can be latent infections, Dr. Miller of UCSF replied, noting metagenomics and DNA testing aren’t able to distinguish between a latent and an active virus.
“In some cases, because we do RNA sequencing as well, we’re able to find reads in the RNA portion, which is evidence of active viral replication, at least,” Dr. Miller said. “This is all part of the interpretive report, and it’s important that we try to communicate this from the lab side.” The clinician, too, he said, must have a better understanding of the possibility of there being a latent organism or something that can be detected but may not be clinically significant.
Dr. Miller described the Texas Children’s Bacillus case as interesting, “because if we found that in a culture, even of an abscess fluid, we would probably write it off as being a contaminant. Probably 99 percent of the time that’s the case. Then there’s the tiny sliver of cases in which it’s real.” No test will be “the definitive end-all, be-all,” he said, and clinical context is vital. “Is this consistent with the patient? We know that in infants, Bacillus infections can happen” and there can be multiple abscesses. If Bacillus were detected in an adult patient with nonabscess-forming meningitis, it probably would not be a consideration.
Sample types do make a difference, Dr. Miller said. “Most of our experience has been with CSF where you’re assessing the actual intrathecal space for organism. That’s typically thought to be the site of infection.”
Plasma is trickier, he said, because from what he has seen in the literature and from UCSF’s own data thus far, “in cases where you do suspect a true bloodstream infection with a fastidious organism or one previously treated with antibiotics, those sites are more reasonable cases. We’ve certainly seen good utility in endocarditis cases.”
The majority of the plasma sample requests come when a patient has a suspected localized infection and clinicians want to do a liquid biopsy to see if an organism is present. That will not rule out infection, Dr. Miller said, because the negative predictive value is simply not high enough. A positive result can be helpful in those cases.
Dr. Miller said he advises his infectious disease and medicine colleagues that if they’re going to pursue mNGS testing, they need to assess the infectious site in some way. “Is there a plan for biopsies? If there’s a lung infection, is a bronchoscopy being performed? How do we look at that? Because the Hail Marys are usually pretty low yield.” The more disseminated presentations are more amenable to plasma metagenomics, he said.
“We’re all coming up with our own thresholds about what we think are good cases and not good cases to send.”
[dropcap]D[/dropcap]ozens of case reports of successful applications of mNGS have been published, but less well published, moderator Dr. Erin Graf said, are the negative consequences of positive results that do not reflect infection. She noted that Drs. Muller and Miller reported data showing their laboratories having had similar numbers of clinically relevant versus clinically irrelevant positive test results, or similar numbers of false and real detections.
What, she asked Dr. Mitchell, do you think about the potential unintended consequences of clinically irrelevant positive results?
“This is where I fall so strongly on the con side,” Dr. Mitchell said. She agrees that “publication bias” exists in favor of mNGS successes but said more data are being reported on negative consequences, which she said will be helpful in developing criteria for which patients benefit.
Specificity becomes an issue with plasma-based samples, Dr. Mitchell said. “We’re finding there’s a lot of DNA background that we pick up using metagenomics, and it can be hard to interpret some of those results, especially if you see something that could be a true pathogen in a certain patient population.” Sending this test broadly without that key pretest probability can make it difficult to interpret the result. “And in the absence of having a medical director at the lab, or a dedicated ID physician who’s willing to look over this data, you can imagine there would be overtreatment or potentially misdiagnoses.”
Her biggest fear: “A false-positive will lead them down a rabbit hole that prevents them from looking at other reasons for the patient having these symptoms and making a true diagnosis.” It all comes down to developing criteria, she added, so the test is done on the right patients. Sensitivity “can be thrown in there as well,” she said, “especially in the highly cellular samples.” And contamination can be introduced at any point in the process.
The sensitivity issue makes her wonder if mNGS will replace conventional PCR testing, “or are they going to be sending everything plus this? And what do you do with discrepant or unexpected results?”
Texas Children’s Hospital has had sensitivity problems, too, “for the reasons described,” Dr. Palazzi said, noting missed cases of Mycobacterium tuberculosis, HSV, and a case of Staphylococcus aureus and Moraxella catarrhalis (Niles DT, et al. J Clin Microbiol. 2020;58[11]:e00794–20). And specificity is a limitation. In a retrospective study of cases, “the vast majority had organisms identified for which there was no clinical way that it could explain disease or the lab didn’t know what to do with the results,” and excessive antibiotic use results. “That’s a huge limitation,” she said.
[dropcap]S[/dropcap]ensitivity may be limited for a variety of reasons, UCSF’s Dr. Miller said, and some can be overcome. “Some of our earlier cases were gaps in the database more than the sequencing data itself. Over time, I think those holes will get plugged.” But in cases in which the organism counts in a specimen are low, there are inherent sensitivity issues, he said, “and that speaks to how confident you can be that this is a good test and whether you are willing to accept that.”
For tuberculosis, his laboratory does a lot of PCR testing. Culture is the gold standard, and he explains often that PCR is faster, not more sensitive. Metagenomics is “comparable to PCR in terms of detection sensitivity,” he said, “but a low organism load is going to be an issue.”
“The specificity issue is real,” he said, but it can be addressed by considering and acting on the results in the clinical context rather than treating every organism as a pathogen and increasing antibiotic use.
Contamination as a cause of false-positives can come from the source itself during collection, most commonly a flora, whether it’s skin, oral, or GI. “Sometimes teasing out what might be significant from that is difficult,” though in some ways not much different from assessing organism growth in culture, he said.
“We’ve learned there are ways you can assess it in terms of the metagenomic sequencing data,” Dr. Miller said. When flora is seen in a sample, there tends to be a variety of species and a spectrum of organism detection, whereas in true infections there are fewer organisms that can be collapsed into one or two categories.
“In that way, I think of it almost like a urine culture,” he said, “because if you get a urine sample that’s growing a lot of different Gram-negative rods, that’s probably just a contaminated urine.”
In metagenomics, there could be flora, laboratory contamination, or environmental species, “and that is a challenge we try to address with good technique and controls.” But in some cases the samples are contaminated with organisms that may be clinically significant, such as Ralstonia or Pantoea species. “Those can be difficult to assess,” he said.
“Looking at this from the whole patient perspective is important,” Dr. Miller said, “so we try to make it clear in the interpretive portion of the report what the assessment is from the laboratory side.” UCSF’s clinical microbial sequencing boards offer opportunities for discussion with providers.
And that’s an important distinction, in Dr. Mitchell’s view. “We have to remember,” she said, “that we’re in an academic bubble and have opportunities to collaborate with infectious disease and to make these multidisciplinary teams.” She worries about commercial laboratories that perform mNGS on urine samples sent from clinics with potentially complicated patients, “where they think it’s an advantage. They are sequencing and working up everything. Even if it’s seven different organisms, they consider that significant and are treating as if the patient has a UTI with all of those organisms.”
“The answer to that is education and developing guidance documents,” replied Dr. Miller, who said the practice she described is not supported by the literature, especially for a nonsterile site like urine.
“This isn’t just a lab medicine problem,” Lurie Children’s Dr. Muller pointed out. “This is a societal problem. Just look at the Internet for examples of misuse of technology or at least use in a way that wasn’t intended.”
Of a finding that is not clinically relevant, Dr. Muller asked, “Is that a limitation of a test if it tells you something about the patient? I don’t know if I would say that.” He does admit it requires an understanding of how to interpret the information, “which not everybody is equipped to do.”
[dropcap]F[/dropcap]or Texas Children’s, the cost question is an open one. “We haven’t done a deep dive because it’s so multifactorial,” Dr. Palazzi said. “We know these tests just experientially are not decreasing costs in our hospital because it’s not avoiding other testing. It’s almost like it results in even more testing sometimes to confirm or refute what is found.”
Most mNGS testing is performed on inpatients, and with UPMC having its own health insurance plan, UPMC pays for the metagenomic testing when it’s done. “That takes away from the ability to acquire new equipment, new laboratory staff,” at a time when the lab is facing staff shortages just for routine testing, Dr. Mitchell said. “When you pull that money away, we start to lose the ability to do other things elsewhere in the lab.” Thus the need to use mNGS on the right patients.
The problem, Dr. Muller said, is not the cost of the test per se but the way health care is paid for in the U.S. And cost does enter into the calculation at Lurie Children’s for when and whom to test. “We are not sending it on patients who were previously healthy and will likely get better with empiric treatment,” Dr. Muller said. “That will be incrementally not as cost-effective as sending it on the patient who’s already cost the health care system hundreds of thousands if not a million dollars and for whom the incremental cost to that patient’s care in the big picture is low.”
Dr. Miller agrees the cost benefit is likely to be seen in the patients for whom the most health care resources are used. “If you want to predict who is a potential beneficiary based on cost,” he said, “a patient who is in the ICU is a high predictor.”
At UCSF, he said, patients scheduled for brain biopsies are also a high predictor. “If you can manage them better with the results, whether positive or negative, you can potentially decrease their overall cost of care,” whether by avoiding the biopsy or reducing length of stay or therapy, for example.
Metagenomic NGS was piloted at Texas Children’s Hospital for a period, at no cost. The hospital’s laboratory stewardship committee initially wasn’t involved in mNGS testing, but the committee has been collecting data with an eye to using the data to develop an algorithm and negotiate for resources, Dr. Palazzi said. Someone has to look at the tests, she said, “and we are in the process of determining what that’s going to look like.” In addition to the financial impact on the lab, there’s an antimicrobial stewardship impact.
“When I think of costs from a stewardship standpoint, I think of all the unintended consequences of the adverse effects of the antibiotics for our institution and the individual patient,” Dr. Palazzi said. “We are in the process of presenting the data and trying to come up with a rational plan to start stewarding this resource in a better way.”
At Lurie Children’s Hospital, “the lab medicine folks were initially very skeptical” about the impact of mNGS, which explains the laboratory’s oversight for all testing that’s done, Dr. Muller said. “It does matter that people are paying attention.”
Said Dr. Mitchell, “Healthy skepticism is good because it causes people to think and pause and put a policy in place.”
Amy Carpenter Aquino is CAP TODAY senior editor.