Amy Carpenter Aquino
July 2020—A machine learning algorithm, used in conjunction with BioFire’s Syndromic Trends, demonstrated a mechanism for near real-time outbreak detection of enterovirus D68, says a study reported in the Journal of Clinical Virology.
To predict an outbreak, the algorithm, called Pathogen Extended Resolution, or PER, identified EV-D68 among respiratory panel data obtained from routine testing of patients with infection.
The authors, reporting in the recently published study, say that using the epidemiology software service Trends in this way “could provide a potential early-warning system of EV-D68 predictions from BioFire RP tests as well as augment public health response through increased situational awareness of novel pathogens” (Meyers L, et al. J Clin Virol. 2020;124[3]:104262).
Nationwide Children’s Hospital in Columbus, Ohio, was one of six institutions in the United States and Europe to contribute to the creation of the PER algorithm for detection of EV-D68. Amy Leber, PhD, D(ABMM), the hospital’s senior director of clinical laboratories and director of microbiology and immunoserology, shared her experience with Trends and PER at last year’s AMP annual meeting and in a May 22 interview with CAP TODAY about that presentation and SARS-CoV-2.
“The session at AMP was designed to look at artificial intelligence and its impact in the clinical lab, particularly in molecular diagnostics. The example I gave was about predicting EV-D68 from the pattern of results for rhinovirus and enterovirus PCR results” on the BioFire FilmArray respiratory panel version 1.7. “BioFire was able to use machine learning to look at those targets and certain aspects and say, ‘This looks like it would be an EV-D68.’”
In the EV-D68 study evaluating the PER algorithm, one of the first sites to experience a significant increase in EV-D68 during the 2018 monitoring period was Nationwide Children’s, which confirmed the outbreak and put specific testing into place.
Could Trends or PER be useful in the fight against COVID-19? SARS-CoV-2 does not cross-react with any of the four coronavirus targets on the FilmArray respiratory panels. The FDA on May 1 issued emergency use authorization to BioFire for its FilmArray respiratory panel version 2.1 for detection of SARS-CoV-2 and 21 other pathogens. “They’re adding it as a separate, discrete target,” Dr. Leber says, so there could be real-time monitoring of COVID-19 as more laboratories adopt and submit test results to the Syndromic Trends central database.
“Let’s say COVID-19 dies down in its current wave. But if we’re testing for it routinely in all respiratory samples, it may help us during the flu season because there are several common symptoms. There’s no real pathognomonic presentation for COVID-19.”
For SARS-CoV-2, Nationwide Children’s Hospital’s positivity rate for inpatients has remained relatively low, Dr. Leber says. None of the patients thus far has developed the multisystem inflammatory syndrome sometimes linked to the infection in children.
“I’m part of an infectious disease research group that is very interested in that,” she says. “Part of this syndrome is going to inevitably involve the immune response to the virus, so we’re looking at ways to see which genes are turned on and off in these children. If we could develop a model that looks at a number of laboratory tests, and which genes are turned on and off, to try to predict who might develop this syndrome, that’s a very powerful tool for understanding the long-term effects of this.”
It would take a multidisciplinary approach and the use of multiple tools to develop a profile of the syndrome that’s affecting pediatric patients. “The common belief early on was that this was benign in children and we didn’t have to worry,” she says. “But we’re three months into this disease and we have so much more to learn.”
Data sharing can help ensure there is enough pediatric patient data to build an effective predictive tool for the multisystem inflammatory syndrome. “Conceptually, there are examples of sharing of data already with COVID,” Dr. Leber says. Google’s DeepMind, for one, has links to open-access databases to help people use sequences to predict protein structures that can be used to develop vaccines or drugs. “To be successful at this, we have to be able to share the data,” she says.
If Dr. Leber had given her AMP presentation during the pandemic, she says, she would have driven home the importance of predicting and having an early-warning system for new and emerging disease. To that point, “Should metagenomics or next-generation sequencing be used as a standardized test up front for viruses, as opposed to the approach of using a FilmArray that tests for 20 viruses? NGS can test for any virus,” she points out.
In her view, the concerns about the cost and complexity of NGS are outweighed by the need for early warning and detection of emerging viruses, and to be able to share data.
“I think this will change how we think about viral diagnostics because we don’t know what the next emerging pathogen is.” Relying on a pathogen-specific approach, such as PCR for influenza or for RSV, “limits our ability to detect emerging disease.”
[dropcap]N[/dropcap]ationwide Children’s Hospital receives real-time, actionable infectious disease outbreak information as a participant in Syndromic Trends. It is one of four clinical sites in Ohio that shares deidentified patient test results from the FilmArray RP v1.7. As of March 2019, “more than a million files have been analyzed,” Dr. Leber said of the tool.
“This is the part where everyone gets paranoid because we’re sending data across the hospital firewall,” she said at the AMP meeting. Clinical site users send their FilmArray respiratory panel test results to a centralized database, and infectious disease trend analyses are fed back to the clinical sites on an individual basis and in aggregate to look at the regions on the BioFire Syndromic Trends website (Meyers L, et al. JMIR Public Health Surveill. 2018;4[3]:e59).
While the IT staff were nervous about transferring patient test results outside the hospital, “We felt secure that we could transmit these data for Trends without compromising protected health information,” Dr. Leber said. “The good thing is it’s safe; the bad thing is it’s hooked to no other information,” such as patient sex, age, or location, that could inform deeper analysis of infectious disease trends.
The BioFire RP v1.7 can detect 17 viral and three bacterial pathogens. The public respiratory panel detection rate chart, on the BioFire Syndromic Trends website, displays pathogen trends across the United States.
“We use it locally for RSV,” Dr. Leber said. “We have vulnerable patient populations and we want to know when it’s a good time to start using palivizumab.” The trigger point for Nationwide Children’s to administer palivizumab (Synagis) to high-risk pediatric patients is a 10 percent RSV detection rate for greater than two weeks. Last year the hospital reached that level in September.
Influenza outbreaks can be tracked by subtype throughout the country or in regions. “This is a powerful tool. The more people are hooked into this network, the more powerful the data.”
PER was created with the goal of diving deeper into all the data generated to determine if FilmArray influenza A targets could be used to predict novel influenza A.
The algorithm was validated using isolates from eight strains of avian influenza (H1N2, H2N2, H5N2, H5N3, H5N1, H7N2, H7N3, and H10N7) and isolates from common strains of influenza at concentrations near the limit of detection. The target was pan1. “When you have novel flu, it will be positive in the pan-flu targets but it will not subtype. That’s the first clue,” Dr. Leber said.
The other clue is that a novel subtype should have a relatively high level, as opposed to a flu that isn’t novel but is near the assay cutoff. To train the machine, they used common strains near the limit of detection, which clustered at a relatively high Ct value above the buffer range, she said. The novel strains all had influenza A interpretations, with no subtype detected, and minimum Ct values for the FluA-pan1 assay below the buffer range. “There’s a good buffer zone in between” for clear separation of the groups.
“This might be a powerful tool to determine if we’re having increased numbers of nontypable flu across the country in different regions,” Dr. Leber said.
The influenza A PER algorithm was next applied to more than 22,000 influenza A positive FilmArray test results. If a subtype was detected, the result was not a novel flu. The vast majority of the positive results were subtyped influenza A. Of the 394 samples with no subtype detected, two had higher levels of virus and were determined to be possible novel strains of influenza A, though “one didn’t pan out, and one they think was really a novel flu that emerged.”
“Here we see the power of this tool, even on a very simple level of machine learning.”
Dr. Leber points out that she talked about one manufacturer but that there are many other multiplex panels. She asks, “Could we, in the spirit of science, share these data in an agnostic way, in a cloud, to add to the power of the predictive modeling?”
[dropcap]E[/dropcap]V-D68, a picornavirus with rhinovirus-like qualities that is associated with mild to severe respiratory illness, was reported sporadically in the United States before 2014. “But then 2014 hit, and we had a huge outbreak of EV-D68. Now it occurs with a seasonality of every two years in the summer,” Dr. Leber said, and has a strong association with acute flaccid myelitis (AFM), an uncommon but serious neurologic disease.
The CDC began tracking AFM in August 2014 and had recorded 626 cases as of June 5, with peaks arriving every two years between August and October. The national outbreak pattern of AFM has been similar to that of EV-D68, though no direct causal link has been established between the syndrome and the virus (Wang H, et al. Emerg Infect Dis. 2019;25[11]: 2055–2063).
An EV-D68 outbreak hit Nationwide Children’s Hospital in summer 2014 and coincided with a sharp rise in asthma-related admissions. “We called it ‘asthmageddon,’” Dr. Leber said. “There were so many asthma admissions at a time of year when it wasn’t expected, telling us something was up.”
She asked, “Could we apply this Pathogen Extended Resolution to determine if the rhinovirus/enterovirus detections on FilmArray were actually EV-D68?”
The team at BioFire had trained the system to look at the four targets for rhinovirus/enterovirus on the FilmArray RP v1.7. “And you use various real-time PCR metrics, such as cycle time and melt, to come up with an algorithm,” which was trained and validated using data from six independent sites. The algorithm was found to have an overall sensitivity of 87 percent and a specificity of 86 percent. “Remember, this isn’t a test specifically for EV-D68. It’s just interrogating data to try to tell if there’s a signal.” This algorithm is employed now on Trends test results for field EV-D68 predictions at sites using RP v1.7.
At Nationwide Children’s, 2016 was a low-activity period for EV-D68, but the virus returned in summer 2018. Could PER be applied prospectively to a data set to verify the outbreak? Dr. Leber and colleagues compared the algorithm’s performance with that of the hospital’s laboratory-developed qualitative reverse transcriptase real-time PCR for EV-D68 (Moyer K, et al. PLoS One. 2016;11[11]:e0167111). “We did sequencing of the VP1 region of the viral genome in a subset to confirm that we really were detecting EV-D68,” she said.
Data from the PER algorithm results showed that the hospital’s predicted EV-D68 cases were in sync with the Midwest baseline throughout spring and early summer 2018. “By about July 17, I got an email from the BioFire Trends team that said, ‘We think you’re seeing EV-D68 at your site.’ There were two weeks above the threshold,” Dr. Leber said.
The laboratory tested about 3,600 patient samples on the FilmArray RP v1.7, of which about 45 percent were positive for rhinovirus/enterovirus. “We ended up with about 1,000 samples for our analysis,” Dr. Leber said. “If we assume our EV-D68 PCR as truth, it [the algorithm] had a sensitivity of 85 percent and a specificity of 95 percent, so it met and exceeded slightly the training set data.”
Thirty-one discordant samples, considered BioFire false-positives, were then tested on the lab’s enterovirus PCR, and about half were confirmed as enteroviruses, “meaning they could be EV-D68.” A few samples were excluded based on in-house testing, and the average Ct value was 35. “This may suggest that it was at or near the limit of sensitivity of our own assay,” Dr. Leber said.
The laboratory tested 60 false-negative samples with its LDT and found an average Ct value of 30.2. “Some had relatively high levels of virus, meaning there could have been polymorphisms or something that caused a less sensitive test in this case,” Dr. Leber said. She and her colleagues confirmed 10 of the samples by sequencing them for the VP1 region of the viral genome, “and indeed we were detecting EV-D68.”
A comparison of the EV-D68 PER algorithm-predicted positive case results with the LDT-positive case results revealed close concordance.
The hospital’s asthma-related admissions were compared with EV-D68 detection between June and October of 2014, 2016, and 2018. Generally similar infection peaks and durations were seen in 2014 and 2018. Lower levels of asthma-related admissions and EV-D68 detection were reported in 2016 (Wang H, et al. Emerg Infect Dis. 2019;25[11]:2055–2063).
“So,” she asks, “could we use, through machine learning or simply tallies, asthma-related admissions as they increase in the summer as our cue to start monitoring for EV-D68?”
[dropcap]D[/dropcap]r. Leber makes clear to CAP TODAY that she is not an expert in AI but someone at the front line who needs the tools AI can provide at the level of diagnostic testing. “We need the tools to help us do better with assay development, materials distribution, and predictive modeling in terms of disease emergence and reemergence, and for early-warning systems and early detection from a laboratory perspective.”
It would have been helpful in the pandemic, she says, if AI had been used to improve the management of test supplies. “Did anyone apply that in terms of looking at the emerging detection of the virus and distribution of test systems and test supplies? Because to me, it was the Wild, Wild West. We were just begging for whatever we could get.”
Dr. Leber’s laboratory team is using two different SARS-CoV-2 tests, and she was “trying desperately” in May to get other testing platforms in-house. “Much like other people, we’re going to have three or four platforms for this testing, mainly because we can’t be assured of supply.”
Amy Carpenter Aquino is CAP TODAY senior editor. For a discussion of point-of-care geospatial strategies to stop and contain outbreaks as soon as they appear in time, place, and space, see Kost GJ. Geospatial hotspots need point-of-care strategies to stop highly infectious outbreaks: Ebola and coronavirus. Arch Pathol Lab Med. Epub ahead of print April 16, 2020. 10.5858/arpa.2020-0172-RA.