3 new NGS Surveys on CAP 2016 PT launchpad

Anne Paxton

November 2015—More than two years ago, when the CAP decided to move forward with proficiency testing for next-generation sequencing, the decision point was modest. “We estimated that about 35 labs would subscribe, based on survey information, and that was sufficient for us to move forward,” says Karl V. Voelkerding, MD, chair of the CAP Next-Generation Sequencing project team.

The first round of PT in a new area, known as the A mailing, is traditionally an educational challenge, with labs not formally graded on their results but instead provided with a summary with which to crosscheck their answers against the CAP answers. But even within this limited framework, signups for the first NGS Survey, covering germline variants, exceeded hopes. “There were quite a bit more in the end than was initially anticipated,” says Jason Merker, MD, PhD, chair of the CAP Molecular Oncology Committee. “We had approximately 130 labs enroll.”

Now the CAP plans to crown that success with a broader menu. For 2016, it will add three more sets of PT challenges in next-generation sequencing: solid tumors, hematologic malignancies, and bioinformatics.

Any time the CAP envisions a new Survey, the first question a team asks is, “Will this be a PT product that a sufficient number of labs will subscribe to, to make this a sustainable effort economically for the College?” says Dr. Voelkerding, professor of pathology at the University of Utah and medical director of genomics and bioinformatics at ARUP Laboratories. To answer that question for NGS PT, the CAP asks accredited labs to list on their activity menu whether they are using NGS.

Proficiency testing for NGS bioinformatics is one of the boldest among the CAP’s PT ventures, Dr. Voelkerding says. “It’s a very important initiative of the College, with the goal of diversifying and augmenting the types of PT challenges we can provide labs that are doing NGS.”

In the NGS world, an important component of the overall testing process is the analysis of sequence data. “It’s all done computationally, and then the summation of that data is interpreted by the laboratory director,” he says. “We foresaw a couple of years ago that we could create additional PT challenges by querying the labs’ ability to analyze the sequence data that we would provide in silico [performed by computer simulation]. We use in-silico computational tools to take a given data set and then artificially introduce into it mutations and genetic variations in areas of interest, then send those data sets to clinical labs to process through their own bioinformatics computational pipeline. And they report back to the CAP what they’ve observed.”

The first bioinformatics PT challenge will have a focus on molecular oncology and the detection of somatic mutations. Twenty-four genes are included in the 2016 NGS Bioinformatics Survey: AKT1, ALK, APC, BRAF, CDKN2A, CTNNB1, EGFR, ERBB2, FBXW7, FGFR2, GNAQ, GNAS, HRAS, IDH1, KIT, KRAS, NRAS, PDGFRA, PIK3CA, PTEN, RET, SMAD4, STK11, and TP53.

“In 2017 and going forward, there will be further emphasis on bioinformatics-based challenges,” Dr. Voelkerding says, specifically in the area of germline variation and genetic disorders.

NGS proficiency testing in bioinformatics is new, he emphasizes. “It’s a response to and reflection of the fact that NGS has a substantial and unprecedented amount of bioinformatics data analysis, and because of that, it really adds a complexity to the overall testing process that needs to be assessed. Interestingly, those testing processes can be assessed as computational bioinformatics-type PT challenges.”

Laboratories that perform NGS take diverse approaches to their information pipelines, employing varying degrees of automation. “In some cases, the way the information is generated and reviewed may involve a series of Excel spreadsheets. Some labs are likely doing the transfer and summation of their data set outputs manually, or the data set may be interfaced into a Web-based program through a commercial vendor or one that is developed internally by the lab.”

“The complexity of NGS results requires a tremendous amount of creativity in how you compile those results in a report that can be reviewed by a laboratory director and then ultimately delivered to the ordering physician,” Dr. Voelkerding says.

So there is great diversity in approach, but it’s important to assess the range of those approaches through proficiency testing of bioinformatics pipelines, Dr. Voelkerding says. In fact, an initial feasibility pilot for the NGS bioinformatics proficiency test signaled there were difficulties in accommodating sequence data formats. “We observed that the format in terms of the sequencing data we provided to the labs could become a barrier. If we provided a data set formatted in a specific way and an individual lab’s bioinformatics computational pipeline uses input data in a different format, then some labs’ pipelines were able to accommodate that, and some labs’ pipelines were not.”

The CAP, guided by the experience of the initial feasibility pilot, conducted a second feasibility pilot based on specific data formats that are generated when using commercially available reagents for molecular oncology testing by NGS for specific instruments. “The second feasibility pilot demonstrated that labs could accommodate that standardized format, because they were using specific commercial reagents and software pipelines and could analyze and process data very efficiently and very accurately,” Dr. Voelkerding says. The bioinformatics PT set to launch in 2016 is based on the approach developed during this second feasibility pilot. (John Pfeifer, MD, PhD, provided collaborative input to the NGS project team members in his role as chair of the CAP Personalized Health Care Committee.)