What’s new in next-gen sequencing checklist requirements

Create PDF

Dr. Voelkerding

But is NGS different enough from other clinical laboratory testing to justify a new regulatory framework? Even as the CAP Laboratory Accreditation Program puts forth revised accreditation checklist requirements for NGS in the new 2015 checklist edition, the Food and Drug Administration is considering whether a more direct federal role is needed to maintain quality of this powerful diagnostic testing technology, Dr. Voelkerding reports.

Historically, the accreditation program’s NGS project team conceptualized the overall NGS test process as composed of two major analytical components: a “wet bench” component and a bioinformatics, or “dry bench,” component, he says. “The first comprised sample handling, NGS library preparation, and sequencing. The second involves taking the sequencing data, processing it through bioinformatics algorithms, and analytically distilling it until it’s ready for review and interpretation by a laboratory director.” The wet bench and bioinformatics dry bench components compose the analytical NGS test process. “The review and interpretation of the NGS test results, from our perspective, constitutes medical practice. And we have maintained this general framework going forward,” Dr. Voelkerding says.

Operationally, this breakdown makes sense because the first component involves individuals working at the lab bench processing samples and running sequencing instrumentation, while the second involves individuals working at computers executing bioinformatics programs and preparing data for subsequent review. To facilitate inspection, separate checklist requirements apply to the analytical wet bench process and to the analytical bioinformatics process.

Most NGS testing in laboratories is conducted under one roof, Dr. Voelkerding says. But there are situations in which labs will generate the sequencing data, then transfer the data for bioinformatics analysis to another laboratory or service provider. Recognizing that this “distributive” model can be employed led the project team to one of this year’s checklist revisions. “We decided to make it clearer that if another laboratory or service provider is performing the bioinformatics, then that other group is operationally serving as a reference laboratory.”

The challenge was to make the accreditation requirements reflect both the appropriate tracking and the provenance of the sample. “We had to make certain that at every step in the total NGS process, there was a requirement for accountability for that sample as it moved from point A to point B, potentially from one lab to another, and then perhaps back to the primary lab for interpretation.” Revised checklist requirement MOL.35846, for example, calls for laboratories to have records of each transfer step that describe unambiguously when and how specimens and data (including file formats) are transferred and exchanged.

Validation can be complex where an NGS sample is involved, Dr. Voelkerding notes. “There has to be an integrated validation between the wet bench and bioinformatics processes, whether performed under one roof or separate roofs.” As checklist requirement MOL.36015 explains, it is the responsibility of the primary laboratory director to review and approve all validations to ensure acceptable “beginning-to-end” test performance. “When a component of the total NGS process is outsourced to a reference laboratory,” he says, “the primary laboratory director should work with the reference lab to review its test relevant validations to ensure an integrated validation for the intended test use.”

The NGS total process can be monitored step by step, Dr. Voelkerding says. “At most process steps, you can ask, ‘What are my expected results or metrics for this step?’ For example, in many NGS tests, genomic DNA is fragmented prior to conversion to an NGS library. The success of fragmentation can be assessed by monitoring fragment size distribution using electrophoresis.” Another example is that NGS library yield can be determined by quantitative PCR prior to sequencing the library, he says. “During the bioinformatics component, one can assess metrics such as numbers and percent of reads mapping to the target reference sequence.” Metrics per step can be determined empirically during test development and further established during validation, he adds, and these can include metrics for both wet bench and dry bench or bioinformatics process steps.

If expected outputs or quality metrics are defined for each step of the process, “then you would expect that, as you go through, if your processes don’t meet those metrics, you would be concerned something is not right with the testing process or perhaps something is abnormal about the sample,” Dr. Voelkerding says. In the revised checklist, the NGS project team added recommended quality metrics to the requirements so that laboratories would monitor the metrics during the overall testing process to ensure analytical quality.

Going forward, the project team will continue to define metrics while also expanding into the concept of NGS test performance standards. Specifically, the project team will work to determine if NGS test performance standards can be developed and articulated in future checklist requirements.

“Defining performance standards is an important step beyond defining quality metrics. If an NGS test meets all quality requirements,” Dr. Voelkerding explains, “then it should be possible to state how well it should detect the different types of mutations the assay is being used for.” This is an extension of stating the test’s overall analytical sensitivity and specificity by adding additional detail for these parameters per mutation type in a given clinical application or intended use, he says.

For example, during test optimization and validation of NGS multigene panel tests for germline or somatic disorders, characterizing the sensitivity and specificity for detecting different size insertions and deletions is critical. Determining the ability of the test to detect different mutation types in specific clinical applications establishes the performance of the test for the intended use, Dr. Voelkerding says. “An important next question is to determine if it is feasible to define for a given clinical application the performance standards that should be achievable given current NGS technology and associated bioinformatics.”

The move toward defining performance standards will address three major interests. “First, defining and reporting performance standards would facilitate the CAP’s ability to review and accredit laboratories, by providing another measure of quality and equivalency.

“Second, as the FDA weighs options for regulatory oversight of NGS, an approach that incorporates performance standards is of substantial interest to the FDA.

“Third, reimbursement entities are interested in measures that would allow them to know if a next-gen sequencing test ordered from one laboratory demonstrated equivalent performance compared with that offered by another laboratory.”

On the matter of potential FDA regulatory oversight of NGS testing, the FDA convened a one-day workshop in February that included panel sessions with those who work in NGS clinical diagnostics and translational research. “They had several panel discussions and then a public comment period on regulatory oversight of NGS. The public comments largely focused on the value of continued oversight under the CLIA umbrella and deemed entities, including CAP,” Dr. Voelkerding says.

A presentation by the FDA highlighted the potential of a more direct oversight role for the FDA, through its traditional mechanism of review of individual tests or, alternatively, through the establishment of performance standards that laboratories would need to meet for their NGS tests. “By considering the feasibility of defining performance standards for NGS tests,” he says, “the NGS project team will explore key questions that will more fully inform the dialogue between the CAP and the FDA on NGS regulatory oversight.”

The FDA plans to hold a second workshop on NGS regulatory oversight this November. The goal is to continue the discussion among key stakeholders, and the CAP will be represented at the meeting.
[hr]

Anne Paxton is a writer in Seattle.