Medical genetics labs shine in 10-year proficiency test data

While analytical accuracy was very high, clinical interpretation was slightly less accurate. “Most interpretive errors—81 percent—were due to incorrect interpretation of the lifetime risk of breast cancer for a patient with a negative result,” Dr. Weck says. The proficiency testing samples were identified as from a woman with an Ashkenazi Jewish background and a strong family history of early-onset breast cancer. Participants were given a multiple-choice question, with the best answer for a negative result being “The lifetime risk of breast cancer cannot be determined without BRCA mutation testing of the affected relative.” The correct response was given in 92.5 percent of cases. “The woman is still at greater risk than the average population lifetime risk based on her family history,” Dr. Weck explains. Without knowing whether the affected relative has a BRCA mutation, the negative result is uninformative. For instance, a deletion would not be picked up by Sanger sequencing. (Three-fourths of participants used Sanger sequencing.) However, Dr. Weck cautions, “Our ability to analyze how laboratories interpret their own results in their laboratory reports is somewhat limited.”

The cystic fibrosis Surveys provided from 2003 to 2013 were designed to test performance of the ACMG/ACOG recommended 23-gene panel. One hundred seventy-nine laboratories participated. Overall analytical sensitivity was 98.8 percent and specificity was 99.6 percent (Lyon E, et al. Genet Med. Published online July 31, 2014. doi:10.1038/gim. 2014.93). Dr. Schrijver says there was a significant trend toward higher analytical sensitivity between 2003 and 2008. Only 21 percent of Surveys in which an error occurred were distributed after 2007.

However, as with the BRCA Survey, there were interpretive errors, with the greatest variation seen in how laboratories interpreted the clinical significance of heterozygous samples. In the Survey, the clinical scenario was a child with failure to thrive. In this context, there is a spectrum of possible findings, including two distinct mutations. “One thing we have been educating around is that laboratories should know that many more mutations are possible in people with CF than the 23 on the panel,” Dr. Schrijver explains. “So finding one mutation on the panel doesn’t mean that CF is less likely.” In 10.8 percent of cases where one mutation was found, the laboratory gave this incorrect interpretation.

A Survey for genotyping and interpreting the basic Ashkenazi Jewish panel—with multiple conditions in addition to Tay-Sachs disease, Canavan disease, and familial dysautonomia—has been provided since 2006. Analytic sensitivity was 97.2 percent and specificity was 99.8 percent; analytic interpretations were correct in 99.3 percent of challenges (Feldman GL, et al. Genet Med. 2014;16:695–702). Laboratories provided accurate test results in both diagnostic and screening settings.

The number of people tested with the Ashkenazi Jewish panel increased in 2011 about fivefold, from approximately 2,500 per month to around 12,500 per month. “At first we thought there could be multiple reasons,” Dr. Schrijver recalls. “Now we know that a few laboratories started offering large carrier screening panels to anyone, regardless of ethnicity.” Expanding the pool of people screened to those outside the Ashkenazi Jewish community can be a problem—those with different ethnic backgrounds have different mutations not captured by typical panels. So a negative finding could give people a false sense of security. Dr. Schrijver acknowledges that the laboratories that are screening non-Jewish persons may have expanded their panels to contain a large number of mutations. “But we don’t know that,” she says.

CAP proficiency testing for Ashkenazi Jewish conditions was expanded in 2012 to include Gaucher disease, Bloom syndrome, Niemann–Pick disease type A, glycogen storage disease type 1a, mucolipidosis type IV, and Fanconi anemia type C.

Of the overall high quality of the results on the molecular genetics Surveys, Dr. Weck says: “Expertise in molecular testing by clinical laboratories is at a high level. Molecular pathologists and directors of molecular laboratories are molecular professionals who apply rigor in their own work and are very nimble at being able to take a new technology or technique, validate it very quickly, and use it to detect whatever new analyte needs to be tested.”

Moving from analyte-specific proficiency testing to methods-based testing, Dr. Schrijver noted that this approach is already established in CAP PT programs for immunohistochemistry, FISH, and flow cytometry. “Now with the advent of next-generation sequencing, which has a very large scope of genetic testing, analyte-specific proficiency testing is not feasible anymore,” she says.

“There are about 22,000 genes that could be potentially tested in whole exome sequencing; that is what you are going for. There is no way the College could provide proficiency testing for all of these genes. Yet you want to make sure that laboratories that do larger-scope testing can identify and name mutations correctly and interpret whether they are pathogenic or benign” (Schrijver I, et al. J Mol Diagn. 2014;16:283–287).

The Sanger sequencing PT program has the advantage of covering testing for rare diseases. These are conditions for which there are no interlaboratory exchange partners. When an analyte-specific test is available, Dr. Schrijver says, the laboratory has to subscribe to it also. “That’s because gene-specific expertise can be measured and you can address more detailed interpretation of individual mutations, which is different from methods-based testing.”

In the sequencing educational challenge (SEC), a dry test for Sanger sequencing, “the laboratory knows which gene is being sent, which is not necessarily a gene they test for in their lab,” Dr. Schrijver says. “CAP sends three challenge files with sequences and three normal ones and asks the laboratory to interpret the data.” Challenges require the laboratory to address zygosity and use correct Human Genome Variation Society (HGVS) nomenclature, to identify all variants and say whether they would change the protein, and to provide a basic interpretation, adding up to four questions per abnormal file, for a total of 12 answers. “A passing grade is 10 of 12 correct,” Dr. Schrijver explains. Of the 67 U.S. participants, 98.3 percent had acceptable performance, compared with 88.9 percent for the 50 international participants (Richards CS, et al. Genet Med. 2014;16:25–32). “These data provide a high level of confidence that most U.S. laboratories offering rare disease testing are providing consistent and reliable clinical interpretations,” the authors concluded.

Areas in which there is room for improvement include correctly naming predicted proteins for frameshift mutations, following HGVS nomenclature rules, and following guidelines for interpretation of pathogenicity.

A wet challenge became available in 2013 for Sanger sequencing. Laboratories receive three DNA samples plus primers and are asked to generate a sequence and interpret it.

Dr. Schrijver foresees more methods-based testing in the near future, such as for multiplex ligation-dependent probe amplification and chromosomal microarrays.

“Now it is time to apply this approach to next-generation sequencing,” she says. About two-thirds of participants in the molecular genetics Surveys expected to introduce NGS technology in the near future. “We realized that proficiency testing for next-generation sequencing is a logical and important next step to ensure correct variant identification,” Dr. Schrijver says. But applying proficiency testing to NGS is an entirely different level of complexity. For whole genome sequencing, she notes, the number of variants per person is about 3 million; for whole exome sequencing it is between 15,000 and 20,000.

The genomic DNA used in the NGS pilot PT program, Dr. Voelkerding says, was sourced from an individual who gave extensive informed consent. “This genomic DNA was subjected to exome and whole genome sequencing,” he says, “followed by bioinformatics analysis to derive a consensus set of sequence variants and wild-type reference positions.”

In the pilot, genomic DNA was sent to eight laboratories. They were asked to query up to 200 genomic loci consisting of a mix of single nucleotide variants, indels, and wild-type nucleotides. Some laboratories did targeted gene panels; others did exome sequencing. Three laboratories had 100 percent correct identification of the genomic loci they assessed. Three others were between 91 percent and 97 percent. For the two laboratories with lower percentages, Dr. Voelkerding says, “We think there were typographical errors on the part of the reporting laboratory. That allowed us to revisit how to structure the test result form to minimize incorrect answers due to typographical errors.”

With results from the pilot program considered satisfactory, an educational Survey will begin in March. “Operationally, that means that proficiency testing will be administered twice in 2015 and results will be returned to laboratories so they have an understanding of how they performed, but they will not be officially graded,” Dr. Voelkerding says. “If proficiency testing as designed appears appropriate with acceptable results, I would anticipate in 2016 we would move to formal proficiency testing, where laboratories that sign up and participate would not only receive results but would receive a grade in relationship to all laboratories.”

As an indication of how quickly laboratories are adopting NGS, and how urgently PT for NGS is needed, Dr. Voelkerding says about 130 laboratories recently indicated the activity code for NGS on their test menu update submitted to the CAP. “What is interesting about that number,” he says, “is that a couple of years ago it was more like 25. So the number of laboratories listing the activity code for NGS has gone up fivefold over a couple of years.”

Some laboratories are using NGS for germline disorders or inherited disorders and others to detect somatic variants and mutations in cancer biopsies. “Because there are differences between those two applications,” Dr.Voelkerding says, “new NGS proficiency testing for somatic mutations is also being developed, and some of the available PT challenges developed by the CAP Molecular Oncology Committee can be used for NGS PT assessment for specific somatic mutations.”

Dr. Weck calls the design of the proficiency test for germline mutations “pretty slick.” Laboratories report variants for particular variant positions and genomic coordinates that are given to them. “And they only report for genes for which they are doing clinical testing,” she explains. “So we can offer a broad Survey but allow laboratories to target their responses to those genes for which they have expertise.” It will be a challenge to the CAP to analyze and collate all the results, she adds. “It will be very informative in the educational phase to see how much data CAP gets back and how easy it will be to give results to participating laboratories.”

Developing proficiency testing that is robust and accurately assesses laboratories’ proficiency takes time, Dr. Voelkerding notes. It will be about three years from conception of the pilot to the educational phase to launching a formal graded PT. “Within the College we are discussing how to meet the demand to create a robust proficiency testing process as expeditiously as possible.” Much has been learned in developing the first NGS PT, he says. “Subsequent Surveys should be accelerated just by virtue of our having undergone a learning curve already.”

“What I envision,” he continues, “is essentially a portfolio of proficiency testing that will encompass traditional applications of Sanger sequencing as well as new approaches and applications of next-generation sequencing in areas of inherited disorders and molecular oncology, both solid tumors and hematologic malignancies.” Next-generation sequencing is now in the developmental phase for infectious diseases and for tissue typing with HLA markers. “Sanger sequencing is employed for tissue typing, but there are technological and logistic and cost-efficiency gains that can be made by introducing next-generation sequencing into HLA testing,” he says.

Proficiency testing addressing all those applications of NGS has to be developed. “Our overarching challenge,” Dr. Voelkerding says, “is sourcing appropriate materials for proficiency testing—the specific samples that will be used—and designing Surveys to accurately gauge the ability of laboratories.” The job ahead is big indeed. But with labs using the traditional methods for molecular genetics testing having set a high performance bar, the motivation is strong to ensure the same for next-gen sequencing.
[hr]

William Check is a writer in Ft. Lauderdale, Fla.