Good Laboratory Standards for Clinical Next-Generation Sequencing Cancer Panel Tests
Article information
Abstract
Next-generation sequencing (NGS) has recently emerged as an essential component of personalized cancer medicine due to its high throughput and low per-base cost. However, no sufficient guidelines for implementing NGS as a clinical molecular pathology test are established in Korea. To ensure clinical grade quality without inhibiting adoption of NGS, a taskforce team assembled by the Korean Society of Pathologists developed laboratory guidelines for NGS cancer panel testing procedures and requirements for clinical implementation of NGS. This consensus standard proposal consists of two parts: laboratory guidelines and requirements for clinical NGS laboratories. The laboratory guidelines part addressed several important issues across multistep NGS cancer panel tests including choice of gene panel and platform, sample handling, nucleic acid management, sample identity tracking, library preparation, sequencing, analysis and reporting. Requirements for clinical NGS tests were summarized in terms of documentation, validation, quality management, and other required written policies. Together with appropriate pathologist training and international laboratory standards, these laboratory standards would help molecular pathology laboratories to successfully implement NGS cancer panel tests in clinic. In this way, the oncology community would be able to help patients to benefit more from personalized cancer medicine.
By virtue of completion of the human genome project, a set of technologies that are referred to as next-generation sequencing (NGS) has been developed [1,2]. Being able to sequence millions of short fragments of DNA simultaneously, NGS technologies have achieved unprecedented high-throughput and low perbase cost. The combination of its powerful analytic performances and the rapidly dropping cost of NGS rapidly brought NGS to clinic. However, the amount and complexity of sequencing data, especially for cancer genome sequencing, raised concerns about the reliability of NGS cancer panels. Regarding this issue, several working groups, including the College of American Pathologists (CAP) or the American College of Medical Genetics and Genomics (ACMG), issued guidelines on NGS clinical tests [3,4]. Yet, there is no reliable guideline for NGS cancer panel tests that accommodates the regulatory environment in Korea. Considering that NGS-based testing is rapidly evolving and improving, the Molecular Pathology Study Group aimed to establish a necessary regulatory framework for clinical NGS tests without inhibiting further upgrade of NGS-based technologies.
Unlike traditional single gene-based sequencing tests, the NGS cancer panel involves a complex two-step process: (1) wet bench process and (2) bioinformatics analysis of sequence data. The wet bench process includes the handling of patient samples, extraction of nucleic acids, fragmentation and barcoding, target enrichment, adaptor ligation, library preparation, and generation of sequence reads. Bioinformatics analysis includes mapping sequence reads to the human reference genome, variant calling, annotation, and reviewing data in the right clinical context. We regarded these two steps as distinct processes requiring separate standards because some laboratories might use external facilities to perform either portion of the NGS cancer panel tests. This report aims to provide a general guidance for developing more detailed NGS checklists, that will be done by a separate working group involving regulatory agencies in the near future.
LABORATORY GUIDELINES FOR NEXT-GENERATION SEQUENCING CANCER PANEL TESTS
Choice of genes to be tested and test platforms
Traditionally, only a limited number of clinically validated predictive or prognostic biomarkers have been approved as clinical tests. However, accumulating genomic data obtained from multiple tumor types and expansion of genome-based clinical trials suggest that many of the future anti-cancer treatments and clinical trials will require comprehensive diagnostic panels that allow the detection of multiple mutations at the same time. Such tests might involve either probe-based capture or primer-based amplification for the enrichment of genomic regions to be tested.
The number and scope of genes to be tested depend on the purpose of the test. If the purpose is limited to companion diagnostics for current standard care, the number of genes would be very limited. However, if there is a need for clinical trials for which NGS-based tests are required to stratify patients, a broader range of genes should be interrogated. In this context, the currently used cancer panel tests are mainly focused on clinically actionable genomic alterations at selected protein-coding regions that are defined by the availability of approved drugs and pathognomonic molecular features. However, there is an increasing need for adding genomic alterations associated with resistance to molecularly targeted therapies or predicted response to investigational drugs. Furthermore, genomic data could be used to refine pathological diagnosis. For example, mutations in IDH1 or IDH2 and co-deletion of chromosome 1p and 19q are already incorporated in recently revised pathological diagnosis of diffuse glioma [5].
There are several NGS platforms such as Illumina, Ion Torrent, and Roche 454, and all of which have been reported to be as reliable as the current standard genotyping tools at least for the important cancer genes provided that laboratories perform adequate quality control [6-8]. Thus, pathology laboratories may choose their preferred platforms based on their individual requirements, such as expected sample status (i.e., small biopsies, resected tissue samples, or liquid biopsies, etc.), expected number of samples, and/or types of variants to be analyzed (e.g., copynumber analysis might be limited in platforms that use primer-based amplification during target enrichment). It is important for laboratories to be aware of platform characteristics and perform adequate quality controls depending on the platform characteristics.
Specimen handling
Sample transportation, receipt, and storage
Adequate processing of tissue samples is essential in a reliable NGS cancer panel test. Required specimen handling procedures are nearly the same as those required for traditional single-gene tests. Briefly, the quality and the amount of neutral buffered formalin relative to the size of the specimen should be monitored. The time interval from specimen acquisition to fixation should be minimized, and optimal fixation duration should be monitored [9]. The optimal fixation duration depends on the dimension of each sample because formalin penetrates tissue at a rate of approximately 1 mm per hour. The rule of thumb for recommended fixation duration for surgically resected specimen is 24 hours [10].
Morphological assessment and designation of ideal tumor area
Correct pathologic diagnosis and assessment of the fraction of neoplastic cell nuclei in tumor area by pathologists are crucial for the interpretation of NGS results. Once the diagnosis and tumor purity are established, it is strongly recommended that the pathologists circle an area from which nucleic acid is extracted so that neoplastic cells are enriched to the appropriate level of tumor purity. The minimum requirement of tumor purity can be established in the initial validation of test performances (see “Validation” section). The estimated tumor purity of a selected area should be documented. When tumor purity is assessed, the pathologists should be aware of the fact that the relative number of tumor cell nuclei reflects tumor purity rather than occupied tumor area. Thus, an area with heavy inflammatory cell infiltration should be avoided (Fig. 1A, B). Also, to ensure DNA quality, pathologists should try to avoid necrotic areas (Fig. 1C) and areas with extensive extracellular mucin (Fig. 1D).
Nucleic acid extraction, quantification, and storage
For clinical tests, DNA extraction kits should have a high level of performance specification to obtain DNA of sufficient quality and quantity for intended NGS tests from formalin-fixed paraffin embedded (FFPE) samples. In addition, DNA extraction procedures should have appropriate mechanisms by which sample contamination or misidentification could be avoided. Several commercially available kits that use silica-based or magnetic bead–based extraction protocols meet such requirements.
A major cause of sequence artifacts is deamination of cytosine resulting in C to T transitions during amplification [11-14]. Formalin fixation and longer storage period contribute to this process [15]. Since those sequence artifacts are usually present at a very low frequency, such artifacts are unlikely to affect test results if sufficient amounts of unique DNA molecules are available. However, if the DNA input amount is too small or if the purpose is to detect variants with low allelic fraction, these artifacts would be a problem. In addition, amplicon-based methods are more susceptible to these artifacts than hybridization capture-based methods [11]. Thus, laboratories may choose DNA extraction protocols with enzymatic removal of uracil-containing templates when they interrogate FFPE samples with highly fragmented DNA or with low tumor purity by amplicon-based methods. In the case of decalcified specimens, DNA quality or quantity is inferior to those of the specimens without decalcification. Regarding this, it has been reported that a decalcification protocol involving ethylenediaminetetraacetic acid might be better than that involving hydrochloric acid in terms of usable nucleic acid for subsequent NGS analysis [16].
Quantitation of extracted nucleic acids can be done by Nano-Drop, Qubit, or the Picogreen method. Among them, Nano-Drop is not recommended because it also detects nucleic acids that are not suitable for downstream analyses. Nucleic acids should be stored under highly controlled conditions in order to maintain sample identity and integrity. Extracted DNA is to be stored at –20˚C and RNA at –80˚C. Sequencing libraries and polymerase chain reaction (PCR) products may be stored in –20˚C but should be separated from pre-amplification materials to prevent them from contaminating pre-amplification materials.
Sample identity tracking
Like other single gene–based tests, verification of sample identity is the most basic and important aspect in clinical NGS test. The NGS cancer panel tests involve many steps, making them inherently subject to sample mix-up or swapping. Thus, test procedures should have an appropriate system to minimize such critical events. For example, panels could be designed to include a number of single nucleotide polymorphism (SNP) markers that allow molecular barcoding of patient samples [17] so that sample identity can be traced. In addition, an electronic laboratory information management system could be useful for this purpose. With any tool in place, it is important to enable end-to-end sample tracking in clinical NGS cancer panel tests.
Library preparation
Library preparation is the step where extracted nucleic acids are prepared for the sequencing reaction. It involves DNA fragmentation, adaptor ligation, and enrichment of the target of interest. Target enrichment can be done either by an amplicon-based approach or by hybridization-based capture. Molecular barcodes are usually introduced to enable sample identification and pooling of multiple samples in a single flow cell lane. To avoid contamination, all steps before amplification should be done in a separate space. In addition, extra-caution should be in place during sample transportation from pre-PCR area to post-PCR area. After library preparation, appropriate quality controls should be applied to determine whether the rest of the procedures should be continued. Quality controls include quantitation, fragment size analysis, and quantitative PCR using adaptor sequences for priming.
Data analysis of sequence reads
Bioinformatics pipelines used for the analysis of NGS data consists of multiple steps, such as de-multiplexing, read alignment, de-duplication, base calibration, variant calling, filtering, and annotation. Currently, no single “gold-standard” algorithm exists. Therefore, laboratories should choose the most suitable algorithm for the types of variants to be reported and optimize them. In the absence of a “gold standard,” it is important to validate the analytic performance of the bioinformatics pipelines [18-20]. It is also important to make sure that all versions of algorithms are traceable and properly updated (see “Validation” section).
Sequencing read
The initial step after sequencing includes converting the base intensities in a sequencer to digital-level nucleotide sequences, called FASTQ. Although the types of signals differ among various sequencing platforms, most have their internal software for translating base calling into the compressed and de-multiplexed FASTQ files. The nucleotides in FASTQ have corresponding base quality scores that are in a form of logarithmic scales indicating the probabilistic confidence level of the bases. Appropriate quality control should be done to confirm the general integrity of the sequencing data, such as the total number of bases, sequence contents (including GC contents), per base sequence quality, etc. Read trimming is recommended if the base quality or composition does not meet the quality control threshold that each laboratory has empirically set up or that the sequencer manufacturer has recommended.
Alignment
This step is to find where the short read sequences are located. In general, single or pair-end reads are first aligned (mapped) to a human genome reference. It is essential to include the version of the human reference sequence in the clinical report. Since poorly mapped sequences may lead to compromise the reliability of called variants, especially in solid tumors with low tumor content, sequencing reads should be filtered based on mapping quality score so that only confidently mapped reads are processed further. After the initial mapping, read duplicates should be removed, because unwanted clonal amplification of reads with sequence artifacts may lead to false-positive variant calls. Alignment is often challenging when the sequence reads come from genomic regions having large insertions or deletions, repetitive regions, pseudogenes, or homologous genes because there can be several other similar-looking genomic regions across the reference genome. Realignment using known references that have the suspected insertions or deletions may be necessary in such cases.
Variant calling: single nucleotide variation, insertion/deletion, copy-number variation, and translocation
Somatic variants can be identified by subtracting variants found in non-neoplastic cells from those found in cancer. If a laboratory chooses not to sequence the corresponding normal samples, it should be noted that some germline variations might exist in the result. Laboratories should consider the implementation of modular analysis pipelines, in which different algorithms or settings are used to call different types of variants: single nucleotide variations (SNVs), insertion/deletion (Indels), copy-number variations (CNVs), and translocations.
SNV/Indel
The quality of variant calls is strongly related to the quality of alignment. The key challenge of variant calling is to distinguish real variants from sequencing errors. In general, the more times the variant is sequenced, the more reliable the variant call is. The minimum depth of coverage depends on the required sensitivity of the intended assay, the sequencing platform and the types of mutations to be detected. Although sensitivity is increased in proportion to sequencing depth, false-positive calls may also be increased especially in cases with low tumor content. There are various reasons for false-positive variant calls; they often result from PCR errors, sequencing errors, mis-mapped reads on repetitive sequences or homologous regions and so forth. Because each variant calling algorithm uses different strategies to filter out false calls, different algorithms sometimes generate discrepant results. Thus, laboratories should find optimal parameter settings during assay development and validation to minimize algorithm-dependent result variability.
Reliable identification of Indels is particularly challenging because sequence reads containing this type of variants are not often accurately mapped. Thus, the sensitivity and specificity for this type of variants is often reduced. With regard to this, laboratories should identify clinically important Indels and validate assay performances to establish reasonable sensitivity and positive predictive value for the identification of those Indels. Due to the high rate of false-positives, manual review by using visualization tools and comparison with the same regions in other samples on the same run is recommended for all Indel calls.
Copy-number variation
The reliable identification of CNV is quite difficult in NGS cancer panel sequencing because of the uneven target coverage related to hybridization capturing steps, the absence of matched normal data, or the lack of coverage uniformity [21]. Although algorithms for detecting CNV in targeted NGS tests are improving [22], the inherent limitations in cancer panel sequencing of clinical FFPE samples require robust validation of test performance. This type of validation can be done by testing characterized cancer cell lines or clinical samples with known CNV profiles (see “Validation” section) [23], although there is no agreement upon the minimal number of samples for appropriate validation.
Translocation
Translocation can be identified based on the DNA level, using discordant or split sequencing reads. However, the inherent limitations of short-reads in terms of alignment can result in many false positive calls. Therefore, the test performance should be appropriately validated using reference materials with known translocations across targeted genomic regions that have been confirmed by the current gold standard. Furthermore, it is highly recommended that all translocation calls be manually reviewed by using genome visualization tools such as the Integrative Genomics Viewer. Finally, it should be noted that a translocation can be missed if the breakpoint of fusion (may be somewhere in the introns not covered by the panel) is not included in targeted genomic regions, even though protein-coding regions of the translocation partners are included in the panel.
Variant annotation and filtering
Variant annotation determines if a sequence variant is real and provides predicted resulting amino acid changes. To identify true somatic variants, false variants should be properly filtered. Important sources of false variants include cytosine deamination, amplification errors, and sequencing errors. Cytosine deaminations are introduced ex vivo; these variants are not copied to the opposite strand, meaning that the artifacts are only present on one strand. To facilitate the detection of cytosine deamination artifacts, laboratories may use techniques such as molecular inversion probe and HaloPlex and Duplex sequencing, to enrich and sequence both the sense and antisense DNA strands [24]. Amplification errors can be introduced due to DNA polymerase errors during amplification steps of library enrichment. These errors might be minimized by the application of unique barcodes to individual DNA molecules during library enrichment. If the same variant is detected in multiple unique molecules, the variant might be real because it is unlikely that individual molecules acquire the same polymerase error during amplification. Sequencing errors are highly dependent on the sequencing platform and sequencing chemistry. The ability to call SNVs and Indels is known to be similarly accurate for data generated on the PGM and Illumina platform, provided that there is sufficient coverage [24].
Interpretation of computational output
Depending on the types of variants (e.g., missense, nonsense, etc.) and the types of genes (e.g., hotspots in oncogenes vs randomly distributed mutations in tumor suppressor genes), interpretation of detected variants can be simple or quite complicated. Basically, the ACMG strongly recommends that interpretation is performed by trained staffs such as clinical molecular geneticists or molecular pathologists [4]. A multi-disciplinary sequencing data analysis team with various scientific backgrounds including clinical oncology, genomics, bioinformatics, and pathology, is recommended for accurate interpretation.
For variants that are not hotspot mutations, germline variants should be excluded first. Since most cancer panel tests do not analyze matched non-neoplastic tissue, laboratories should prepare mechanisms to filter out potential germline variants based on the genetic polymorphism data on the population to which the tested patient belongs. In most instances, laboratories use public databases on germline polymorphism such as the 1000 Genomes Project, ExAC, or dbSNP (Table 1), but ideally, these data should be derived from the same ethnic group as the tested patient.
Since the clinical and biological significance of cancer-related genomic variants are increasingly characterized, many variants detected in most cancer panel tests have related information in public databases (see “Reporting” section for details). However, previously uncharacterized variants may also be detected considering the characteristics of NGS tests. Potential biological significance, or pathogenicity, could be inferred from the archived genotype-phenotype correlation data such as ClinVar, Human Gene Mutation Database, and Leiden Open Variation Database (Table 1). In addition, the in silico prediction of functional impacts is available in dbNSFP or Ensembl Variant Effector (Table 1). Since no single database is perfect, it is essential to refer to multiple resources for appropriate interpretation. Finally, knowledge about variants will be continuously improved with the accumulation of each lab’s experiences and feedbacks from clinicians.
Reporting
General format
Reporting of NGS cancer panel test results should follow the general professional organizations’ recommendations and guidelines [25]. There are two major essential parts of a report: proper patient identification and detected actionable variants. The patient identification part is the same as that in the current standard single gene-based tests, but the detected variant part is more complex and sophisticated in clinical NGS cancer panel tests because typical NGS cancer panel tests detect multiple variants at the same time. The clinical NGS report should include the essential information listed in Table 2, and the most pertinent information, such as actionable variants and a critical summary of those variants, should be placed in a clearly visible section on the first page. Detected sequence variants should be annotated in concordance with the Human Genome Variation Society mutation nomenclature and the version of the human reference sequence to which sequence reads were aligned should also be included in the report. Variants, which are not clinically actionable but are potentially useful for future practice, might follow the actionable variants. Then, detailed information about the detected variants (see the “Interpretation” section) and essential technical information, such as genes or genomic regions included in the panel and key quality control metrics, may be listed. Inclusion of granular details of technical information is not recommended, but a description of how clinicians can obtain the details may be included [26].
Presentation of detected variants and clinical translation
Since typical cancer panel results include several cancer-related genomic variants with different levels of clinical or biological evidences, variants should be classified and reported according to the level of evidences. Several leading institutions use their own variant classification system and each institution may discuss with clinicians about which variant classification system should be used. However, a recently published related guideline is a good example [27]: (1) Tier 1, strong clinical significance such as Food and Drug Administration (FDA)–approved therapies, professional guidelines, prognostic and diagnostic markers based on well-powered studies with expert consensus; (2) Tier 2, potential clinical significance such as FDA-approved therapies for different tumor types, investigational therapies, prognostic and diagnostic markers based on small published studies with some consensus and preclinical trials or case reports without consensus; (3) Tier 3, unknown clinical significance; and (4) Tier 4, benign or likely benign. Tier 1 to 3 variants should be included in the clinical NGS report in a decreasing order of clinical significance and it is not recommended to include Tier 4 variants in clinical NGS report.
Interpretation of detected variants in terms of their clinical impact and pathogenicity is a daunting task. As mentioned in the “Interpretation of computational output” section, many information sources such as public databases, published guidelines, and computational prediction algorithms should be integrated for proper interpretation. In addition to the previously mentioned public databases on germline polymorphism, in silico prediction of the functional impacts, genotype-phenotype relationship, and several cancer-specific mutation databases are available online (Table 1). These resources are very useful in the interpretation of variants in the context of cancer, but it is important to make sure that the database is properly curated, referenced and updated in a regular basis. Regarding this, FDA guidance suggested that appropriate databases should implement decision matrices with published details of each variant’s interpretation and have documented standard operating procedures (SOPs) for the curation and update of this information [28].
When assigning variants to each tier, it is essential that they should be interpreted in the context of both variant details (e.g., BRAF V600E) and patient tumor type (e.g., malignant melanoma vs colorectal cancer). This is because different variants in a single gene may have different clinical impacts and the clinical significance of a particular variant may vary depending on the tumor type. For tier 1 variants, a clear statement on their clinical relevance and any appropriate action should be included. Also, it might be helpful to clarify the absence of other variants that are important in the patient tumor type. For tier 2 variants, advising the ordering physician about available clinical trials may be useful. In addition, tier information is subject to change as new evidences and therapeutic options are continuously emerging.
In cases where only limited interpretation can be made and full quality control standards could not be met, it is essential that pathologists make a professional judgment on whether the result should be reported. Also, any limitation of the analysis should be clarified in the report. For example, negative results in samples with low neoplastic cell content at near or below the established sample rejection criteria should be accompanied by a note on potential false negatives. Furthermore, clinical NGS laboratories should have clearly defined protocols about when additional confirmatory tests should be advised, as well as performance validation data in cases where those confirmatory tests are not necessary.
REQUIREMENTS FOR CLINICAL NEXT-GENERATION SEQUENCING LABORATORIES
Wet Bench Analytic Process
Documentation
The detailed documentation of SOPs is critical for quality assurance of a complex, multi-step wet bench process. All SOPs of each step of the wet bench process must be documented so that each step can be traced. This includes documentation of all methods, reagents, instruments and controls (if applicable). Most of the documentations should be similar to those of current standard single gene testing, but those specific to NGS testing include detailed information regarding captured regions, such as genomic coordinates of captured probes and lists of genes and target enrichment protocols. Clinical laboratories that process different types of samples, such as FFPE samples or blood, should establish SOPs for each validated sample type. Metrics for quality control to assess run status must also be documented. Examples include mean target coverage, percentage of reads that map to target regions, and the fraction of bases meeting specified quality and coverage thresholds. Laboratories must define and document acceptance or rejection criteria for each step of the wet bench process, such as DNA extraction, library preparation, and sequencing.
Validation
Before testing patient samples, clinical NGS laboratories must establish the analytical validity of the intended tests. If the intended test is approved by the Ministry of Food and Drug Safety (MFDS), laboratories can verify the performance specifications established by the manufacturer. If the intended test is not approved by the MFDS, i.e., a laboratory-developed test, laboratories must establish performance characteristics such as accuracy, precision, sensitivity, specificity, and reportable range.
Because NGS cancer panel tests involve complex, multistep processes, each step needs to be empirically optimized to determine optimal assay conditions. Once those optimizations are done, an analytic validation should be performed for a whole test in a “beginning to end” fashion. Test performance should be separately validated for different types of samples, such as FFPE or blood.
For “beginning to end” validation, a number of samples should be analyzed to assess the test performance. There is no general agreement on how many samples are required for this type of validation. With regard to this issue, the CAP concluded that adding a minimum number of samples for validation is premature given the ongoing evolution of NGS technologies and the diversity of diagnostic applications [3]. Literature review revealed that the sample number for validation varied with a range of 20–80 samples [29-32]. The minimum number of samples needed for an appropriate validation might be affected by the size of the test, i.e., a larger gene panel will require more extensive validation, and by the range of samples to be tested, i.e., FFPE samples with variable tumor content. Here, we suggest the general principles of validation in terms of several important analytic performance parameters.
Considering the inherent characteristics of NGS cancer panel tests, it is important to evaluate as many different genomic regions as possible because sequence context can influence sequencing results. In addition, laboratories should determine analytic performances for all variant types relevant to the intended test (e.g., SNV, Indel, CNV, and translocations). Since NGS-based tests interrogate multiple variants at the same time, the validation of test performances involves two parts: method-specific (detection of as many variants as possible in a single sample) and analyte-specific (detection of a certain genomic variant in multiple samples). Laboratories can establish the test performances by combining those two approaches.
Accuracy validation is a “method-specific” way of validation. For this type of validation, laboratories might use well-characterized cancer cell line samples to interrogate whether the intended test accurately detects all known variants within the genomic regions covered by the panel. In this case, reference cell line samples should be periodically monitored for identity and passage number to prevent a significant genetic drift. In addition, laboratories might use well-characterized HapMap samples (NA12878 and NA19240), variants of which are readily available on the web [33,34]. Details of available reference materials are summarized in Table 3. Evaluation of these reference samples might be quite useful to validate the accuracy across a wide range of genomic regions [23]. During this type of validation, an adequate depth of coverage threshold [35] necessary to accurately call all expected variants can be empirically established by plotting false-positives and false-negatives as a function of coverage.
Analytic sensitivity can be evaluated by comparing NGS test results with the current gold standard test results for known genetic variants in reference materials. To this end, laboratories may use historical controls such as accumulated clinical samples with well-characterized genomic variants by gold standard methods, provided that the gold standard tests were done in an appropriately accredited environment.
Analytic specificity is theoretically calculated by determining the fraction of test negatives (wild type sequence calls by the intended test) per true-negatives (samples that are known to have wild type sequences by the gold standard method). However, this concept often does not work well for NGS-based tests because too many potential variants are included in typical cancer panels. For most laboratories, it might be reasonable to leverage specificity by determining the average number of false-positive calls for the regions tested in a number of well-characterized clinical samples.
The limit of detection can be determined by cell line “mixing” experiments, in which a cell line with known genetic variants is serially diluted with another cell line without such variants to have different mixture ratios, which are then tested to determine whether the intended test accurately detects all specified variants under a certain coverage threshold. Alternatively, laboratories may determine the limit of detection by using the HapMap samples (NA12878 and NA19240) mentioned earlier by mixing them in various ratios and interrogating variants that differ between the two different HapMap samples.
Precision (repeatability for intra-run and reproducibility for inter-run variability) can be determined by repeating the same samples in the same run (repeatability) or by repeating the same sample in a different run (on a different day by a different operator). Ideally, testing at least three samples is recommended to adequately establish precision adequately [3].
Any changes to a clinical NGS test, such as changes in instruments, specimen types, reagents, or software, require that performance specifications be re-established or be shown to be unchanged. For example, inclusion of new genes to an existing gene panel requires revalidation to make sure that new sequence variations are reliably detected without compromising the quality of the original assay. The extent of re-validation depends on the predicted influence on the test performance related to the change. For example, if only the bioinformatics pipeline is updated, it may not be necessary to re-validate all steps before data analysis.
Quality management
Clinical NGS laboratories should establish and follow a quality management plan. This plan should be integrated within the institution’s overall quality assurance program. Components of the NGS quality management program are not much different from the traditional single gene-based tests. Once laboratories establish an initial validation of test performances, laboratories must perform internal quality controls daily and external quality controls periodically. Each component of the quality assurance program specific to NGS cancer panel tests is discussed in detail below.
Internal quality assurance program
Recently, guidelines for quality control and recommendations for the use of NGS in different applications have been published [3,4,25,36-42] and are summarized in Table 4. This guidance will work as a checklist for each component of the intended NGS test and related quality control metrics that require reviewing in order to feel confident about the quality of results. It is impractical to include multiple positive controls with different variant types during each run due to unacceptable cost and time. Instead, a single characterized external control with known variants in each run may be sufficient to demonstrate that the procedure is successful [36]. Typically, this can be done by preparing a number of DNA aliquots from a large FFPE sample block whose genome has already been well-characterized genomically and include those aliquots in each run of patient samples.
External quality assurance program: proficiency testin and alternate assessment
The Clinical Laboratory Improvement Amendment (CLIA) requires participation in a proficiency testing (PT) program twice a year for most clinical tests. For any clinical tests without such PT requirements, laboratories must verify test performance twice a year [24]. Since PT programs specific to NGS cancer panel tests do not currently exist, laboratories may verify their test performance by using alternate assessment (AA), where deidentified patient samples are exchanged and re-tested between different laboratories until formal PT programs are set [24]. A Korean guideline published by the MFDS follows the same principle and requires documentation of all procedures related to PT or AA [43]. There is no general agreement upon the number of reference materials per one round of PT or AA but analysis of two samples has been recommended [36].
Policies: confirmatory testing, laboratory records, upgrades
As NGS cancer panel tests with appropriate quality controls are reported to meet a clinical grade performance [23,44-47], routine confirmatory testing is not recommended. However, laboratories should have a policy that clearly documents both any indication for confirmatory testing as well as any performance validation data upon which they decided that such confirmatory testing is not necessary. In addition, the CAP is flexible regarding the methods used for any needed confirmatory testing.
Keeping comprehensive laboratory records is essential in monitoring complex, multi-step NGS cancer panel tests. In this regards, such records should be maintained in such a way that all detailed information about test procedures including reagents, sequencing runs, wet lab, and bioinformatics procedures and responsible technicians is traceable. While all details need not be included in the clinical report, laboratories should maintain a database from which detailed information regarding the analysis of individual specimen can be obtained.
Laboratories must be prepared for upgrades to make sure that they are not using obsolete methods. A policy for the upgrade of instruments, sequencing chemistries, and reagents or kits, as well as subsequent post-upgrade validation of test performances should be in place. The policy may include specified intervals for upgrade and required validation processes, depending on the type or extent of upgrade.
Bioinformatics process
Documentation
Laboratories should document all bioinformatics processes, including all data files, variant caller’s parameters, and versions of the bioinformatics algorithms. Sources and versions of all bioinformatics algorithms should be documented and updated properly. Quality control information on bioinformatics analysis, such as the cut-off of read depth, base quality score, and mapping quality, should also be documented.
Validation and quality management
General validation principles were already discussed in the Validation proportion of the “Wet Bench Analytic Process” section. Briefly, for bioinformatics pipelines, laboratories should iteratively find parameters for optimal performance of computational algorithms before applying it to the lab process. Once the pipeline is initially validated, variations between sequencing runs should be monitored daily. Principles of the quality management were already discussed in the “Wet Bench Analytic Process” section. In short, laboratories should monitor any deviation from established performance characteristics in terms of quality metrics and analysis results. For any deviation, laboratories should document the investigational measures and corrective actions made to resolve the deviation. Essential quality metrics for bioinformatics performance verification include depth of coverage, uniformity of coverage and base call quality scores. In addition, GC bias, proportion of reads that map to nontargeted regions, and percentage of duplicated reads could also be used to monitor performances of sequencing reaction and subsequent bioinformatics analyses (Table 5).
Policies: upgrades, storage, and data management
The bioinformatics pipeline should be revalidated upon any changes in operating systems, software, or overall pipelines, which may otherwise affect its analytic performance. Since a huge amount of data files are generated from the bioinformatics pipeline, it is impractical to store all sorts of files considering the significant cost. Instead, the CAP NGS workgroup recommended that some important file formats, such as FASTQ, BAM, and VCF, should be stored for quality controls or investigational use [3]. There is no general agreement on the required storage period, but it is important for laboratories to set their own storage policies in accordance with local or national requirements (if any) and inform clients of those policies.
CONCLUSION
Although the advancement of NGS technologies unraveled the genomic landscape of human malignancies and opened an era of genome-guided anti-cancer therapies, the complexity of NGS technologies made it especially difficult to set up regulatory or professional standards for assuring analytical validity of test results. In this report, the workgroup addressed general guidelines related to the test procedures and proposed requirements for clinical NGS cancer panel tests such as test validation, quality controls, reference materials and participation in PT or AA. Due to space limitations and the clinical implementation of NGS cancer panel tests being in its infancy, granular details were not covered in this review and those details should be addressed in further publications.
As clinical experiences accumulate and technologies evolve, more refined recommendations or guidelines could be published in the near future. In addition, good clinical decision support systems or knowledge bases need to be developed to help patients in terms of benefits from personalized FDA-approved or investigational therapies and prognostic information. To realize personalized cancer medicine in the end, it is important for medical communities to share information on genotype (oncogenic variants in patients’ samples)–phenotype (drug response or prognosis) relationships, and all these efforts start with accurate genomic profiling of cancer tissue. To ensure reliability of NGS-based genomic profiling, a multi-disciplinary team approach, involving clinicians, pathologists, laboratory scientists, bioinformaticians, manufacturers, professional organizations, and government agencies, is essential.
Notes
Conflicts of Interest
No potential conflict of interest relevant to this article was reported.
Acknowledgements
The study was supported by a grant from Ministry of Food and Drug Safety, Republic of Korea, in 2016 (grant No. 16173 MFDS004; to W.-Y.P.).
The study was supported by the Korean Society of Pathologists Grant (2016).