github.com/broadinstitute/gatk-sv/GatherSampleEvidence

Abstract

GATK-SV A structural variation discovery pipeline for Illumina short-read whole-genome sequencing (WGS) data. Table of Contents Requirements Citation Acknowledgements Quickstart Pipeline Overview Cohort mode Single-sample mode gCNV model Generating a reference panel Module Descriptions GatherSampleEvidence - Raw callers and evidence collection EvidenceQC - Batch QC TrainGCNV - gCNV model creation GatherBatchEvidence - Batch evidence merging, BAF generation, and depth callers ClusterBatch - Site clustering GenerateBatchMetrics - Site metrics FilterBatch - Filtering MergeBatchSites - Cross-batch site merging GenotypeBatch - Genotyping RegenotypeCNVs - Genotype refinement (optional) MakeCohortVcf - Cross-batch integration, complex event resolution, and VCF cleanup Module 07 - Downstream Filtering AnnotateVcf - Annotation Module 09 - QC and Visualization Additional modules - Mosaic and de novo CI/CD Troubleshooting Requirements Deployment and execution: A Google Cloud account. A workflow execution system supporting the Workflow Description Language (WDL), either: Cromwell (v36 or higher). A dedicated server is highly recommended. or Terra (note preconfigured GATK-SV workflows are not yet available for this platform) Recommended: MELT. Due to licensing restrictions, we cannot provide a public docker image or reference panel VCFs for this algorithm. Recommended: cromshell for interacting with a dedicated Cromwell server. Recommended: WOMtool for validating WDL/json files. Alternative backends Because GATK-SV has been tested only on the Google Cloud Platform (GCP), we are unable to provide specific guidance or support for other execution platforms including HPC clusters and AWS. Contributions from the community to improve portability between backends will be considered on a case-by-case-basis. We ask contributors to please adhere to the following guidelines when submitting issues and pull requests: Code changes must be functionally equivalent on GCP backends, i.e. not result in changed output Increases to cost and runtime on GCP backends should be minimal Avoid adding new inputs and tasks to workflows. Simpler changes are more likely to be approved, e.g. small in-line changes to scripts or WDL task command sections Avoid introducing new code paths, e.g. conditional statements Additional backend-specific scripts, workflows, tests, and Dockerfiles will not be approved Changes to Dockerfiles may require extensive testing before approval We still encourage members of the community to adapt GATK-SV for non-GCP backends and share code on forked repositories. Here are a some considerations: Refer to Cromwell's documentation for configuration instructions. The handling and ordering of glob commands may differ between platforms. Shell commands that are potentially destructive to input files (e.g. rm, mv, tabix) can cause unexpected behavior on shared filesystems. Enabling copy localization may help to more closely replicate the behavior on GCP. For clusters that do not support Docker, Singularity is an alternative. See Cromwell documentation on Singularity. The GATK-SV pipeline takes advantage of the massive parallelization possible in the cloud. Local backends may not have the resources to execute all of the workflows. Workflows that use fewer resources or that are less parallelized may be more successful. For instance, some users have been able to run GatherSampleEvidence on a SLURM cluster. Data: Illumina short-read whole-genome CRAMs or BAMs, aligned to hg38 with bwa-mem. BAMs must also be indexed. Family structure definitions file in PED format. PED file format The PED file format is described here. Note that GATK-SV imposes additional requirements: The file must be tab-delimited. The sex column must only contain 0, 1, or 2: 1=Male, 2=Female, 0=Other/Unknown. Sex chromosome aneuploidies (detected in EvidenceQC) should be entered as sex = 0. All family, individual, and parental IDs must conform to the sample ID requirements. Missing parental IDs should be entered as 0. Header lines are allowed if they begin with a # character. To validate the PED file, you may use src/sv-pipeline/scripts/validate_ped.py -p pedigree.ped -s samples.list. Sample Exclusion We recommend filtering out samples with a high percentage of improperly paired reads (>10% or an outlier for your data) as technical outliers prior to running GatherSampleEvidence. A high percentage of improperly paired reads may indicate issues with library prep, degradation, or contamination. Artifactual improperly paired reads could cause incorrect SV calls, and these samples have been observed to have longer runtimes and higher compute costs for GatherSampleEvidence. Sample ID requirements: Sample IDs must: Be unique within the cohort Contain only alphanumeric characters and underscores (no dashes, whitespace, or special characters) Sample IDs should not: Contain only numeric characters Be a substring of another sample ID in the same cohort Contain any of the following substrings: chr, name, DEL, DUP, CPX, CHROM The same requirements apply to family IDs in the PED file, as well as batch IDs and the cohort ID provided as workflow inputs. Sample IDs are provided to GatherSampleEvidence directly and need not match sample names from the BAM/CRAM headers. GetSampleID.wdl can be used to fetch BAM sample IDs and also generates a set of alternate IDs that are considered safe for this pipeline; alternatively, this script transforms a list of sample IDs to fit these requirements. Currently, sample IDs can be replaced again in GatherBatchEvidence. The following inputs will need to be updated with the transformed sample IDs: Sample ID list for GatherSampleEvidence or GatherBatchEvidence PED file Citation Please cite the following publication: Collins, Brand, et al. 2020. "A structural variation reference for medical and population genetics." Nature 581, 444-451. Additional references: Werling et al. 2018. "An analytical framework for whole-genome sequence association studies and its implications for autism spectrum disorder." Nature genetics 50.5, 727-736. Acknowledgements The following resources were produced using data from the All of Us Research Program and have been approved by the Program for public dissemination: Genotype filtering model: "aou_recalibrate_gq_model_file" in "inputs/values/resources_hg38.json" The All of Us Research Program is supported by the National Institutes of Health, Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026553; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA #: AOD 16037; Federally Qualified Health Centers: HHSN 263201600085U; Data and Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163; Communications and Engagement: 3 OT2 OD023205; 3 OT2 OD023206; and Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276. In addition, the All of Us Research Program would not be possible without the partnership of its participants. Quickstart WDLs There are two scripts for running the full pipeline: wdl/GATKSVPipelineBatch.wdl: Runs GATK-SV on a batch of samples. wdl/GATKSVPipelineSingleSample.wdl: Runs GATK-SV on a single sample, given a reference panel Building inputs Example workflow inputs can be found in /inputs. Build using scripts/inputs/build_default_inputs.sh, which generates input jsons in /inputs/build. Except the MELT docker image, all required resources are available in public Google buckets. Some workflows require a Google Cloud Project ID to be defined in a cloud environment parameter group. Workspace builds require a Terra billing project ID as well. An example is provided at /inputs/values/google_cloud.json but should not be used, as modifying this file will cause tracked changes in the repository. Instead, create a copy in the same directory with the format google_cloud.my_project.json and modify as necessary. Note that these inputs are required only when certain data are located in requester pays buckets. If this does not apply, users may use placeholder values for the cloud configuration and simply delete the inputs manually. MELT Important: The example input files contain MELT inputs that are NOT public (see Requirements). These include: GATKSVPipelineSingleSample.melt_docker and GATKSVPipelineBatch.melt_docker - MELT docker URI (see Docker readme) GATKSVPipelineSingleSample.ref_std_melt_vcfs - Standardized MELT VCFs (GatherBatchEvidence) The input values are provided only as an example and are not publicly accessible. In order to include MELT, these values must be provided by the user. MELT can be disabled by deleting these inputs and setting GATKSVPipelineBatch.use_melt to false. Execution We recommend running the pipeline on a dedicated Cromwell server with a cromshell client. A batch run can be started with the following commands: > mkdir gatksv_run && cd gatksv_run > mkdir wdl && cd wdl > cp $GATK_SV_ROOT/wdl/*.wdl . > zip dep.zip *.wdl > cd .. > echo '{ "google_project_id": "my-google-project-id", "terra_billing_project_id": "my-terra-billing-project" }' > inputs/values/google_cloud.my_project.json > bash scripts/inputs/build_default_inputs.sh -d $GATK_SV_ROOT -c google_cloud.my_project > cp $GATK_SV_ROOT/inputs/build/ref_panel_1kg/test/GATKSVPipelineBatch/GATKSVPipelineBatch.json GATKSVPipelineBatch.my_run.json > cromshell submit wdl/GATKSVPipelineBatch.wdl GATKSVPipelineBatch.my_run.json cromwell_config.json wdl/dep.zip where cromwell_config.json is a Cromwell workflow options file. Note users will need to re-populate batch/sample-specific parameters (e.g. BAMs and sample IDs). Pipeline Overview The pipeline consists of a series of modules that