Nicholas R. Wheeler1,2, Penelope Benchek1,2, Brian W. Kunkle3, Kara L. Hamilton-Nelson3, Mike Warfe1,4, Jeremy R. Fondran1,4, Jonathan L. Haines1,2, William S. Bush1,2
1Cleveland Institute for Computational Biology, Case Western Reserve University
2Department of Population and Quantitative Health Sciences, Case Western Reserve University
3John P. Hussman Institute for Human Genomics, Miller School of Medicine, University of Miami
4Center for Advanced Research Computing, University Technology, Case Western Reserve University
Email: nrw16@case.edu, phb16@case.edu, bkunkle@med.miami.edu, khamil@med.miami.edu, jmw22@case.edu, jrf16@case.edu, jlh213@case.edu, wsb36@case.edu
Pacific Symposium on Biocomputing 25:523-534(2020)
© 2020 World Scientific
Open Access chapter published by World Scientific Publishing Company and distributed under the terms of the Creative Commons Attribution (CC BY) 4.0 License.
Modern genomic studies are rapidly growing in scale, and the analytical approaches used to analyze genomic data are increasing in complexity. Genomic data management poses logistic and computational challenges, and analyses are increasingly reliant on genomic annotation resources that create their own data management and versioning issues. As a result, genomic datasets are increasingly handled in ways that limit the rigor and reproducibility of many analyses. In this work, we examine the use of the Spark infrastructure for the management, access, and analysis of genomic data in comparison to traditional genomic workflows on typical cluster environments. We validate the framework by reproducing previously published results from the Alzheimer's Disease Sequencing Project. Using the framework and analyses designed using Jupyter notebooks, Spark provides improved workflows, reduces user-driven data partitioning, and enhances the portability and reproducibility of distributed analyses required for large-scale genomic studies.