Scalable Reference Genome Assembly from Compressed Pan-Genome Index with Spark

Altti Ilari Maarala*, Ossi Arasalo, Daniel Valenzuela, Keijo Heljanko, Veli Mäkinen

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

2 Citations (Scopus)


High-throughput sequencing (HTS) technologies have enabled rapid sequencing of genomes and large-scale genome analytics with massive data sets. Traditionally, genetic variation analyses have been based on the human reference genome assembled from a relatively small human population. However, genetic variation could be discovered more comprehensively by using a collection of genomes i.e., pan-genome as a reference. The pan-genomic references can be assembled from larger populations or a specific population under study. Moreover, exploiting the pan-genomic references with current bioinformatics tools requires efficient compression and indexing methods. To be able to leverage the accumulating genomic data, the power of distributed and parallel computing has to be harnessed for the new genome analysis pipelines. We propose a scalable distributed pipeline, PanGenSpark, for compressing and indexing pan-genomes and assembling a reference genome from the pan-genomic index. We experimentally show the scalability of the PanGenSpark with human pan-genomes in a distributed Spark cluster comprising 448 cores distributed to 26 computing nodes. Assembling a consensus genome of a pan-genome including 50 human individuals was performed in 215 min and with 500 human individuals in 1468 min. The index of 1.41 TB pan-genome was compressed into a size of 164.5 GB in our experiments.

Original languageEnglish
Title of host publicationBig Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings
EditorsSurya Nepal, Wenqi Cao, Aziz Nasridinov, MD Zakirul Alam Bhuiyan, Xuan Guo, Liang-Jie Zhang
Number of pages17
ISBN (Print)9783030596118
Publication statusPublished - 1 Jan 2020
MoE publication typeA4 Article in a conference publication
EventIEEE International Conference on Big Data - Honolulu, United States
Duration: 18 Sept 202020 Sept 2020
Conference number: 9

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12402 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceIEEE International Conference on Big Data
Abbreviated titleBigData
Country/TerritoryUnited States


  • Big data
  • Compression
  • Computational genomics
  • Distributed computing
  • Genome assembly
  • Indexing


Dive into the research topics of 'Scalable Reference Genome Assembly from Compressed Pan-Genome Index with Spark'. Together they form a unique fingerprint.

Cite this