Lightweight data indexing and compression in external memory

Paolo Ferragina*, Travis Gagie, Giovanni Manzini

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

19 Citations (Scopus)


In this paper we describe algorithms for computing the BWT and for building (compressed) indexes in external memory. The innovative feature of our algorithms is that they are lightweight in the sense that, for an input of size n, they use only n bits of disk working space while all previous approaches use Θ(n logn) bits of disk working space. Moreover, our algorithms access disk data only via sequential scans, thus they take full advantage of modern disk features that make sequential disk accesses much faster than random accesses. We also present a scan-based algorithm for inverting the BWT that uses Θ(n) bits of working space, and a lightweight internal-memory algorithm for computing the BWT which is the fastest in the literature when the available working space is o(n) bits. Finally, we prove lower bounds on the complexity of computing and inverting the BWT via sequential scans in terms of the classic product: internal-memory space × number of passes over the disk data, showing that our algorithms are within an O(logn) factor of the optimal.

Original languageEnglish
Title of host publicationLATIN 2010: Theoretical Informatics - 9th Latin American Symposium, Proceedings
Number of pages14
Volume6034 LNCS
Publication statusPublished - 2010
MoE publication typeA4 Article in a conference publication
EventLatin American Theoretical Informatics Symposium - Oaxaca, Mexico
Duration: 19 Apr 201023 Apr 2010
Conference number: 9

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6034 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349


ConferenceLatin American Theoretical Informatics Symposium
Abbreviated titleLATIN

Fingerprint Dive into the research topics of 'Lightweight data indexing and compression in external memory'. Together they form a unique fingerprint.

Cite this