scholarly journals Succinct Colored de Bruijn Graphs

2016 ◽  
Author(s):  
Keith Belk ◽  
Christina Boucher ◽  
Alexander Bowe ◽  
Travis Gagie ◽  
Paul Morley ◽  
...  
Keyword(s):  
Data Structure ◽  
Genetic Variants ◽  
Population Level ◽  
De Bruijn Graph ◽  
De Bruijn Graphs ◽  
Graph Representations ◽  
Additional Information ◽  
Level Data ◽  
De Bruijn ◽  
Colored De Bruijn Graph

Iqbal et al. (Nature Genetics, 2012) introduced the colored de Bruijn graph, a variant of the classic de Bruijn graph, which is aimed at "detecting and genotyping simple and complex genetic variants in an individual or population". Because they are intended to be applied to massive population level data, it is essential that the graphs be represented efficiently. Unfortunately, current succinct de Bruijn graph representations are not directly applicable to the colored de Bruijn graph, which require additional information to be succinctly encoded as well as support for non-standard traversal operations. Our data structure dramatically reduces the amount of memory required to store and use the colored de Bruijn graph, with some penalty to runtime, allowing it to be applied in much larger and more ambitious sequence projects than was previously possible.

scholarly journals A space and time-efficient index for the compacted colored de Bruijn graph

2017 ◽  
Author(s):  
Fatemeh Almodaresi ◽  
Hirak Sarkar ◽  
Rob Patro
Keyword(s):  
Data Structure ◽  
Pattern Search ◽  
De Bruijn Graph ◽  
Existing Structures ◽  
Link Type ◽  
Reference Information ◽  
De Bruijn ◽  
Colored De Bruijn Graph ◽  
Asymptotically Efficient ◽  
Fast Access

AbstractWe present a novel data structure for representing and indexing the compacted colored de Bruijn graph, which allows for efficient pattern matching and retrieval of the reference information associated with each k-mer. As the popularity of the de Bruijn graph as an index has increased over the past few years, so have the number of proposed representations of this structure. Existing structures typically fall into two categories; those that are hashing-based and provide very fast access to the underlying k-mer information, and those that are space-frugal and provide asymptotically efficient but practically slower pattern search.Our representation achieves a compromise between these two extremes. By building upon minimum perfect hashing, carefully organizing our data structure, and making use of succinct representations where applicable, our data structure provides practically fast k-mer lookup while greatly reducing the space compared to traditional hashing-based implementations. Further, we describe a sampling scheme built on the same underlying representation, which provides the ability to trade off k-mer query speed for a reduction in the de Bruijn graph index size. We believe this representation strikes a desirable balance between speed and space usage, and it will allow for fast search on large reference sequences.Pufferfish is developed in C++11, is open source (GPL v3), and is available at https://github.com/COMBINE-lab/Pufferfish. The scripts used to generate the results in this manuscript are available at https://github.com/COMBINE-lab/pufferfish_experiments.

scholarly journals Cuttlefish: Fast, parallel, and low-memory compaction of de Bruijn graphs from large-scale genome collections

2020 ◽  
Author(s):  
Jamshed Khan ◽  
Rob Patro
Keyword(s):  
Large Scale ◽  
De Bruijn Graph ◽  
Comparative Genomic ◽  
De Bruijn Graphs ◽  
Long Reads ◽  
Genomic Analyses ◽  
Finite State ◽  
De Bruijn ◽  
Colored De Bruijn Graph ◽  
Memory Compaction

AbstractMotivationThe construction of the compacted de Bruijn graph from a large collection of reference genomes is a task of increasing interest in genomic analyses. For example, compacted colored reference de Bruijn graphs are increasingly used as sequence indices for the purposes of alignment of short and long reads. Also, as we sequence and assemble a greater diversity of individual genomes, the compacted colored de Bruijn graph can be used as the basis for methods aiming to perform comparative genomic analyses on these genomes. While algorithms have been developed to construct the compacted colored de Bruijn graph from reference sequences, there is still room for improvement, especially in the memory and the runtime performance as the number and the scale of the genomes over which the de Bruijn graph is built grow.ResultsWe introduce a new algorithm, implemented in the tool Cuttlefish, to construct the colored compacted de Bruijn graph from a collection of one or more genome references. Cuttlefish introduces a novel modeling scheme of the de Bruijn graph vertices as finite-state automata, and constrains the state-space for the automata to enable tracking of their transitioning states with very low memory usage. Cuttlefish is also fast and highly parallelizable. Experimental results demonstrate that the algorithm scales much better than existing approaches, especially as the number and scale of the input references grow. For example, on a typical shared-memory machine, Cuttlefish constructed the compacted graph for 100 human genomes in less than 7 hours, using ~29 GB of memory; no other tested tool successfully completed this task on the testing hardware. We also applied Cuttlefish on 11 diverse conifer plant genomes, and the compacted graph was constructed in under 11 hours, using ~84 GB of memory, while the only other tested tool able to complete this compaction on our hardware took more than 16 hours and ~289 GB of memory.AvailabilityCuttlefish is written in C++14, and is available under an open source license at https://github.com/COMBINE-lab/[email protected]

scholarly journals Metagenome SNP calling via read-colored de Bruijn graphs

2020 ◽  
Author(s):  
Bahar Alipanahi ◽  
Martin D Muggli ◽  
Musa Jundi ◽  
Noelle R Noyes ◽  
Christina Boucher
Keyword(s):  
Pathogenic Bacteria ◽  
Read Length ◽  
Supplementary Information ◽  
De Bruijn Graph ◽  
Nucleotide Polymorphisms ◽  
Chromosomal Dna ◽  
Shotgun Metagenomics ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
Colored De Bruijn Graph

Abstract Motivation Metagenomics refers to the study of complex samples containing of genetic contents of multiple individual organisms and, thus, has been used to elucidate the microbiome and resistome of a complex sample. The microbiome refers to all microbial organisms in a sample, and the resistome refers to all of the antimicrobial resistance (AMR) genes in pathogenic and non-pathogenic bacteria. Single-nucleotide polymorphisms (SNPs) can be effectively used to ‘fingerprint’ specific organisms and genes within the microbiome and resistome and trace their movement across various samples. However, to effectively use these SNPs for this traceability, a scalable and accurate metagenomics SNP caller is needed. Moreover, such an SNP caller should not be reliant on reference genomes since 95% of microbial species is unculturable, making the determination of a reference genome extremely challenging. In this article, we address this need. Results We present LueVari, a reference-free SNP caller based on the read-colored de Bruijn graph, an extension of the traditional de Bruijn graph that allows repeated regions longer than the k-mer length and shorter than the read length to be identified unambiguously. LueVari is able to identify SNPs in both AMR genes and chromosomal DNA from shotgun metagenomics data with reliable sensitivity (between 91% and 99%) and precision (between 71% and 99%) as the performance of competing methods varies widely. Furthermore, we show that LueVari constructs sequences containing the variation, which span up to 97.8% of genes in datasets, which can be helpful in detecting distinct AMR genes in large metagenomic datasets. Availability and implementation Code and datasets are publicly available at https://github.com/baharpan/cosmo/tree/LueVari. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Integrating long-range connectivity information into de Bruijn graphs

2017 ◽  
Author(s):  
Isaac Turner ◽  
Kiran V Garimella ◽  
Zamin Iqbal ◽  
Gil McVean
Keyword(s):  
Data Structure ◽  
Long Range ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Genomic Context ◽  
De Bruijn Graphs ◽  
Connectivity Information ◽  
String Graph ◽  
Free Data ◽  
De Bruijn

AbstractMotivationThe de Bruijn graph is a simple and efficient data structure that is used in many areas of sequence analysis including genome assembly, read error correction and variant calling. The data structure has a single parameter k, is straightforward to implement and is tractable for large genomes with high sequencing depth. It also enables representation of multiple samples simultaneously to facilitate comparison. However, unlike the string graph, a de Bruijn graph does not retain long range information that is inherent in the read data. For this reason, applications that rely on de Bruijn graphs can produce sub-optimal results given their input.ResultsWe present a novel assembly graph data structure: the Linked de Bruijn Graph (LdBG). Constructed by adding annotations on top of a de Bruijn graph, it stores long range connectivity information through the graph. We show that with error-free data it is possible to losslessly store and recover sequence from a Linked de Bruijn graph. With assembly simulations we demonstrate that the LdBG data structure outperforms both the de Bruijn graph and the String Graph Assembler (SGA). Finally we apply the LdBG to Klebsiella pneumoniae short read data to make large (12 kbp) variant calls, which we validate using PacBio sequencing data, and to characterise the genomic context of drug-resistance genes.AvailabilityLinked de Bruijn Graphs and associated algorithms are implemented as part of McCortex, available under the MIT license at https://github.com/mcvean/[email protected].

scholarly journals Bifrost – Highly parallel construction and indexing of colored and compacted de Bruijn graphs

2019 ◽  
Author(s):  
Guillaume Holley ◽  
Páll Melsted
Keyword(s):  
High Throughput Sequencing ◽  
Genomic Analysis ◽  
Main Memory ◽  
De Bruijn Graph ◽  
Direct Construction ◽  
De Bruijn Graphs ◽  
Wide Range ◽  
User Data ◽  
De Bruijn ◽  
Colored De Bruijn Graph

AbstractMotivationDe Bruijn graphs are the core data structure for a wide range of assemblers and genome analysis software processing High Throughput Sequencing datasets. For population genomic analysis, the colored de Bruijn graph is often used in order to take advantage of the massive sets of sequenced genomes available for each species. However, memory consumption of tools based on the de Bruijn graph is often prohibitive, due to the high number of vertices, edges or colors in the graph. In order to process large and complex genomes, most short-read assemblers based on the de Bruijn graph paradigm reduce the assembly complexity and memory usage by compacting first all maximal non-branching paths of the graph into single vertices. Yet, de Bruijn graph compaction is challenging as it requires the uncompacted de Bruijn graph to be available in memory.ResultsWe present a new parallel and memory efficient algorithm enabling the direct construction of the compacted de Bruijn graph without producing the intermediate uncompacted de Bruijn graph. Bifrost features a broad range of functions such as sequence querying, storage of user data alongside vertices and graph editing that automatically preserve the compaction property. Bifrost makes full use of the dynamic index efficiency and proposes a graph coloring method efficiently mapping eachk-mer of the graph to the set of genomes in which it occurs. Experimental results show that our algorithm is competitive with state-of-the-art de Bruijn graph compaction and coloring tools. Bifrost was able to build the colored and compacted de Bruijn graph of about 118,000 Salmonella genomes on a mid-class server in about 4 days using 103 GB of main memory.Availabilityhttps://github.com/pmelsted/bifrostavailable with a BSD-2 [email protected]

scholarly journals LOGAN: A framework for LOssless Graph-based ANalysis of high throughput sequence data

2017 ◽  
Author(s):  
Anthony Bolger ◽  
Alisandra Denton ◽  
Marie Bolger ◽  
Björn Usadel
Keyword(s):  
Data Structure ◽  
Data Structures ◽  
Sequence Data ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Solanum Pennellii ◽  
Desktop Computer ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
Memory Efficient

AbstractRecent massive growth in the production of sequencing data necessitates matching improve-ments in bioinformatics tools to effectively utilize it. Existing tools suffer from limitations in both scalability and applicability which are inherent to their underlying algorithms and data structures. We identify the key requirements for the ideal data structure for sequence analy-ses: it should be informationally lossless, locally updatable, and memory efficient; requirements which are not met by data structures underlying the major assembly strategies Overlap Layout Consensus and De Bruijn Graphs. We therefore propose a new data structure, the LOGAN graph, which is based on a memory efficient Sparse De Bruijn Graph with routing information. Innovations in storing routing information and careful implementation allow sequence datasets for Escherichia coli (4.6Mbp, 117x coverage), Arabidopsis thaliana (135Mbp, 17.5x coverage) and Solanum pennellii (1.2Gbp, 47x coverage) to be loaded into memory on a desktop computer in seconds, minutes, and hours respectively. Memory consumption is competitive with state of the art alternatives, while losslessly representing the reads in an indexed and updatable form. Both Second and Third Generation Sequencing reads are supported. Thus, the LOGAN graph is positioned to be the backbone for major breakthroughs in sequence analysis such as integrated hybrid assembly, assembly of exceptionally large and repetitive genomes, as well as assembly and representation of pan-genomes.

scholarly journals Building large updatable colored de Bruijn graphs via merging

2019 ◽  
Vol 35 (14) ◽  
pp. i51-i60 ◽  
Author(s):  
Martin D Muggli ◽  
Bahar Alipanahi ◽  
Christina Boucher
Keyword(s):  
Bloom Filter ◽  
Supplementary Information ◽  
Metagenomic Data ◽  
De Bruijn Graph ◽  
De Bruijn Graphs ◽  
Working Space ◽  
Fold Reduction ◽  
De Bruijn ◽  
Colored De Bruijn Graph ◽  
Public Datasets

Abstract Motivation There exist several large genomic and metagenomic data collection efforts, including GenomeTrakr and MetaSub, which are routinely updated with new data. To analyze such datasets, memory-efficient methods to construct and store the colored de Bruijn graph were developed. Yet, a problem that has not been considered is constructing the colored de Bruijn graph in a scalable manner that allows new data to be added without reconstruction. This problem is important for large public datasets as scalability is needed but also the ability to update the construction is also needed. Results We create a method for constructing the colored de Bruijn graph for large datasets that is based on partitioning the data into smaller datasets, building the colored de Bruijn graph using a FM-index based representation, and succinctly merging these representations to build a single graph. The last step, merging succinctly, is the algorithmic challenge which we solve in this article. We refer to the resulting method as VariMerge. This construction method also allows the graph to be updated with new data. We validate our approach and show it produces a three-fold reduction in working space when constructing a colored de Bruijn graph for 8000 strains. Lastly, we compare VariMerge to other competing methods—including Vari, Rainbowfish, Mantis, Bloom Filter Trie, the method of Almodaresi et al. and Multi-BRWT—and illustrate that VariMerge is the only method that is capable of building the colored de Bruijn graph for 16 000 strains in a manner that allows it to be updated. Competing methods either did not scale to this large of a dataset or do not allow for additions without reconstruction. Availability and implementation VariMerge is available at https://github.com/cosmo-team/cosmo/tree/VARI-merge under GPLv3 license. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Building Large Updatable Colored de Bruijn Graphs via Merging

2017 ◽  
Author(s):  
Martin D Muggli ◽  
Bahar Alipanahi ◽  
Christina Boucher
Keyword(s):  
Bloom Filter ◽  
Metagenomic Data ◽  
De Bruijn Graph ◽  
De Bruijn Graphs ◽  
Working Space ◽  
Fold Reduction ◽  
De Bruijn ◽  
Colored De Bruijn Graph ◽  
Public Datasets ◽  
Memory Efficient

MOTIVATION: There exists several massive genomic and metagenomic data collection efforts, including GenomeTrakr and MetaSub, which are routinely updated with new data. To analyze such datasets, memory-efficient methods to construct and store the colored de Bruijn graph have been developed. Yet, a problem that has not been considered is constructing the colored de Bruijn graph in a scalable manner that allows new data to be added without reconstruction. This problem is important for large public datasets as scalability is needed but also the ability to update the construction is also needed. RESULTS: We create a method for constructing and updating the colored de Bruijn graph on a very-large dataset through partitioning the data into smaller subsets, building the colored de Bruijn graph using a FM-index based representation, and succinctly merging these representations to build a single graph. The last step, merging succinctly, is the algorithmic challenge which we solve in this paper. We refer to the resulting method as VariMerge. We validate our approach, and show it produces a three-fold reduction in working space when constructing a colored de Bruijn graph for 8,000 strains. Lastly, we compare VariMerge to other competing methods --- including Vari, Rainbowfish, Mantis, Bloom Filter Trie, the method by Almodaresi(2019) and Multi-BRWT --- and illustrate that VariMerge is the only method that is capable of building the colored de Bruijn graph for 16,000 strains in a manner that allows additional samples to be added. Competing methods either did not scale to this large of a dataset or cannot allow for additions without reconstruction. AVAILABILITY: Our software is available under GPLv3 at https://github.com/cosmo-team/cosmo/tree/VARI-merge.

scholarly journals Resistome SNP Calling via Read Colored de Bruijn Graphs

2017 ◽  
Author(s):  
Bahar Alipanahi ◽  
Martin D. Muggli ◽  
Musa Jundi ◽  
Noelle Noyes ◽  
Christina Boucher
Keyword(s):  
Pathogenic Bacteria ◽  
Read Length ◽  
Metagenomic Data ◽  
De Bruijn Graph ◽  
Significant Advance ◽  
Nucleotide Polymorphisms ◽  
Single Nucleotide ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
Colored De Bruijn Graph

AbstractMotivationThe resistome, which refers to all of the antimicrobial resistance (AMR) genes in pathogenic and non-pathogenic bacteria, is frequently studied using shotgun metagenomic data [14, 47]. Unfortunately, few existing methods are able to identify single nucleotide polymorphisms (SNPs) within metagenomic data, and to the best of our knowledge, no methods exist to detect SNPs within AMR genes within the resistome. The ability to identify SNPs in AMR genes across the resistome would represent a significant advance in understanding the dissemination and evolution of AMR, as SNP identification would enable “fingerprinting” of the resistome, which could then be used to track AMR dynamics across various settings and/or time periods.ResultsWe present LueVari, a reference-free SNP caller based on the read colored de Bruijn graph, an extension of the traditional de Bruijn graph that allows repeated regions longer than the k-mer length and shorter than the read length to be identified unambiguously. We demonstrate LueVari was the only method that had reliable sensitivity (between 73% and 98%) as the performance of competing methods varied widely. Furthermore, we show LueVari constructs sequences containing the variation which span 93% of the gene in datasets with lower coverage (15X), and 100% of the gene in datasets with higher coverage (30X).AvailabilityCode and datasets are publicly available at https://github.com/baharpan/cosmo/tree/LueVari.

scholarly journals Detecting High Scoring Local Alignments in Pangenome Graphs

2020 ◽  
Author(s):  
Tizian Schulz ◽  
Roland Wittler ◽  
Sven Rahmann ◽  
Faraz Hach ◽  
Jens Stoye
Keyword(s):  
Sequence Similarity ◽  
Query Sequence ◽  
Heuristic Method ◽  
De Bruijn Graph ◽  
Local Alignment ◽  
Memory Usage ◽  
Sequence Comparisons ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
Colored De Bruijn Graph

AbstractMotivationIncreasing amounts of individual genomes sequenced per species motivate the usage of pangenomic approaches. Pangenomes may be represented as graphical structures, e.g. compacted colored de Bruijn graphs, which offer a low memory usage and facilitate reference-free sequence comparisons. While sequence-to-graph mapping to graphical pangenomes has been studied for some time, no local alignment search tool in the vein of BLAST has been proposed yet.ResultsWe present a new heuristic method to find maximum scoring local alignments of a DNA query sequence to a pangenome represented as a compacted colored de Bruijn graph. Our approach additionally allows a comparison of similarity among sequences within the pangenome. We show that local alignment scores follow an exponential-tail distribution similar to BLAST scores, and we discuss how to estimate its parameters to separate local alignments representing sequence homology from spurious findings. An implementation of our method is presented, and its performance and usability are shown. Our approach scales sublinearly in running time and memory usage with respect to the number of genomes under consideration. This is an advantage over classical methods that do not make use of sequence similarity within the pangenome.

Sign in / Sign up

Export Citation Format

Share Document