scholarly journals Improving on hash-based probabilistic sequence classification using multiple spaced seeds and multi-index Bloom filters

2018 ◽  
Author(s):  
Justin Chu ◽  
Hamid Mohamadi ◽  
Emre Erhan ◽  
Jeffery Tse ◽  
Readman Chiu ◽  
...  
Keyword(s):  
Hash Table ◽  
Bloom Filter ◽  
Bloom Filters ◽  
Sequencing Data ◽  
Multi Index ◽  
Sequencing Errors ◽  
Spaced Seeds ◽  
The Face ◽  
Order Of Magnitude ◽  
Higher Sensitivity

ABSTRACTAlignment-free classification of sequences against collections of sequences has enabled high-throughput processing of sequencing data in many bioinformatics analysis pipelines. Originally hash-table based, much work has been done to improve and reduce the memory requirement of indexing of k-mer sequences with probabilistic indexing strategies. These efforts have led to lower memory highly efficient indexes, but often lack sensitivity in the face of sequencing errors or polymorphism because they are k-mer based. To address this, we designed a new memory efficient data structure that can tolerate mismatches using multiple spaced seeds, called a multi-index Bloom Filter. Implemented as part of BioBloom Tools, we demonstrate our algorithm in two applications, read binning for targeted assembly and taxonomic read assignment. Our tool shows a higher sensitivity and specificity for read-binning than BWA MEM at an order of magnitude less time. For taxonomic classification, we show higher sensitivity than CLARK-S at an order of magnitude less time while using half the memory.

scholarly journals Mismatch-tolerant, alignment-free sequence classification using multiple spaced seeds and multiindex Bloom filters

2020 ◽  
Vol 117 (29) ◽  
pp. 16961-16968
Author(s):  
Justin Chu ◽  
Hamid Mohamadi ◽  
Emre Erhan ◽  
Jeffery Tse ◽  
Readman Chiu ◽  
...  
Keyword(s):  
False Positive Rate ◽  
Bloom Filter ◽  
Bloom Filters ◽  
Computational Time ◽  
Sequencing Data ◽  
Sequencing Errors ◽  
Alignment Free ◽  
Spaced Seeds ◽  
Order Of Magnitude ◽  
Higher Sensitivity

Alignment-free classification tools have enabled high-throughput processing of sequencing data in many bioinformatics analysis pipelines primarily due to their computational efficiency. Originallyk-mer based, such tools often lack sensitivity when faced with sequencing errors and polymorphisms. In response, some tools have been augmented with spaced seeds, which are capable of tolerating mismatches. However, spaced seeds have seen little practical use in classification because they bring increased computational and memory costs compared to methods that usek-mers. These limitations have also caused the design and length of practical spaced seeds to be constrained, since storing spaced seeds can be costly. To address these challenges, we have designed a probabilistic data structure called a multiindex Bloom Filter (miBF), which can store multiple spaced seed sequences with a low memory cost that remains static regardless of seed length or seed design. We formalize how to minimize the false-positive rate of miBFs when classifying sequences from multiple targets or references. Available within BioBloom Tools, we illustrate the utility of miBF in two use cases: read-binning for targeted assembly, and taxonomic read assignment. In our benchmarks, an analysis pipeline based on miBF shows higher sensitivity and specificity for read-binning than sequence alignment-based methods, also executing in less time. Similarly, for taxonomic classification, miBF enables higher sensitivity than a conventional spaced seed-based approach, while using half the memory and an order of magnitude less computational time.

scholarly journals Spaced Seed Data Structures forDe NovoAssembly

2015 ◽  
Vol 2015 ◽  
pp. 1-8 ◽  
Author(s):  
Inanç Birol ◽  
Justin Chu ◽  
Hamid Mohamadi ◽  
Shaun D. Jackman ◽  
Karthika Raghavan ◽  
...  
Keyword(s):  
Data Structure ◽  
Data Structures ◽  
De Novo ◽  
Bloom Filters ◽  
De Bruijn Graph ◽  
Sequence Specificity ◽  
Sequencing Errors ◽  
Spaced Seeds ◽  
Read Error Correction ◽  
Seed Data

De novoassembly of the genome of a species is essential in the absence of a reference genome sequence. Many scalable assembly algorithms use the de Bruijn graph (DBG) paradigm to reconstruct genomes, where a table of subsequences of a certain length is derived from the reads, and their overlaps are analyzed to assemble sequences. Despite longer subsequences unlocking longer genomic features for assembly, associated increase in compute resources limits the practicability of DBG over other assembly archetypes already designed for longer reads. Here, we revisit the DBG paradigm to adapt it to the changing sequencing technology landscape and introduce three data structure designs for spaced seeds in the form of paired subsequences. These data structures address memory and run time constraints imposed by longer reads. We observe that when a fixed distance separates seed pairs, it provides increased sequence specificity with increased gap length. Further, we note that Bloom filters would be suitable to implicitly store spaced seeds and be tolerant to sequencing errors. Building on this concept, we describe a data structure for tracking the frequencies of observed spaced seeds. These data structure designs will have applications in genome, transcriptome and metagenome assemblies, and read error correction.

FMPC: A Fast Multi-Dimensional Packet Classification Algorithm

2014 ◽  
Vol 644-650 ◽  
pp. 3365-3370
Author(s):  
Zhen Hong Guo ◽  
Lin Li ◽  
Qing Wang ◽  
Meng Lin ◽  
Rui Pan
Keyword(s):  
Rapid Development ◽  
Hash Table ◽  
Bloom Filter ◽  
Packet Classification ◽  
Binary Search ◽  
Bloom Filters ◽  
Algorithm Analysis ◽  
Hash Tables ◽  
Worst Case ◽  
Embedded Sram

With the rapid development of the Internet, the number of firewall rules is increasing. The enormous quantity of rules challenges the performance of the packet classification that has already become a bottleneck in firewalls. This dissertation proposes a rapid and multi-dimensional algorithm for packet classification based on BSOL(Binary Search On Leaves), which is named FMPC(FastMulti-dimensional Packet Classification). Different from BSOL, FMPC cuts all dimensions at the same time to decompose rule spaces and stores leaf spaces into hash tables; FMPC constructs a Bloom Filter for every hash table and stores them into embedded SRAM. When classifying a packet, FMPC performs parallel queries on Bloom Filters and determines how to visit hash tables according to the results. Algorithm analysis and the result of simulations show: the average number of hash-table lookups of FMPC is 1 when classifying a packet, which is much smaller than that of BSOL; inthe worst case, the number of hash-table lookups of FMPCisO(logwmax+1⁡), which is also smaller than that of BSOL in multi-dimensional environment, where wmax is the length, in bits, of the dimension whose length is the longest..

scholarly journals kmtricks: Efficient construction of Bloom filters for large sequencing data collections

2021 ◽  
Author(s):  
Téo Lemane ◽  
Paul Medvedev ◽  
Rayan Chikhi ◽  
Pierre Peterlongo
Keyword(s):  
Bloom Filter ◽  
Bloom Filters ◽  
Complex Data ◽  
Sequencing Data ◽  
Novel Approach ◽  
Metagenomics Data ◽  
Efficient Construction ◽  
Common Operation ◽  
Data Collections ◽  
Multiple Samples

AbstractWhen indexing large collection of sequencing data, a common operation that has now been implemented in several tools (Sequence Bloom Trees and variants, BIGSI, ..) is to construct a collection of Bloom filters, one per sample. Each Bloom filter is used to represent a set of k-mers which approximates the desired set of all the non-erroneous k-mers present in the sample. However, this approximation is imperfect, especially in the case of metagenomics data. Erroneous but abundant k-mers are wrongly included, and non-erroneous but low-abundant ones are wrongly discarded. We propose kmtricks, a novel approach for generating Bloom filters from terabase-sized collections of sequencing data. Our main contributions are 1/ an efficient method for jointly counting k-mers across multiple samples, including a streamlined Bloom filter construction by directly counting hashes instead of k-mers; 2/ a novel technique that takes advantage of joint counting to preserve low-abundant k-mers present in several samples, improving the recovery of non-erroneous k-mers. In addition, our experimental results highlight that the usual yet crude filtering of low-abundant k-mers is inappropriate for complex data such as metagenomes.Availabilityhttps://github.com/tlemane/[email protected]

scholarly journals Lighter: fast and memory-efficient error correction without counting

2014 ◽  
Author(s):  
Li Song ◽  
Liliana Florea ◽  
Ben Langmead
Keyword(s):  
Bloom Filter ◽  
The Other ◽  
Bloom Filters ◽  
Efficient Tool ◽  
Sequencing Errors ◽  
Sampling Fraction ◽  
Filter Size ◽  
Comparable Accuracy ◽  
Fast Memory ◽  
Memory Efficient

Lighter is a fast, memory-efficient tool for correcting sequencing errors. Lighter avoids counting k-mers. Instead, it uses a pair of Bloom filters, one holding a sample of the input k-mers and the other holding k-mers likely to be correct. As long as the sampling fraction is adjusted in inverse proportion to the depth of sequencing, Bloom filter size can be held constant while maintaining near-constant accuracy. Lighter is parallelized, uses no secondary storage, and is both faster and more memory-efficient than competing approaches while achieving comparable accuracy.

scholarly journals Increased yields of duplex sequencing data by a series of quality control tools

2021 ◽  
Vol 3 (1) ◽  
Author(s):  
Gundula Povysil ◽  
Monika Heinzl ◽  
Renato Salazar ◽  
Nicholas Stoler ◽  
Anton Nekrutenko ◽  
...  
Keyword(s):  
Low Frequency ◽  
Variant Calling ◽  
Data Loss ◽  
Sequencing Data ◽  
Bioinformatics Pipeline ◽  
Consensus Sequences ◽  
Sequencing Errors ◽  
Data Output ◽  
Reverse Strand ◽  
Duplex Sequencing

Abstract Duplex sequencing is currently the most reliable method to identify ultra-low frequency DNA variants by grouping sequence reads derived from the same DNA molecule into families with information on the forward and reverse strand. However, only a small proportion of reads are assembled into duplex consensus sequences (DCS), and reads with potentially valuable information are discarded at different steps of the bioinformatics pipeline, especially reads without a family. We developed a bioinformatics toolset that analyses the tag and family composition with the purpose to understand data loss and implement modifications to maximize the data output for the variant calling. Specifically, our tools show that tags contain polymerase chain reaction and sequencing errors that contribute to data loss and lower DCS yields. Our tools also identified chimeras, which likely reflect barcode collisions. Finally, we also developed a tool that re-examines variant calls from raw reads and provides different summary data that categorizes the confidence level of a variant call by a tier-based system. With this tool, we can include reads without a family and check the reliability of the call, that increases substantially the sequencing depth for variant calling, a particular important advantage for low-input samples or low-coverage regions.

scholarly journals How Cooperative Engagement Programs Strengthen Sequencing Capabilities for Biosurveillance and Outbreak Response

2021 ◽  
Vol 9 ◽  
Author(s):  
Andrew W. Bartlow ◽  
Earl A. Middlebrook ◽  
Alicia T. Romero ◽  
Jeanne M. Fair
Keyword(s):  
Global Health ◽  
Technical Assistance ◽  
Emerging Infectious Diseases ◽  
Sequencing Data ◽  
Time Data ◽  
Outbreak Response ◽  
Health Security ◽  
The Face ◽  
Working Together ◽  
Etiological Agents

The threat of emerging and re-emerging infectious diseases continues to be a challenge to public and global health security. Cooperative biological engagement programs act to build partnerships and collaborations between scientists and health professionals to strengthen capabilities in biosurveillance. Biosurveillance is the systematic process of detecting, reporting, and responding to especially dangerous pathogens and pathogens of pandemic potential before they become outbreaks, epidemics, and pandemics. One important tool in biosurveillance is next generation sequencing. Expensive sequencing machines, reagents, and supplies make it difficult for countries to adopt this technology. Cooperative engagement programs help by providing funding for technical assistance to strengthen sequencing capabilities. Through workshops and training, countries are able to learn sequencing and bioinformatics, and implement these tools in their biosurveillance programs. Cooperative programs have an important role in building and sustaining collaborations among institutions and countries. One of the most important pieces in fostering these collaborations is trust. Trust provides the confidence that a successful collaboration will benefit all parties involved. With sequencing, this enables the sharing of pathogen samples and sequences. Obtaining global sequencing data helps to identify unknown etiological agents, track pathogen evolution and infer transmission networks throughout the duration of a pandemic. Having sequencing technology in place for biosurveillance generates the capacity to provide real-time data to understand and respond to pandemics. We highlight the need for these programs to continue to strengthen sequencing in biosurveillance. By working together to strengthen sequencing capabilities, trust can be formed, benefitting global health in the face of biological threats.

scholarly journals A Novel Hardware Security Architecture: PD-CRP(PUF Database & Challenge-Response Pair) Bloom Filter on Memristor Based PUF

2020 ◽  
Author(s):  
Jungwon Lee ◽  
Seoyeon Choi ◽  
Dayoung Kim ◽  
Yunyoung Choi ◽  
Wookyung Sun
Keyword(s):  
Data Transmission ◽  
Hardware Security ◽  
Bloom Filter ◽  
Transmission Error ◽  
Bloom Filters ◽  
Search Performance ◽  
Security Technology ◽  
Security Environment ◽  
Filter Size ◽  
Simulation Results

Because the development of the internet of things (IoT) requires technology that transfers information between objects without human intervention, the core of IoT security will be secure authentication between devices or between devices and servers. Software-based authentication may be a security vulnerability in IoT, but hardware-based security technology can provide a strong security environment. A physical unclonable functions (PUFs) are a hardware security element suitable for lightweight applications. PUFs can generate challenge-response pairs(CRPs) that cannot be controlled or predicted by utilizing inherent physical variations that occur in the manufacturing process. In particular, pulse width memristive PUF (PWM-PUF) improves security performance by applying different write pulse widths and bank structures. Bloom filter (BF) is probabilistic data structures that answer membership queries using small memories. Bloom filter can improve search performance and reduce memory usage and are used in areas such as networking, security, big data, and IoT. In this paper, we propose a structure that applies Bloom filters based on the PWM-PUF to reduce PUF data transmission errors. The proposed structure uses two different Bloom filter types that store different information and that are located in front of and behind the PWM-PUF, improving security by removing challenges from attacker access. Simulation results show that the proposed structure decreases the data transmission error rate and reuse rate as the Bloom filter size increases, the simulation results also show that the proposed structure improves PWM-PUF security with a very small Bloom filter memory.

scholarly journals K-mer counting with low memory consumption enables fast clustering of single-cell sequencing data without read alignment

2019 ◽  
Author(s):  
Christina Huan Shi ◽  
Kevin Y. Yip
Keyword(s):  
Single Cell ◽  
State Of The Art ◽  
Rna Seq ◽  
Sequencing Data ◽  
Memory Consumption ◽  
Analysis Pipeline ◽  
Cell Clusters ◽  
Single Cell Sequencing ◽  
Sequencing Errors ◽  
Full Analysis

AbstractK-mer counting has many applications in sequencing data processing and analysis. However, sequencing errors can produce many false k-mers that substantially increase the memory requirement during counting. We propose a fast k-mer counting method, CQF-deNoise, which has a novel component for dynamically identifying and removing false k-mers while preserving counting accuracy. Compared with four state-of-the-art k-mer counting methods, CQF-deNoise consumed 49-76% less memory than the second best method, but still ran competitively fast. The k-mer counts from CQF-deNoise produced cell clusters from single-cell RNA-seq data highly consistent with CellRanger but required only 5% of the running time at the same memory consumption, suggesting that CQF-deNoise can be used for a preview of cell clusters for an early detection of potential data problems, before running a much more time-consuming full analysis pipeline.

Improved Compression of DNA Sequencing Data with Cascading Bloom Filters

2018 ◽  
Vol 29 (08) ◽  
pp. 1249-1255
Author(s):  
Kamil Salikhov
Keyword(s):  
Dna Sequencing ◽  
Sequence Data ◽  
Real Data ◽  
Compression Algorithm ◽  
Computational Experiments ◽  
Bloom Filters ◽  
Dna Fragments ◽  
Sequencing Data ◽  
Sequencing Technologies ◽  
Memory Reduction

Modern DNA sequencing technologies generate prodigious volumes of sequence data consisting of short DNA fragments (reads). Storing and transferring this data is often challenging. With this motivation, several specialized compression methods have been developed. In this paper, we present an improvement of the lossless reference-free compression algorithm, suggested by Rozov et al., based on the technique of cascading Bloom filters. Through computational experiments on real data, we demonstrate that our method results in a significant associated memory reduction in practice.

Sign in / Sign up

Export Citation Format

Share Document