An Improved High-Throughput Data Processing Based on Combinatorial Materials Chip Approach for Rapid Construction of Fe–Cr–Ni Composition-Phase Map

2019 ◽  
Vol 21 (12) ◽  
pp. 833-842 ◽  
Author(s):  
Zhaoyang Zhao ◽  
Ying Jin ◽  
Peng Shi ◽  
Yanpeng Xue ◽  
Bingbing Zhao ◽  
...  
Keyword(s):  
Data Processing ◽  
High Throughput ◽  
High Throughput Data ◽  
Rapid Construction ◽  
Phase Map ◽  
Combinatorial Materials

scholarly journals Semiautomated Alignment of High-Throughput Metabolite Profiles with Chemometric Tools

2017 ◽  
Vol 2017 ◽  
pp. 1-9 ◽  
Author(s):  
Ze-ying Wu ◽  
Zhong-da Zeng ◽  
Zi-dan Xiao ◽  
Daniel Kam-Wah Mok ◽  
Yi-zeng Liang ◽  
...  
Keyword(s):  
Data Processing ◽  
High Throughput ◽  
Biomarker Discovery ◽  
Linear Interpolation ◽  
Time Shift ◽  
Data Partition ◽  
High Throughput Data ◽  
Automated Method ◽  
Metabolite Profiles ◽  
Standard Profile

The rapid increase in the use of metabolite profiling/fingerprinting techniques to resolve complicated issues in metabolomics has stimulated demand for data processing techniques, such as alignment, to extract detailed information. In this study, a new and automated method was developed to correct the retention time shift of high-dimensional and high-throughput data sets. Information from the target chromatographic profiles was used to determine the standard profile as a reference for alignment. A novel, piecewise data partition strategy was applied for the determination of the target components in the standard profile as markers for alignment. An automated target search (ATS) method was proposed to find the exact retention times of the selected targets in other profiles for alignment. The linear interpolation technique (LIT) was employed to align the profiles prior to pattern recognition, comprehensive comparison analysis, and other data processing steps. In total, 94 metabolite profiles of ginseng were studied, including the most volatile secondary metabolites. The method used in this article could be an essential step in the extraction of information from high-throughput data acquired in the study of systems biology, metabolomics, and biomarker discovery.

scholarly journals Optimizing performance of GATK workflows using Apache Arrow In-Memory data framework

BMC Genomics ◽  
2020 ◽  
Vol 21 (S10) ◽  
Author(s):  
Tanveer Ahmad ◽  
Nauman Ahmed ◽  
Zaid Al-Ars ◽  
H. Peter Hofstee
Keyword(s):  
Data Processing ◽  
Shared Memory ◽  
High Throughput ◽  
Data Representation ◽  
Data Sets ◽  
Computing Systems ◽  
Disk Storage ◽  
High Throughput Data ◽  
Data Framework ◽  
Development Framework

Abstract Background Immense improvements in sequencing technologies enable producing large amounts of high throughput and cost effective next-generation sequencing (NGS) data. This data needs to be processed efficiently for further downstream analyses. Computing systems need this large amounts of data closer to the processor (with low latency) for fast and efficient processing. However, existing workflows depend heavily on disk storage and access, to process this data incurs huge disk I/O overheads. Previously, due to the cost, volatility and other physical constraints of DRAM memory, it was not feasible to place large amounts of working data sets in memory. However, recent developments in storage-class memory and non-volatile memory technologies have enabled computing systems to place huge data in memory to process it directly from memory to avoid disk I/O bottlenecks. To exploit the benefits of such memory systems efficiently, proper formatted data placement in memory and its high throughput access is necessary by avoiding (de)-serialization and copy overheads in between processes. For this purpose, we use the newly developed Apache Arrow, a cross-language development framework that provides language-independent columnar in-memory data format for efficient in-memory big data analytics. This allows genomics applications developed in different programming languages to communicate in-memory without having to access disk storage and avoiding (de)-serialization and copy overheads. Implementation We integrate Apache Arrow in-memory based Sequence Alignment/Map (SAM) format and its shared memory objects store library in widely used genomics high throughput data processing applications like BWA-MEM, Picard and GATK to allow in-memory communication between these applications. In addition, this also allows us to exploit the cache locality of tabular data and parallel processing capabilities through shared memory objects. Results Our implementation shows that adopting in-memory SAM representation in genomics high throughput data processing applications results in better system resource utilization, low number of memory accesses due to high cache locality exploitation and parallel scalability due to shared memory objects. Our implementation focuses on the GATK best practices recommended workflows for germline analysis on whole genome sequencing (WGS) and whole exome sequencing (WES) data sets. We compare a number of existing in-memory data placing and sharing techniques like ramDisk and Unix pipes to show how columnar in-memory data representation outperforms both. We achieve a speedup of 4.85x and 4.76x for WGS and WES data, respectively, in overall execution time of variant calling workflows. Similarly, a speedup of 1.45x and 1.27x for these data sets, respectively, is achieved, as compared to the second fastest workflow. In some individual tools, particularly in sorting, duplicates removal and base quality score recalibration the speedup is even more promising. Availability The code and scripts used in our experiments are available in both container and repository form at: https://github.com/abs-tudelft/ArrowSAM.

Low-Overhead Fault Tolerance for High-Throughput Data Processing Systems

2011 ◽  
Author(s):  
Andre Martin ◽  
Thomas Knauth ◽  
Stephan Creutz ◽  
Diogo Becker ◽  
Stefan Weigert ◽  
...  
Keyword(s):  
Fault Tolerance ◽  
Data Processing ◽  
High Throughput ◽  
High Throughput Data

scholarly journals Rapid Construction of Fe–Co–Ni Composition-Phase Map by Combinatorial Materials Chip Approach

2018 ◽  
Vol 20 (3) ◽  
pp. 127-131 ◽  
Author(s):  
Hui Xing ◽  
Bingbing Zhao ◽  
Yujie Wang ◽  
Xiaoyi Zhang ◽  
Yang Ren ◽  
...  
Keyword(s):  
Rapid Construction ◽  
Phase Map ◽  
Combinatorial Materials
Sign in / Sign up

Export Citation Format

Share Document