scholarly journals THBase: A Coprocessor-Based Scheme for Big Trajectory Data Management

2019 ◽  
Vol 11 (1) ◽  
pp. 10 ◽  
Author(s):  
Jiwei Qin ◽  
Liangli Ma ◽  
Jinghua Niu
Keyword(s):  
Data Management ◽  
Data Storage ◽  
Data Transmission ◽  
Rapid Development ◽  
Index Structure ◽  
Trajectory Data ◽  
Parallel Query ◽  
Object Based ◽  
Secondary Index ◽  
Spatiotemporal Queries

The rapid development of distributed technology has made it possible to store and query massive trajectory data. As a result, a variety of schemes for big trajectory data management have been proposed. However, the factor of data transmission is not considered in most of these, resulting in a certain impact on query efficiency. In view of that, we present THBase, a coprocessor-based scheme for big trajectory data management in HBase. THBase introduces a segment-based data model and a moving-object-based partition model to solve massive trajectory data storage, and exploits a hybrid local secondary index structure based on Observer coprocessor to accelerate spatiotemporal queries. Furthermore, it adopts certain maintenance strategies to ensure the colocation of relevant data. Based on these, THBase designs node-locality-based parallel query algorithms by Endpoint coprocessor to reduce the overhead caused by data transmission, thus ensuring efficient query performance. Experiments on datasets of ship trajectory show that our schemes can significantly outperform other schemes.

scholarly journals Massive Data Management and Sharing Module for Connectome Reconstruction

2020 ◽  
Vol 10 (5) ◽  
pp. 314
Author(s):  
Jingbin Yuan ◽  
Jing Zhang ◽  
Lijun Shen ◽  
Dandan Zhang ◽  
Wenhuan Yu ◽  
...  
Keyword(s):  
Data Management ◽  
Data Storage ◽  
Large Scale ◽  
Rapid Development ◽  
Massive Data ◽  
Storage And Retrieval ◽  
Server Side ◽  
Large Scale Data ◽  
Client Side ◽  
Scale Data

Recently, with the rapid development of electron microscopy (EM) technology and the increasing demand of neuron circuit reconstruction, the scale of reconstruction data grows significantly. This brings many challenges, one of which is how to effectively manage large-scale data so that researchers can mine valuable information. For this purpose, we developed a data management module equipped with two parts, a storage and retrieval module on the server-side and an image cache module on the client-side. On the server-side, Hadoop and HBase are introduced to resolve massive data storage and retrieval. The pyramid model is adopted to store electron microscope images, which represent multiresolution data of the image. A block storage method is proposed to store volume segmentation results. We design a spatial location-based retrieval method for fast obtaining images and segments by layers rapidly, which achieves a constant time complexity. On the client-side, a three-level image cache module is designed to reduce latency when acquiring data. Through theoretical analysis and practical tests, our tool shows excellent real-time performance when handling large-scale data. Additionally, the server-side can be used as a backend of other similar software or a public database to manage shared datasets, showing strong scalability.

scholarly journals The Design and Development of a Ship Trajectory Data Management and Analysis System Based on AIS

Sensors ◽  
2021 ◽  
Vol 22 (1) ◽  
pp. 310
Author(s):  
Chengxu Feng ◽  
Bing Fu ◽  
Yasong Luo ◽  
Houpu Li
Keyword(s):  
Data Management ◽  
Data Storage ◽  
Database Management ◽  
Trajectory Analysis ◽  
Machine Learning Algorithms ◽  
Functional Modules ◽  
Trajectory Data ◽  
Trajectory Mining ◽  
Motion Features ◽  
Analysis System

To address the data storage, management, analysis, and mining of ship targets, the object-oriented method was employed to design the overall structure and functional modules of a ship trajectory data management and analysis system (STDMAS). This paper elaborates the detailed design and technical information of the system’s logical structure, module composition, physical deployment, and main functional modules such as database management, trajectory analysis, trajectory mining, and situation analysis. A ship identification method based on the motion features was put forward. With the method, ship trajectory was first partitioned into sub-trajectories in various behavioral patterns, and effective motion features were then extracted. Machine learning algorithms were utilized for training and testing to identify many types of ships. STDMAS implements such functions as database management, trajectory analysis, historical situation review, and ship identification and outlier detection based on trajectory classification. STDMAS can satisfy the practical needs for the data management, analysis, and mining of maritime targets because it is easy to apply, maintain, and expand.

Regional Bitmap Index: A Secondary Index for Data Management in Could Computing Environment

2012 ◽  
Vol 35 (11) ◽  
pp. 2306 ◽  
Author(s):  
Bi-Ping MENG ◽  
Teng-Jiao WANG ◽  
Hong-Yan LI ◽  
Dong-Qing YANG
Keyword(s):  
Data Management ◽  
Computing Environment ◽  
Bitmap Index ◽  
Secondary Index

scholarly journals The on-premise data sharing infrastructure e!DAL: Foster FAIR data for faster data acquisition

GigaScience ◽  
2020 ◽  
Vol 9 (10) ◽  
Author(s):  
Daniel Arend ◽  
Patrick König ◽  
Astrid Junker ◽  
Uwe Scholz ◽  
Matthias Lange
Keyword(s):  
Data Management ◽  
Data Storage ◽  
Best Practice ◽  
Research Process ◽  
Research Data ◽  
Primary Data ◽  
Data Repository ◽  
Plant Genomics ◽  
Quality Of Data

Abstract Background The FAIR data principle as a commitment to support long-term research data management is widely accepted in the scientific community. Although the ELIXIR Core Data Resources and other established infrastructures provide comprehensive and long-term stable services and platforms for FAIR data management, a large quantity of research data is still hidden or at risk of getting lost. Currently, high-throughput plant genomics and phenomics technologies are producing research data in abundance, the storage of which is not covered by established core databases. This concerns the data volume, e.g., time series of images or high-resolution hyper-spectral data; the quality of data formatting and annotation, e.g., with regard to structure and annotation specifications of core databases; uncovered data domains; or organizational constraints prohibiting primary data storage outside institional boundaries. Results To share these potentially dark data in a FAIR way and master these challenges the ELIXIR Germany/de.NBI service Plant Genomic and Phenomics Research Data Repository (PGP) implements a “bring the infrastructure to the data” approach, which allows research data to be kept in place and wrapped in a FAIR-aware software infrastructure. This article presents new features of the e!DAL infrastructure software and the PGP repository as a best practice on how to easily set up FAIR-compliant and intuitive research data services. Furthermore, the integration of the ELIXIR Authentication and Authorization Infrastructure (AAI) and data discovery services are introduced as means to lower technical barriers and to increase the visibility of research data. Conclusion The e!DAL software matured to a powerful and FAIR-compliant infrastructure, while keeping the focus on flexible setup and integration into existing infrastructures and into the daily research process.

TrajMesa: A Distributed NoSQL-Based Trajectory Data Management System

2021 ◽  
pp. 1-1
Author(s):  
Ruiyuan Li ◽  
Huajun He ◽  
Rubin Wang ◽  
Sijie Ruan ◽  
Tianfu He ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
Data Management System ◽  
Trajectory Data

The Design and Implementation of ECC High-Speed Encryption Engine Based on FPGA

2012 ◽  
Vol 459 ◽  
pp. 544-548 ◽  
Author(s):  
Wei Liang ◽  
Jian Bo Xu ◽  
Wei Hong Huang ◽  
Li Peng
Keyword(s):  
Data Transmission ◽  
High Speed ◽  
Lower Cost ◽  
Low Cost ◽  
Rapid Development ◽  
Security Technology ◽  
Multiplication Algorithm ◽  
Secure Data Transmission ◽  
And Performance ◽  
Computation Speed

Network security technology ensures secure data transmission in network. Meanwhile, it brings extra overhead of security system in terms of cost and performance, which seriously affects the rapid development of existing high-speed encryption systems. The existing encryption technology cannot meet the demand of high security, low cost and high real-time. For solving above problems, an ECC encryption engine architecture based on scalable public key cipher and a high-speed configurable multiplication algorithm are designed. The algorithm was tested on FPGA platform and the experiment results show that the system has better computation speed and lower cost overhead. By comparing with other systems, our system has benefits in terms of hardware overhead and encryption time ratio

scholarly journals DBE2 – Management of experimental data for the VANTED system

2011 ◽  
Vol 8 (2) ◽  
pp. 85-94
Author(s):  
Hendrik Mehlhorn ◽  
Falk Schreiber
Keyword(s):  
Experimental Data ◽  
Information System ◽  
Data Integration ◽  
Data Management ◽  
Data Storage ◽  
Controlled Vocabulary ◽  
Biological Experiment ◽  
Experiment Data ◽  
Measurement Units ◽  
Persistent Data

Summary DBE2 is an information system for the management of biological experiment data from different data domains in a unified and simple way. It provides persistent data storage, worldwide accessibility of the data and the opportunity to load, save, modify, and annotate the data. It is seamlessly integrated in the VANTED system as an add-on, thereby extending the VANTED platform towards data management. DBE2 also utilizes controlled vocabulary from the Ontology Lookup Service to allow the management of terms such as substance names, species names, and measurement units, aiming at an eased data integration.

Data Storage Layout for Object-Based De-Duplication System

2013 ◽  
Vol 303-306 ◽  
pp. 2284-2288
Author(s):  
Fang Yan ◽  
Yu An Tan
Keyword(s):  
Data Storage ◽  
Storage System ◽  
Unstructured Data ◽  
Effective Solution ◽  
Advanced Method ◽  
Data Object ◽  
Object Based ◽  
The World ◽  
Long Time ◽  
Reduce Energy Consumption

The world is increasingly awash in more and more unstructured data. Object-based data de-duplication is the current most advanced method and is the effective solution for detecting duplicate data. We developed an energy saving policy for conventional disk based RAID systems. According to the characteristics of object-based data de-duplication, we introduce object layout strategies for unstructured data applications; disk accesses are concentrated in a part of the disks in a long time which is conducive to scheduling other disks into standby or shutdown mode. Our proposed methods reduce energy consumption of de-duplication storage system.

scholarly journals The Construction and Thinking of Cloud Computing and Big Data Technology in Smart Campus

2019 ◽  
Vol 3 (2) ◽  
pp. 152
Author(s):  
Xianglan Wu
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Data Storage ◽  
Information Technologies ◽  
Rapid Development ◽  
Smart Campus ◽  
Huge Data ◽  
New Information ◽  
Big Data Technology

<p>In today's society, the rise of the Internet and rapid development make every day produce a huge amount of data. Therefore, the traditional data processing mode and data storage can not be fully analyzed and mined these data. More and more new information technologies (such as cloud computing, virtualization and big data, etc.) have emerged and been applied, the network has turned from informationization to intelligence, and campus construction has ushered in the stage of smart campus construction.The construction of intelligent campus refers to big data and cloud computing technology, which improves the informatization service quality of colleges and universities by integrating, storing and mining huge data.</p>

Sign in / Sign up

Export Citation Format

Share Document