Data management and query---Multiple range query optimization with distributed cache indexing

2006 ◽  
Author(s):  
Beomseok Nam ◽  
Henrique Andrade ◽  
Alan Sussman
Keyword(s):  
Data Management ◽  
Query Optimization ◽  
Range Query ◽  
Distributed Cache

scholarly journals Efficient IoT Data Management for Geological Disasters Based on Big Data-Turbocharged Data Lake Architecture

2021 ◽  
Vol 10 (11) ◽  
pp. 743
Author(s):  
Xiaohui Huang ◽  
Junqing Fan ◽  
Ze Deng ◽  
Jining Yan ◽  
Jiabao Li ◽  
...  
Keyword(s):  
Big Data ◽  
Data Management ◽  
Decision Makers ◽  
Prototype System ◽  
Data Repositories ◽  
Metadata Model ◽  
Parallel Data ◽  
Interactive Environment ◽  
Data Ingestion ◽  
Distributed Cache

Multi-source Internet of Things (IoT) data, archived in institutions’ repositories, are becoming more and more widely open-sourced to make them publicly accessed by scientists, developers, and decision makers via web services to promote researches on geohazards prevention. In this paper, we design and implement a big data-turbocharged system for effective IoT data management following the data lake architecture. We first propose a multi-threading parallel data ingestion method to ingest IoT data from institutions’ data repositories in parallel. Next, we design storage strategies for both ingested IoT data and processed IoT data to store them in a scalable, reliable storage environment. We also build a distributed cache layer to enable fast access to IoT data. Then, we provide users with a unified, SQL-based interactive environment to enable IoT data exploration by leveraging the processing ability of Apache Spark. In addition, we design a standard-based metadata model to describe ingested IoT data and thus support IoT dataset discovery. Finally, we implement a prototype system and conduct experiments on real IoT data repositories to evaluate the efficiency of the proposed system.

Sign in / Sign up

Export Citation Format

Share Document