A Distributed Storage Model for Healthcare Big Data Designed on HBase

2018 ◽  
Author(s):  
Lu Zhang ◽  
Qi Li ◽  
Ye Li ◽  
Yunpeng Cai
Keyword(s):  
Big Data ◽  
Distributed Storage ◽  
Storage Model

Building OLAP Cubes From Columnar NoSQL Data Warehouses

2019 ◽  
pp. 129-157
Author(s):  
Khaled Dehdouh
Keyword(s):  
Big Data ◽  
Database System ◽  
Massive Data ◽  
Data Warehouses ◽  
Online Analysis ◽  
Storage Model ◽  
Data Cubes ◽  
Nosql Database ◽  
Oriented Approach

In the big data warehouses context, a column-oriented NoSQL database system is considered as the storage model which is highly adapted to data warehouses and online analysis. Indeed, the use of NoSQL models allows data scalability easily and the columnar store is suitable for storing and managing massive data, especially for decisional queries. However, the column-oriented NoSQL DBMS do not offer online analysis operators (OLAP). To build OLAP cubes corresponding to the analysis contexts, the most common way is to integrate other software such as HIVE or Kylin which has a CUBE operator to build data cubes. By using that, the cube is built according to the row-oriented approach and does not allow to fully obtain the benefits of a column-oriented approach. In this chapter, the main contribution is to define a cube operator called MC-CUBE (MapReduce Columnar CUBE), which allows building columnar NoSQL cubes according to the columnar approach by taking into account the non-relational and distributed aspects when data warehouses are stored.

scholarly journals Research on Environmental Monitoring System Based on Microservices and Data Mining

2020 ◽  
Vol 143 ◽  
pp. 02031
Author(s):  
Yu Liu ◽  
Junge Huang ◽  
Ningqi Lu
Keyword(s):  
Big Data ◽  
Monitoring System ◽  
Distributed Storage ◽  
Automatic Monitoring ◽  
Selective Extraction ◽  
Monitoring Data ◽  
Environment Monitoring ◽  
Business Systems ◽  
Efficient System ◽  
Environmental Monitoring System

The monitoring systems of various industries have various types and different structures. There are problems of “data chimney” and “information islands”[1]. Monitoring data is difficult to be effectively utilized and cannot provide reliable data information to support for environmental security[2]. In this end, an environment monitoring system based on micro-service architecture is designed. The information management and automatic monitoring business systems are unified into a flexible, robust and efficient system platform to adapt to the big data analysis and the mining applications. Using Hadoop to build environment monitoring big data platform, distributed storage, selective extraction and efficient calculation of the massive environment monitoring data can be achieved. By integrating the detection and monitoring data of the ecological environment and in-depth mining it, a neural network model is established to automatically identify potential safety hazards and recommend corresponding treatment measures, so to assist in the comprehensive research and scientific decision-making of environmental safety and promote intelligent management of safety.

scholarly journals Distributed Storage Strategy and Visual Analysis for Economic Big Data

2021 ◽  
Vol 2021 ◽  
pp. 1-13
Author(s):  
Xiangli Chang ◽  
Hailang Cui
Keyword(s):  
Big Data ◽  
Data Storage ◽  
Visual Analysis ◽  
Distributed Storage ◽  
Storage System ◽  
Service Level Agreement ◽  
Wiener Filter ◽  
Service Level ◽  
Prototype System ◽  
Data Types

With the increasing popularity of a large number of Internet-based services and a large number of services hosted on cloud platforms, a more powerful back-end storage system is needed to support these services. At present, it is very difficult or impossible to implement a distributed storage to meet all the above assumptions. Therefore, the focus of research is to limit different characteristics to design different distributed storage solutions to meet different usage scenarios. Economic big data should have the basic requirements of high storage efficiency and fast retrieval speed. The large number of small files and the diversity of file types make the storage and retrieval of economic big data face severe challenges. This paper is oriented to the application requirements of cross-modal analysis of economic big data. According to the source and characteristics of economic big data, the data types are analyzed and the database storage architecture and data storage structure of economic big data are designed. Taking into account the spatial, temporal, and semantic characteristics of economic big data, this paper proposes a unified coding method based on the spatiotemporal data multilevel division strategy combined with Geohash and Hilbert and spatiotemporal semantic constraints. A prototype system was constructed based on Mongo DB, and the performance of the multilevel partition algorithm proposed in this paper was verified by the prototype system based on the realization of data storage management functions. The Wiener distributed memory based on the principle of Wiener filter is used to store the workload of each workload distributed storage window in a distributed manner. For distributed storage workloads, this article adopts specific types of workloads. According to its periodicity, the workload is divided into distributed storage windows of specific duration. At the beginning of each distributed storage window, distributed storage is distributed to the next distributed storage window. Experiments and tests have verified the distributed storage strategy proposed in this article, which proves that the Wiener distributed storage solution can save platform resources and configuration costs while ensuring Service Level Agreement (SLA).

scholarly journals Environmental Safety Monitoring System Based on Microservice Architecture and Machine Learning

2021 ◽  
Vol 2 (2) ◽  
pp. 2894-2902
Author(s):  
Yu Liu, Junge Huang ◽  
Jihao Wang
Keyword(s):  
Big Data ◽  
Monitoring System ◽  
Distributed Storage ◽  
Automatic Monitoring ◽  
Selective Extraction ◽  
Environmental Safety ◽  
Monitoring Data ◽  
Environment Monitoring ◽  
Business Systems ◽  
Efficient System

The monitoring systems of various industries have various types and different structures. There are problems of “data chimney” and “information islands”. Monitoring data is difficult to be effectively utilized and cannot provide reliable data information to support for environmental security. In this end, an environment monitoring system based on micro-service architecture is designed. The information management and automatic monitoring business systems are unified into a flexible, robust and efficient system platform to adapt to the big data analysis and the mining applications. Using Hadoop to build environment monitoring big data platform, distributed storage, selective extraction and efficient calculation of the massive environment monitoring data can be achieved. By integrating the detection and monitoring data of the ecological environment and in-depth mining it, a neural network model is established to automatically identify potential safety hazards and recommend corresponding treatment measures, so to assist in the comprehensive research and scientific decision-making of environmental safety and promote intelligent management of safety.

Sign in / Sign up

Export Citation Format

Share Document