The Implementation of a Data-Accessing Platform Built from Big Data Warehouse of Electric Loads

2018 ◽  
Author(s):  
Sheng-Cang Chou ◽  
Chao-Tung Yang ◽  
Fuu-Cheng Jiang ◽  
Chih-Hung Chang
Keyword(s):  
Big Data ◽  
Data Warehouse ◽  
Electric Loads ◽  
Big Data Warehouse

On construction of a big data warehouse accessing platform for campus power usages

2019 ◽  
Vol 133 ◽  
pp. 40-50
Author(s):  
Chih-Hung Chang ◽  
Fuu-Cheng Jiang ◽  
Chao-Tung Yang ◽  
Sheng-Cang Chou
Keyword(s):  
Big Data ◽  
Data Warehouse ◽  
Big Data Warehouse

Efficient query processing framework for big data warehouse: an almost join-free approach

2015 ◽  
Vol 9 (2) ◽  
pp. 224-236 ◽  
Author(s):  
Huiju Wang ◽  
Xiongpai Qin ◽  
Xuan Zhou ◽  
Furong Li ◽  
Zuoyan Qin ◽  
...  
Keyword(s):  
Big Data ◽  
Query Processing ◽  
Data Warehouse ◽  
Efficient Query Processing ◽  
Processing Framework ◽  
Big Data Warehouse

Integration and optimization of multiple big data processing platforms

2016 ◽  
Vol 33 (6) ◽  
pp. 1680-1704 ◽  
Author(s):  
Bao-Rong Chang ◽  
Hsiu-Fen Tsai ◽  
Yun-Che Tsai ◽  
Chin-Fu Kuo ◽  
Chi-Chung Chen
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Data Warehouse ◽  
Search Time ◽  
Memory Capacity ◽  
Memory Storage ◽  
Big Data Processing ◽  
Content Type ◽  
Memory Cache ◽  
Big Data Warehouse

Purpose – The purpose of this paper is to integrate and optimize a multiple big data processing platform with the features of high performance, high availability and high scalability in big data environment. Design/methodology/approach – First, the integration of Apache Hive, Cloudera Impala and BDAS Shark make the platform support SQL-like query. Next, users can access a single interface and select the best performance of big data warehouse platform automatically by the proposed optimizer. Finally, the distributed memory storage system Memcached incorporated into the distributed file system, Apache HDFS, is employed for fast caching query results. Therefore, if users query the same SQL command, the same result responds rapidly from the cache system instead of suffering the repeated searches in a big data warehouse and taking a longer time to retrieve. Findings – As a result the proposed approach significantly improves the overall performance and dramatically reduces the search time as querying a database, especially applying for the high-repeatable SQL commands under multi-user mode. Research limitations/implications – Currently, Shark’s latest stable version 0.9.1 does not support the latest versions of Spark and Hive. In addition, this series of software only supports Oracle JDK7. Using Oracle JDK8 or Open JDK will cause serious errors, and some software will be unable to run. Practical implications – The problem with this system is that some blocks are missing when too many blocks are stored in one result (about 100,000 records). Another problem is that the sequential writing into In-memory cache wastes time. Originality/value – When the remaining memory capacity is 2 GB or less on each server, Impala and Shark will have a lot of page swapping, causing extremely low performance. When the data scale is larger, it may cause the JVM I/O exception and make the program crash. However, when the remaining memory capacity is sufficient, Shark is faster than Hive and Impala. Impala’s consumption of memory resources is between those of Shark and Hive. This amount of remaining memory is sufficient for Impala’s maximum performance. In this study, each server allocates 20 GB of memory for cluster computing and sets the amount of remaining memory as Level 1: 3 percent (0.6 GB), Level 2: 15 percent (3 GB) and Level 3: 75 percent (15 GB) as the critical points. The program automatically selects Hive when memory is less than 15 percent, Impala at 15 to 75 percent and Shark at more than 75 percent.

Big Data Warehouse

2020 ◽  
Vol 12 (1) ◽  
pp. 1-24
Author(s):  
Khaled Dehdouh ◽  
Omar Boussaid ◽  
Fadila Bentayeb
Keyword(s):  
Big Data ◽  
Data Warehouse ◽  
Database System ◽  
Massive Data ◽  
Data Warehouses ◽  
Online Analysis ◽  
Storage Model ◽  
Nosql Database ◽  
Big Data Warehouse ◽  
Oriented Approach

In the Big Data warehouse context, a column-oriented NoSQL database system is considered as the storage model which is highly adapted to data warehouses and online analysis. Indeed, the use of NoSQL models allows data scalability easily and the columnar store is suitable for storing and managing massive data, especially for decisional queries. However, the column-oriented NoSQL DBMS do not offer online analysis operators (OLAP). To build OLAP cubes corresponding to the analysis contexts, the most common way is to integrate other software such as HIVE or Kylin which has a CUBE operator to build data cubes. By using that, the cube is built according to the row-oriented approach and does not allow to fully obtain the benefits of a column-oriented approach. In this article, the focus is to define a cube operator called MC-CUBE (MapReduce Columnar CUBE), which allows building columnar NoSQL cubes according to the columnar approach by taking into account the non-relational and distributed aspects when data warehouses are stored.

Sign in / Sign up

Export Citation Format

Share Document