Significance of hierarchical and partitioning based clustering in grouping aware data placement for data intensive applications

2017 ◽  
Author(s):  
S. Vengadeswaran ◽  
S. R. Balasundaram
Keyword(s):  
Data Placement ◽  
Data Intensive ◽  
Data Intensive Applications

scholarly journals Significance of Hierarchical and Markov Clustering in Grouping Aware Data Placement for Data Intensive Applications Having Interest Locality

2018 ◽  
Vol 19 (3) ◽  
pp. 245-258
Author(s):  
Vengadeswaran Shanmugasundaram ◽  
Balasundaram Sadhu Ramakrishnan
Keyword(s):  
Big Data ◽  
Data Placement ◽  
Query Execution ◽  
Access Pattern ◽  
Clustering Techniques ◽  
Data Intensive ◽  
Markov Clustering ◽  
Default Data ◽  
Data Intensive Applications ◽  
Grouping Behavior

In this data era, massive volumes of data are being generated every second in variety of domains such as Geoscience, Social Web, Finance, e-Commerce, Health Care, Climate modelling, Physics, Astronomy, Government sectors etc. Hadoop has been well-recognized as de factobig data processing platform that have been extensively adopted, and is currently widely used, in many application domains processing Big Data. Even though it is considered as an efficient solution for such complex query processing, it has its own limitation when the data to be processed exhibit interest locality. The data required for any query execution follows grouping behavior wherein only a part of the Big-Data is accessed frequently. During such scenarion, the time taken to execute a queryand return results, increases exponentially as the amount of data increases leading to much waiting time for the user. Since Hadoop default data placement strategy (HDDPS) does not consider such grouping behavior, it does not perform efficiently resulting in lacunas such as decreased local map task execution, increased query execution time etc. Hence proposed an Optimal Data Placement Strategy (ODPS) based on grouping semantics. In this paper we experiment the significance oftwo most promising clustering techniques viz. Hierarchical Agglomerative Clustering (HAC) and Markov Clustering (MCL) in grouping aware data placement for data intensive applications having interest locality. Initially user access pattern is identified by dynamically analyzing history log.Then both clustering techniques (HAC & MCL) are separately applied over the access pattern to obtain independent clusters. These clusters are interpreted and validated to extract the Optimal Data Groupings (ODG). Finally proposed strategy reorganizes the default data layouts in HDFSbased on ODG to achieve maximum parallel execution per group subjective to Load Balancer and Rack Awareness. Our proposed strategy is tested in 10 node cluster placed in a multi rack with Hadoop installed in every node deployed in cloud platform. Proposed strategy reduces the query execution time, significantly improves the data locality and has proved to be more efficient for massive datasets processing in heterogeneous distributed environment. Also MCL shows a marginal improved performance over HAC for queries exhibiting interest localities.

scholarly journals Heuristic Data Placement for Data-Intensive Applications in Heterogeneous Cloud

2016 ◽  
Vol 2016 ◽  
pp. 1-8 ◽  
Author(s):  
Qing Zhao ◽  
Congcong Xiong ◽  
Peng Wang
Keyword(s):  
Clustering Algorithm ◽  
Recursive Partitioning ◽  
Data Placement ◽  
Data Intensive ◽  
High Bandwidth ◽  
Tree Data ◽  
Placement Algorithm ◽  
Heterogeneous Cloud ◽  
The Cost ◽  
Data Intensive Applications

Data placement is an important issue which aims at reducing the cost of internode data transfers in cloud especially for data-intensive applications, in order to improve the performance of the entire cloud system. This paper proposes an improved data placement algorithm for heterogeneous cloud environments. In the initialization phase, a data clustering algorithm based on data dependency clustering and recursive partitioning has been presented, and both the factor of data size and fixed position are incorporated. And then a heuristic tree-to-tree data placement strategy is advanced in order to make frequent data movements occur on high-bandwidth channels. Simulation results show that, compared with two classical strategies, this strategy can effectively reduce the amount of data transmission and its time consumption during execution.

scholarly journals A Data Placement Algorithm for Data Intensive Applications in Cloud

2016 ◽  
Vol 9 (2) ◽  
pp. 145-156 ◽  
Author(s):  
Qing Zhao ◽  
Congcong Xiong ◽  
Kunyu Zhang ◽  
Yang Yue ◽  
Jucheng Yang
Keyword(s):  
Data Placement ◽  
Data Intensive ◽  
Placement Algorithm ◽  
Data Intensive Applications
Sign in / Sign up

Export Citation Format

Share Document