An effective data locality aware task scheduling method for MapReduce framework in heterogeneous environments

2011 ◽  
Author(s):  
Xiaohong Zhang ◽  
Yuhong Feng ◽  
Shengzhong Feng ◽  
Jianping Fan ◽  
Zhong Ming
Keyword(s):  
Task Scheduling ◽  
Data Locality ◽  
Heterogeneous Environments ◽  
Mapreduce Framework ◽  
Scheduling Method

scholarly journals Linear Weighted Regression and Energy-Aware Greedy Scheduling for Heterogeneous Big Data

Electronics ◽  
2021 ◽  
Vol 10 (5) ◽  
pp. 554
Author(s):  
Suresh Kallam ◽  
Rizwan Patan ◽  
Tathapudi V. Ramana ◽  
Amir H. Gandomi
Keyword(s):  
Big Data ◽  
Energy Consumption ◽  
Total Energy ◽  
Optimal Solution ◽  
Data Locality ◽  
Weighted Regression ◽  
Heterogeneous Environments ◽  
Energy Aware ◽  
Total Energy Consumption ◽  
Greedy Scheduling

Data are presently being produced at an increased speed in different formats, which complicates the design, processing, and evaluation of the data. The MapReduce algorithm is a distributed file system that is used for big data parallel processing. Current implementations of MapReduce assist in data locality along with robustness. In this study, a linear weighted regression and energy-aware greedy scheduling (LWR-EGS) method were combined to handle big data. The LWR-EGS method initially selects tasks for an assignment and then selects the best available machine to identify an optimal solution. With this objective, first, the problem was modeled as an integer linear weighted regression program to choose tasks for the assignment. Then, the best available machines were selected to find the optimal solution. In this manner, the optimization of resources is said to have taken place. Then, an energy efficiency-aware greedy scheduling algorithm was presented to select a position for each task to minimize the total energy consumption of the MapReduce job for big data applications in heterogeneous environments without a significant performance loss. To evaluate the performance, the LWR-EGS method was compared with two related approaches via MapReduce. The experimental results showed that the LWR-EGS method effectively reduced the total energy consumption without producing large scheduling overheads. Moreover, the method also reduced the execution time when compared to state-of-the-art methods. The LWR-EGS method reduced the energy consumption, average processing time, and scheduling overhead by 16%, 20%, and 22%, respectively, compared to existing methods.

scholarly journals TMaR: a two-stage MapReduce scheduler for heterogeneous environments

2020 ◽  
Vol 10 (1) ◽  
Author(s):  
Neda Maleki ◽  
Hamid Reza Faragardi ◽  
Amir Masoud Rahmani ◽  
Mauro Conti ◽  
Jay Lofstead
Keyword(s):  
Task Scheduling ◽  
Network Traffic ◽  
Input Data ◽  
Power Reduction ◽  
Heterogeneous Environments ◽  
Two Stage ◽  
Finish Time ◽  
Remote Storage ◽  
Simulation Results

Abstract In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.

scholarly journals Data-Intensive Task Scheduling for Heterogeneous Big Data Analytics in IoT System

Energies ◽  
2020 ◽  
Vol 13 (17) ◽  
pp. 4508
Author(s):  
Xin Li ◽  
Liangyuan Wang ◽  
Jemal H. Abawajy ◽  
Xiaolin Qin ◽  
Giovanni Pau ◽  
...  
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Task Scheduling ◽  
Execution Time ◽  
Data Centers ◽  
Big Data Analysis ◽  
Data Locality ◽  
Data Migration ◽  
Task Execution ◽  
Task Execution Time

Efficient big data analysis is critical to support applications or services in Internet of Things (IoT) system, especially for the time-intensive services. Hence, the data center may host heterogeneous big data analysis tasks for multiple IoT systems. It is a challenging problem since the data centers usually need to schedule a large number of periodic or online tasks in a short time. In this paper, we investigate the heterogeneous task scheduling problem to reduce the global task execution time, which is also an efficient method to reduce energy consumption for data centers. We establish the task execution for heterogeneous tasks respectively based on the data locality feature, which also indicate the relationship among the tasks, data blocks and servers. We propose a heterogeneous task scheduling algorithm with data migration. The core idea of the algorithm is to maximize the efficiency by comparing the cost between remote task execution and data migration, which could improve the data locality and reduce task execution time. We conduct extensive simulations and the experimental results show that our algorithm has better performance than the traditional methods, and data migration actually works to reduce th overall task execution time. The algorithm also shows acceptable fairness for the heterogeneous tasks.

scholarly journals A Multi-Objective Task Scheduling Method for Fog Computing in Cyber-Physical-Social Services

IEEE Access ◽  
2020 ◽  
Vol 8 ◽  
pp. 65085-65095
Author(s):  
Ming Yang ◽  
Hao Ma ◽  
Shuang Wei ◽  
You Zeng ◽  
Yefeng Chen ◽  
...  
Keyword(s):  
Social Services ◽  
Task Scheduling ◽  
Fog Computing ◽  
Multi Objective ◽  
Scheduling Method
Sign in / Sign up

Export Citation Format

Share Document