scholarly journals A comparative study of Distributed Large Scale Data Mining Algorithms

2020 ◽  
Author(s):  
Isha Sood ◽  
Varsha Sharma
Keyword(s):  
Data Mining ◽  
Large Scale ◽  
Large Data ◽  
Data Sets ◽  
Data Set ◽  
Data Mining Algorithms ◽  
Large Scale Data ◽  
Mapreduce Model ◽  
Mining Algorithms ◽  
Scale Data

Essentially, data mining concerns the computation of data and the identification of patterns and trends in the information so that we might decide or judge. Data mining concepts have been in use for years, but with the emergence of big data, they are even more common. In particular, the scalable mining of such large data sets is a difficult issue that has attached several recent findings. A few of these recent works use the MapReduce methodology to construct data mining models across the data set. In this article, we examine current approaches to large-scale data mining and compare their output to the MapReduce model. Based on our research, a system for data mining that combines MapReduce and sampling is implemented and addressed

scholarly journals Data Mining Algorithms: An Overview

2016 ◽  
Vol 15 (6) ◽  
pp. 6806-6813 ◽  
Author(s):  
Sethunya R Joseph ◽  
Hlomani Hlomani ◽  
Keletso Letsholo
Keyword(s):  
Data Mining ◽  
Large Scale ◽  
Predictive Analytics ◽  
Large Data ◽  
Knowledge Discovery In Databases ◽  
Data Sets ◽  
Data Mining Algorithms ◽  
Data Extract ◽  
Mining Algorithms ◽  
Operational Processes

The research on data mining has successfully yielded numerous tools, algorithms, methods and approaches for handling large amounts of data for various purposeful use and   problem solving. Data mining has become an integral part of many application domains such as data ware housing, predictive analytics, business intelligence, bio-informatics and decision support systems. Prime objective of data mining is to effectively handle large scale data, extract actionable patterns, and gain insightful knowledge. Data mining is part and parcel of knowledge discovery in databases (KDD) process. Success and improved decision making normally depends on how quickly one can discover insights from data. These insights could be used to drive better actions which can be used in operational processes and even predict future behaviour. This paper presents an overview of various algorithms necessary for handling large data sets. These algorithms define various structures and methods implemented to handle big data. The review also discusses the general strengths and limitations of these algorithms. This paper can quickly guide or an eye opener to the data mining researchers on which algorithm(s) to select and apply in solving the problems they will be investigating.

scholarly journals Performance Optimization System for Hadoop and Spark Frameworks

2020 ◽  
Vol 20 (6) ◽  
pp. 5-17
Author(s):  
Hrachya Astsatryan ◽  
Aram Kocharyan ◽  
Daniel Hagimont ◽  
Arthur Lalayan
Keyword(s):  
Performance Optimization ◽  
Large Scale ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Apache Hadoop ◽  
Compression Factor ◽  
Large Scale Data ◽  
Additional Processing ◽  
Mapreduce Model

AbstractThe optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows splitting large data sets into a set of blocks distributed on several machines. Data compression reduces data size and transfer time between disks and memory but requires additional processing. Therefore, finding an optimal tradeoff is a challenge, as a high compression factor may underload Input/Output but overload the processor. The paper aims to present a system enabling the selection of the compression tools and tuning the compression factor to reach the best performance in Apache Hadoop and Spark infrastructures based on simulation analyzes.

Improvement of K-Means Algorithm for Accelerated Big Data Clustering

2021 ◽  
Vol 14 (2) ◽  
pp. 99-119
Author(s):  
Chunqiong Wu ◽  
Bingwen Yan ◽  
Rongrui Yu ◽  
Zhangshu Huang ◽  
Baoqin Yu ◽  
...  
Keyword(s):  
Data Mining ◽  
Data Clustering ◽  
Large Scale ◽  
Rapid Development ◽  
Large Data ◽  
Data Retrieval ◽  
Research Directions ◽  
Large Scale Data ◽  
Rich Information ◽  
Scale Data

With the rapid development of the computer level, especially in recent years, “Internet +,” cloud platforms, etc. have been used in various industries, and various types of data have grown in large quantities. Behind these large amounts of data often contain very rich information, relying on traditional data retrieval and analysis methods, and data management models can no longer meet our needs for data acquisition and management. Therefore, data mining technology has become one of the solutions to how to quickly obtain useful information in today's society. Effectively processing large-scale data clustering is one of the important research directions in data mining. The k-means algorithm is the simplest and most basic method in processing large-scale data clustering. The k-means algorithm has the advantages of simple operation, fast speed, and good scalability in processing large data, but it also often exposes fatal defects in data processing. In view of some defects exposed by the traditional k-means algorithm, this paper mainly improves and analyzes from two aspects.

scholarly journals Workload Optimization by Horizontal Aggregation in SQL for Data Mining Analysis

2021 ◽  
pp. 304-309
Author(s):  
Prasanna M. Rathod ◽  
Prof. Dr. Anjali B. Raut
Keyword(s):  
Data Mining ◽  
Relational Algebra ◽  
Data Migration ◽  
Data Sets ◽  
Data Set ◽  
Application Performance ◽  
Data Mining Algorithms ◽  
Mining Project ◽  
Pivot Methods ◽  
Mining Algorithms

Preparing a data set for analysis is generally the most time consuming task in a data mining project, requiring many complex SQL queries, joining tables, and aggregating columns. Existing SQL aggregations have limitations to prepare data sets because they return one column per aggregated group. In general, a significant manual effort is required to build data sets, where a horizontal layout is required. We propose simple, yet powerful, methods to generate SQL code to return aggregated columns in a horizontal tabular layout, returning a set of numbers instead of one number per row. This new class of functions is called horizontal aggregations. Horizontal aggregations build data sets with a horizontal denormalized layout (e.g., point-dimension, observation variable, instance-feature), which is the standard layout required by most data mining algorithms. We propose three fundamental methods to evaluate horizontal aggregations: ? CASE: Exploiting the programming CASE construct; ? SPJ: Based on standard relational algebra operators (SPJ queries); ? PIVOT: Using the PIVOT operator, which is offered by some DBMSs. Experiments with large tables compare the proposed query evaluation methods. Our CASE method has similar speed to the PIVOT operator and it is much faster than the SPJ method. In general, the CASE and PIVOT methods exhibit linear scalability, whereas the SPJ method does not. For query optimization the distance computation and nearest cluster in the k-means are based on SQL. Workload balancing is the assignment of work to processors in a way that maximizes application performance. The process of load balancing can be generalized into four basic steps: 1. Monitoring processor load and state; 2. Exchanging workload and state information between processors; 3. Decision making; 4. Data migration. The decision phase is triggered when the load imbalance is detected to calculate optimal data redistribution. In the fourth and last phase, data migrates from overloaded processors to under-loaded ones.

ANALISA POLA PEKERJAAN LULUSAN STMIK BUDI DARMA MENERAPKAN METODE C4.5

2018 ◽  
Vol 2 (1) ◽  
Author(s):  
Anisa Anisa ◽  
Mesran Mesran
Keyword(s):  
Data Mining ◽  
Large Scale ◽  
Large Data ◽  
Analysis Data ◽  
Mining Method ◽  
Training Set ◽  
Data Mining Method ◽  
Large Scale Data ◽  
Scale Data

Data mining is mining or discovery information to the process of looking for patterns or information that contains the search trends in a number of very large data in taking decisions on the future.In determining the patterns of classification techniques garnered record (Training set). The class attribute, which is a decision tree with method C 4.5 builds upon an algorithm of induction can be minimised.By utilizing data jobs graduates expected to generate information about interest & talent, work with benefit from graduate quisioner alumni. A pattern of work that sought from large-scale data and analyzed by various algorithms to compute the C 4.5 can do that work based on the pattern of investigation patterns that affect so that it found the rules are interconnected that can result from the results of the classification of objects of different classes or categories of attributes that influence to shape the patterns of work. The application used is software that used Tanagra data mining for academic and research purposes.That contains data mining method explored starting from the data analysis, and classification data mining.Keywords: analysis, Data Mining, method C 4.5, Tanagra, patterns of work

scholarly journals Data mining fool’s gold

2020 ◽  
Vol 35 (3) ◽  
pp. 182-194
Author(s):  
Gary Smith
Keyword(s):  
Data Mining ◽  
Scientific Method ◽  
Large Data ◽  
Learning Systems ◽  
Large Data Sets ◽  
Data Sets ◽  
Computer Algorithms ◽  
Data Mining Algorithms ◽  
Rigorous Testing ◽  
Mining Algorithms

The scientific method is based on the rigorous testing of falsifiable conjectures. Data mining, in contrast, puts data before theory by searching for statistical patterns without being constrained by prespecified hypotheses. Artificial intelligence and machine learning systems, for example, often rely on data-mining algorithms to construct models with little or no human guidance. However, a plethora of patterns are inevitable in large data sets, and computer algorithms have no effective way of assessing whether the patterns they unearth are truly useful or meaningless coincidences. While data mining sometimes discovers useful relationships, the data deluge has caused the number of possible patterns that can be discovered relative to the number that are genuinely useful to grow exponentially—which makes it increasingly likely that what data mining unearths is likely to be fool’s gold.

scholarly journals GeoBoost: An Incremental Deep Learning Approach toward Global Mapping of Buildings from VHR Remote Sensing Images

2020 ◽  
Vol 12 (11) ◽  
pp. 1794
Author(s):  
Naisen Yang ◽  
Hong Tang
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Large Scale ◽  
Semantic Segmentation ◽  
Data Sets ◽  
Data Set ◽  
Large Scale Data ◽  
Learning Tasks ◽  
Global Mapping ◽  
Scale Data

Modern convolutional neural networks (CNNs) are often trained on pre-set data sets with a fixed size. As for the large-scale applications of satellite images, for example, global or regional mappings, these images are collected incrementally by multiple stages in general. In other words, the sizes of training datasets might be increased for the tasks of mapping rather than be fixed beforehand. In this paper, we present a novel algorithm, called GeoBoost, for the incremental-learning tasks of semantic segmentation via convolutional neural networks. Specifically, the GeoBoost algorithm is trained in an end-to-end manner on the newly available data, and it does not decrease the performance of previously trained models. The effectiveness of the GeoBoost algorithm is verified on the large-scale data set of DREAM-B. This method avoids the need for training on the enlarged data set from scratch and would become more effective along with more available data.

scholarly journals LARGE‐SCALE DATA VISUALIZATION WITH MISSING VALUES

2006 ◽  
Vol 12 (1) ◽  
pp. 44-49
Author(s):  
Sergiy Popov
Keyword(s):  
Dimensionality Reduction ◽  
Large Scale ◽  
Missing Values ◽  
Original Data ◽  
Nonlinear Dimensionality Reduction ◽  
Data Sets ◽  
Data Set ◽  
Large Scale Data ◽  
Learning Procedure ◽  
Scale Data

Visualization of large‐scale data inherently requires dimensionality reduction to 1D, 2D, or 3D space. Autoassociative neural networks with a bottleneck layer are commonly used as a nonlinear dimensionality reduction technique. However, many real‐world problems suffer from incomplete data sets, i.e. some values can be missing. Common methods dealing with missing data include the deletion of all cases with missing values from the data set or replacement with mean or “normal” values for specific variables. Such methods are appropriate when just a few values are missing. But in the case when a substantial portion of data is missing, these methods can significantly bias the results of modeling. To overcome this difficulty, we propose a modified learning procedure for the autoassociative neural network that directly takes the missing values into account. The outputs of the trained network may be used for substitution of the missing values in the original data set.

A Prediction System for Diagnosis of Diabetes Mellitus

2020 ◽  
Vol 17 (1) ◽  
pp. 6-9
Author(s):  
Ramya G. Franklin ◽  
B. Muthukumar
Keyword(s):  
Data Mining ◽  
Large Data ◽  
Heterogeneous Data ◽  
Common Disease ◽  
Prediction System ◽  
Health Issues ◽  
Data Set ◽  
Data Mining Algorithms ◽  
Growth Of Science ◽  
Mining Algorithms

The growth of Science is a priceless asset to the human and society. The plethora of high-end machines has made life a sophistication which in turn is paid back as health issues. The health care data are complex and large. This heterogeneous data are used to diagnose patient’s diseases. It is better to predict the diseases at an earlier stage that can save the life and also have an upper hand in controlling the diseases. Data mining approaches are very useful in analyzing the complex, heterogeneous and large data set. The mining algorithms extract the essential data set from the raw data. This paper presents a survey on the various data mining algorithms used in predicting a very common disease in day a today life “Diabetics Mellitus.” Over 246 million people in the world are diabetic with a majority of them being women. The WHO reports that by 2025 this number is expected to rise to over 380 million.

Sign in / Sign up

Export Citation Format

Share Document