scholarly journals Heterogeneous Distributed Big Data Clustering on Sparse Grids

Algorithms ◽  
2019 ◽  
Vol 12 (3) ◽  
pp. 60 ◽  
Author(s):  
David Pfander ◽  
Gregor Daiß ◽  
Dirk Pflüger
Keyword(s):  
Big Data ◽  
Density Estimation ◽  
High Performance ◽  
Clustering Algorithm ◽  
Sparse Grid ◽  
Peak Performance ◽  
Density Based Clustering ◽  
Wide Range ◽  
Data Points ◽  
And Performance

Clustering is an important task in data mining that has become more challenging due to the ever-increasing size of available datasets. To cope with these big data scenarios, a high-performance clustering approach is required. Sparse grid clustering is a density-based clustering method that uses a sparse grid density estimation as its central building block. The underlying density estimation approach enables the detection of clusters with non-convex shapes and without a predetermined number of clusters. In this work, we introduce a new distributed and performance-portable variant of the sparse grid clustering algorithm that is suited for big data settings. Our computed kernels were implemented in OpenCL to enable portability across a wide range of architectures. For distributed environments, we added a manager–worker scheme that was implemented using MPI. In experiments on two supercomputers, Piz Daint and Hazel Hen, with up to 100 million data points in a ten-dimensional dataset, we show the performance and scalability of our approach. The dataset with 100 million data points was clustered in 1198 s using 128 nodes of Piz Daint. This translates to an overall performance of 352 TFLOPS . On the node-level, we provide results for two GPUs, Nvidia’s Tesla P100 and the AMD FirePro W8100, and one processor-based platform that uses Intel Xeon E5-2680v3 processors. In these experiments, we achieved between 43% and 66% of the peak performance across all computed kernels and devices, demonstrating the performance portability of our approach.

scholarly journals Heterogeneous Distributed Big Data Clustering on Sparse Grids

2019 ◽  
Author(s):  
David Pfander ◽  
Gregor Daiß ◽  
Dirk Pflüger
Keyword(s):  
Big Data ◽  
Density Estimation ◽  
High Performance ◽  
Clustering Algorithm ◽  
Sparse Grid ◽  
Peak Performance ◽  
Density Based Clustering ◽  
Wide Range ◽  
Data Points ◽  
And Performance

Clustering is an important task in data mining that has become more challenging due to the ever-increasing size of available datasets. To cope with these big data scenarios, a high-performance clustering approach is required. Sparse grid clustering is a density-based clustering method that uses a sparse grid density estimation as its central building block. The underlying density estimation approach enables the detection of clusters with non-convex shapes and without a predetermined number of clusters. In this work, we introduce a new distributed and performance-portable variant of the sparse grid clustering algorithm that is suited for big data settings. Our compute kernels were implemented in OpenCL to enable portability across a wide range of architectures. For distributed environments, we added a manager-worker scheme that was implemented using MPI. In experiments on two supercomputers, Piz Daint and Hazel Hen, with up to 100 million data points in a 10-dimensional dataset, we show the performance and scalability of our approach. The dataset with 100 million data points was clustered in 1198s using 128 nodes of Piz Daint. This translates to an overall performance of 352TFLOPS. On the node-level, we provide results for two GPUs, Nvidia's Tesla P100 and the AMD FirePro W8100, and one processor-based platform that uses Intel Xeon E5-2680v3 processors. In these experiments, we achieved between 43% and 66% of the peak performance across all compute kernels and devices, demonstrating the performance portability of our approach.

scholarly journals Task-based programming in COMPSs to converge from HPC to big data

2017 ◽  
Vol 32 (1) ◽  
pp. 45-60 ◽  
Author(s):  
Javier Conejero ◽  
Sandra Corella ◽  
Rosa M Badia ◽  
Jesus Labarta
Keyword(s):  
Big Data ◽  
High Performance ◽  
Programming Model ◽  
Good Alternative ◽  
Programming Models ◽  
Suitable Model ◽  
Advantages And Disadvantages ◽  
Big Data Applications ◽  
And Performance ◽  
The Right

Task-based programming has proven to be a suitable model for high-performance computing (HPC) applications. Different implementations have been good demonstrators of this fact and have promoted the acceptance of task-based programming in the OpenMP standard. Furthermore, in recent years, Apache Spark has gained wide popularity in business and research environments as a programming model for addressing emerging big data problems. COMP Superscalar (COMPSs) is a task-based environment that tackles distributed computing (including Clouds) and is a good alternative for a task-based programming model for big data applications. This article describes why we consider that task-based programming models are a good approach for big data applications. The article includes a comparison of Spark and COMPSs in terms of architecture, programming model, and performance. It focuses on the differences that both frameworks have in structural terms, on their programmability interface, and in terms of their efficiency by means of three widely known benchmarking kernels: Wordcount, Kmeans, and Terasort. These kernels enable the evaluation of the more important functionalities of both programming models and analyze different work flows and conditions. The main results achieved from this comparison are (1) COMPSs is able to extract the inherent parallelism from the user code with minimal coding effort as opposed to Spark, which requires the existing algorithms to be adapted and rewritten by explicitly using their predefined functions, (2) it is an improvement in terms of performance when compared with Spark, and (3) COMPSs has shown to scale better than Spark in most cases. Finally, we discuss the advantages and disadvantages of both frameworks, highlighting the differences that make them unique, thereby helping to choose the right framework for each particular objective.

scholarly journals Design of a Clutchless Hybrid Transmission for a High-Performance Vehicle

2015 ◽  
Author(s):  
Chad L. Jacoby ◽  
Young Suk Jo ◽  
Jake Jurewicz ◽  
Guillermo Pamanes ◽  
Joshua E. Siegel ◽  
...  
Keyword(s):  
High Performance ◽  
Power Flow ◽  
Combustion Engine ◽  
Performance Requirements ◽  
Wide Range ◽  
Friction Clutch ◽  
Hybrid Transmission ◽  
And Performance ◽  
Electrical Motors ◽  
Operational Modes

There exists the potential for major simplifications to current hybrid transmission architectures, which can lead to advances in powertrain performance. This paper assesses the technical merits of various hybrid powertrains in the context of high-performance vehicles and introduces a new transmission concept targeted at high performance hybrid applications. While many hybrid transmission configurations have been developed and implemented in mainstream and even luxury vehicles, ultra high performance sports cars have only recently begun to hybridize. The unique performance requirements of such vehicles place novel constraints on their transmissions designs. The goals become less about improved efficiency and smoothness and more centered on weight reduction, complexity reduction, and performance improvement. To identify the most critical aspects of a high performance transmission, a wide range of existing technologies is studied in concert with basic physical performance analysis of electrical motors and an internal combustion engine. The new transmission concepts presented here emphasize a reduction in inertial, frictional, and mechanical losses. A series of conceptual powertrain designs are evaluated against the goals of reducing mechanical complexity and maintaining functionality. The major innovation in these concepts is the elimination of a friction clutch to engage and disengage gears. Instead, the design proposes that the inclusion of a large electric motor enables the gears to be speed-matched and torque-zeroed without the inherent losses associated with a friction clutch. Additionally, these transmission concepts explore the merits of multiple electric motors and their placement as well as the reduction in synchronization interfaces. Ultimately, two strategies for speed-matched gear sets are considered, and a speed-matching prototype of the chosen methodology is presented to validate the feasibility of the proposed concept. The power flow and operational modes of both transmission architectures are studied to ensure required functionality and identify further areas of optimization. While there are still many unanswered questions about this concept, this paper introduces the base analysis and proof of concept for a technology that has great potential to advance hybrid vehicles at all levels.

scholarly journals The timing mega-study: comparing a range of experiment generators, both lab-based and online

PeerJ ◽  
2020 ◽  
Vol 8 ◽  
pp. e9414 ◽  
Author(s):  
David Bridges ◽  
Alain Pitiot ◽  
Michael R. MacAskill ◽  
Jonathan W. Peirce
Keyword(s):  
High Performance ◽  
Response Times ◽  
Behavioral Experiments ◽  
Research Software ◽  
Software Packages ◽  
Psychophysics Toolbox ◽  
Wide Range ◽  
Stimulus Timing ◽  
And Performance ◽  
Online Methods

Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.

scholarly journals A weighting system to build physical layer measurements maps by crowdsourcing data from smartphones

2020 ◽  
Vol 9 (3) ◽  
pp. 211
Author(s):  
Hosam Alamleh ◽  
Ali Abdullah S. AlQahtani
Keyword(s):  
Mobile Devices ◽  
Access Point ◽  
Physical Layer ◽  
Network Coverage ◽  
Wide Range ◽  
Different Types ◽  
Radio Signals ◽  
Data Points ◽  
And Performance ◽  
Map Accuracy

<p>Mobile devices can sense different types of radio signals. For example, broadcast signals. These broadcasted signals allow the device to establish a connection to the access point broadcasting it. Moreover, mobile devices can record different physical layer measurements. These measurements are an indication of the service quality at the point they were collected. These measurements data can be aggregated to form physical layer measurement maps. These maps are useful for several applications such as location fixing, navigation, access control, and evaluating network coverage and performance. Crowdsourcing can be an efficient way to create such maps. However, users in a crowdsourcing application tend to have different devices with different capabilities, which might impact the overall accuracy of the generated maps. In this paper, we propose a method to build physical layer measurements maps by crowdsourcing physical layer measurements, GPS locations, from participating mobile devices. The proposed system gives different weights to each data point provided by the participating devices based on the data source’s trustworthiness. Our tests showed that the different models of mobile devices return GPS location with different location accuracies. Consequently, when building the physical layer measurements maps our algorithm assigns a higher weight to data points coming from devices with higher GPS location accuracy. This allows accommodating a wide range of mobile devices with different capabilities in crowdsourcing applications. An experiment and a simulation were performed to test the proposed method. The results showed improvement in crowdsourced map accuracy when the proposed method is implemented.</p>

Recognition and labeling of faults in wind turbines with a density-based clustering algorithm

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Shuai Luo ◽  
Hongwei Liu ◽  
Ershi Qi
Keyword(s):  
Wind Turbines ◽  
Clustering Algorithm ◽  
Support Vector ◽  
Scanning Strategy ◽  
Data Set ◽  
Content Type ◽  
Vibration Data ◽  
Density Based Clustering ◽  
Extreme Gradient Boosting ◽  
Data Points

PurposeThe purpose of this paper is to recognize and label the faults in wind turbines with a new density-based clustering algorithm, named contour density scanning clustering (CDSC) algorithm.Design/methodology/approachThe algorithm includes four components: (1) computation of neighborhood density, (2) selection of core and noise data, (3) scanning core data and (4) updating clusters. The proposed algorithm considers the relationship between neighborhood data points according to a contour density scanning strategy.FindingsThe first experiment is conducted with artificial data to validate that the proposed CDSC algorithm is suitable for handling data points with arbitrary shapes. The second experiment with industrial gearbox vibration data is carried out to demonstrate that the time complexity and accuracy of the proposed CDSC algorithm in comparison with other conventional clustering algorithms, including k-means, density-based spatial clustering of applications with noise, density peaking clustering, neighborhood grid clustering, support vector clustering, random forest, core fusion-based density peak clustering, AdaBoost and extreme gradient boosting. The third experiment is conducted with an industrial bearing vibration data set to highlight that the CDSC algorithm can automatically track the emerging fault patterns of bearing in wind turbines over time.Originality/valueData points with different densities are clustered using three strategies: direct density reachability, density reachability and density connectivity. A contours density scanning strategy is proposed to determine whether the data points with the same density belong to one cluster. The proposed CDSC algorithm achieves automatically clustering, which means that the trends of the fault pattern could be tracked.

Correlative Density-Based Clustering

2016 ◽  
Vol 13 (10) ◽  
pp. 6935-6943 ◽  
Author(s):  
Jia-Lin Hua ◽  
Jian Yu ◽  
Miin-Shen Yang
Keyword(s):  
Correlation Analysis ◽  
Clustering Algorithm ◽  
Clustering Methods ◽  
Data Set ◽  
Density Based Clustering ◽  
Inherent Structure ◽  
Data Points ◽  
Artificial Datasets

Mountains, which heap up by densities of a data set, intuitively reflect the structure of data points. These mountain clustering methods are useful for grouping data points. However, the previous mountain-based clustering suffers from the choice of parameters which are used to compute the density. In this paper, we adopt correlation analysis to determine the density, and propose a new clustering algorithm, called Correlative Density-based Clustering (CDC). The new algorithm computes the density with a modified way and determines the parameters based on the inherent structure of data points. Experiments on artificial datasets and real datasets demonstrate the simplicity and effectiveness of the proposed approach.

Big Data Clustering with Kernel k-Means: Resources, Time and Performance

2018 ◽  
Vol 27 (04) ◽  
pp. 1860006
Author(s):  
Nikolaos Tsapanos ◽  
Anastasios Tefas ◽  
Nikolaos Nikolaidis ◽  
Ioannis Pitas
Keyword(s):  
Big Data ◽  
Data Clustering ◽  
Clustering Algorithm ◽  
Learning Task ◽  
Related Data ◽  
Clustering Problem ◽  
Processing Power ◽  
Trade Offs ◽  
Separable Kernel ◽  
And Performance

Data clustering is an unsupervised learning task that has found many applications in various scientific fields. The goal is to find subgroups of closely related data samples (clusters) in a set of unlabeled data. A classic clustering algorithm is the so-called k-Means. It is very popular, however, it is also unable to handle cases in which the clusters are not linearly separable. Kernel k-Means is a state of the art clustering algorithm, which employs the kernel trick, in order to perform clustering on a higher dimensionality space, thus overcoming the limitations of classic k-Means regarding the non-linear separability of the input data. With respect to the challenges of Big Data research, a field that has established itself in the last few years and involves performing tasks on extremely large amounts of data, several adaptations of the Kernel k-Means have been proposed, each of which has different requirements in processing power and running time, while also incurring different trade-offs in performance. In this paper, we present several issues and techniques involving the usage of Kernel k-Means for Big Data clustering and how the combination of each component in a clustering framework fares in terms of resources, time and performance. We use experimental results, in order to evaluate several combinations and provide a recommendation on how to approach a Big Data clustering problem.

scholarly journals The timing mega-study: comparing a range of experiment generators, both lab-based and online

2020 ◽  
Author(s):  
David Bridges ◽  
Alain Pitiot ◽  
Michael R. MacAskill ◽  
Jonathan Westley Peirce
Keyword(s):  
Operating Systems ◽  
High Performance ◽  
Response Times ◽  
Research Software ◽  
Software Packages ◽  
Psychophysics Toolbox ◽  
Wide Range ◽  
Stimulus Timing ◽  
Behavioural Experiments ◽  
And Performance

Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioural experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, MacOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that Mac OS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on a number of browser configurations. For response times (using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between operating systems and browsers, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result.The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.

scholarly journals Mental Readiness for Frontline Workers in Homelessness Services in Ottawa, Canada

2021 ◽  
pp. 1-25
Author(s):  
Judy McDonald ◽  
Katherine Hale
Keyword(s):  
High Performance ◽  
Peak Performance ◽  
Post Traumatic Stress ◽  
Think Tank ◽  
Frontline Workers ◽  
Air Traffic Controllers ◽  
Post Traumatic ◽  
And Performance ◽  
Distraction Control ◽  
First Time

This study investigated factors related to competency by assessing the mental readiness among highly recognized frontline workers in homelessness services (FWHSs) by means of self-completed questionnaires. A total of 35 highly respected FWHSs in Ottawa, Canada were identified by their peers and supervisors as “exceptional” for various specialty areas: addictions, mental health, hoarding, trauma and post-traumatic stress disorder (PTSD). An Operational Readiness Framework was used to examine how FWHSs perform at their best in challenging situations. A series of questionnaires were completed at a Think Tank to determine their mental readiness before, during and after challenging situations. Quantitative and qualitative analyses of mental readiness were performed to prioritize identified challenges. The study findings were then compared to the “Wheel of Excellence” based on results from elite athletes and other high performers such as surgeons, police, and air traffic controllers. The analysis revealed that mental readiness is required to achieve peak performance in addressing the challenges of homelessness. The balance between readiness (physical, technical and mental) and performance contributed to their competency and resiliency. Common elements of success were found: commitment, self-belief, positive imagery, mental preparation, full focus, distraction control and constructive evaluation. This investigation confirmed many similarities in mental readiness practices engaged by excellent FWHSs and other top professionals. This study offered, for the first time, a comprehensive understanding of specific high-performance readiness practices through a streetwise, frontline-worker perspective. Practical recommendations for training and assessment were provided relevant to excellence in homelessness services.

Sign in / Sign up

Export Citation Format

Share Document