scholarly journals Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off

Entropy ◽  
2020 ◽  
Vol 22 (5) ◽  
pp. 544 ◽  
Author(s):  
Emre Ozfatura ◽  
Sennur Ulukus ◽  
Deniz Gündüz
Keyword(s):  
Machine Learning ◽  
Gradient Descent ◽  
Large Scale ◽  
Computation Time ◽  
Distributed Learning ◽  
Trade Off ◽  
Communication Latency ◽  
Advantages And Disadvantages ◽  
Machine Learning Applications ◽  
Coded Communication

When gradient descent (GD) is scaled to many parallel workers for large-scale machine learning applications, its per-iteration computation time is limited by straggling workers. Straggling workers can be tolerated by assigning redundant computations and/or coding across data and computations, but in most existing schemes, each non-straggling worker transmits one message per iteration to the parameter server (PS) after completing all its computations. Imposing such a limitation results in two drawbacks: over-computation due to inaccurate prediction of the straggling behavior, and under-utilization due to discarding partial computations carried out by stragglers. To overcome these drawbacks, we consider multi-message communication (MMC) by allowing multiple computations to be conveyed from each worker per iteration, and propose novel straggler avoidance techniques for both coded computation and coded communication with MMC. We analyze how the proposed designs can be employed efficiently to seek a balance between the computation and communication latency. Furthermore, we identify the advantages and disadvantages of these designs in different settings through extensive simulations, both model-based and real implementation on Amazon EC2 servers, and demonstrate that proposed schemes with MMC can help improve upon existing straggler avoidance schemes.

scholarly journals Statistical and machine learning models for optimizing energy in parallel applications

2019 ◽  
Vol 33 (6) ◽  
pp. 1079-1097 ◽  
Author(s):  
Mark Endrei ◽  
Chao Jin ◽  
Minh Ngoc Dinh ◽  
David Abramson ◽  
Heidi Poxon ◽  
...  
Keyword(s):  
Machine Learning ◽  
Energy Efficiency ◽  
High Performance ◽  
Large Scale ◽  
Energy Use ◽  
Parallel Applications ◽  
Learning Models ◽  
Trade Off ◽  
Time Required ◽  
Machine Learning Models

Rising power costs and constraints are driving a growing focus on the energy efficiency of high performance computing systems. The unique characteristics of a particular system and workload and their effect on performance and energy efficiency are typically difficult for application users to assess and to control. Settings for optimum performance and energy efficiency can also diverge, so we need to identify trade-off options that guide a suitable balance between energy use and performance. We present statistical and machine learning models that only require a small number of runs to make accurate Pareto-optimal trade-off predictions using parameters that users can control. We study model training and validation using several parallel kernels and more complex workloads, including Algebraic Multigrid (AMG), Large-scale Atomic Molecular Massively Parallel Simulator, and Livermore Unstructured Lagrangian Explicit Shock Hydrodynamics. We demonstrate that we can train the models using as few as 12 runs, with prediction error of less than 10%. Our AMG results identify trade-off options that provide up to 45% improvement in energy efficiency for around 10% performance loss. We reduce the sample measurement time required for AMG by 90%, from 13 h to 74 min.

scholarly journals Clinician checklist for assessing suitability of machine learning applications in healthcare

2021 ◽  
Vol 28 (1) ◽  
pp. e100251
Author(s):  
Ian Scott ◽  
Stacey Carter ◽  
Enrico Coiera
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Clinical Decision Making ◽  
Improve Patient Care ◽  
Clinical Decision ◽  
Routine Care ◽  
Machine Learning Algorithms ◽  
Clinical Settings ◽  
Machine Learning Applications ◽  
Key Issues

Machine learning algorithms are being used to screen and diagnose disease, prognosticate and predict therapeutic responses. Hundreds of new algorithms are being developed, but whether they improve clinical decision making and patient outcomes remains uncertain. If clinicians are to use algorithms, they need to be reassured that key issues relating to their validity, utility, feasibility, safety and ethical use have been addressed. We propose a checklist of 10 questions that clinicians can ask of those advocating for the use of a particular algorithm, but which do not expect clinicians, as non-experts, to demonstrate mastery over what can be highly complex statistical and computational concepts. The questions are: (1) What is the purpose and context of the algorithm? (2) How good were the data used to train the algorithm? (3) Were there sufficient data to train the algorithm? (4) How well does the algorithm perform? (5) Is the algorithm transferable to new clinical settings? (6) Are the outputs of the algorithm clinically intelligible? (7) How will this algorithm fit into and complement current workflows? (8) Has use of the algorithm been shown to improve patient care and outcomes? (9) Could the algorithm cause patient harm? and (10) Does use of the algorithm raise ethical, legal or social concerns? We provide examples where an algorithm may raise concerns and apply the checklist to a recent review of diagnostic imaging applications. This checklist aims to assist clinicians in assessing algorithm readiness for routine care and identify situations where further refinement and evaluation is required prior to large-scale use.

Distributed learning with indefinite kernels

2019 ◽  
Vol 17 (06) ◽  
pp. 947-975 ◽  
Author(s):  
Lei Shi
Keyword(s):  
Large Scale ◽  
Substantial Reduction ◽  
Computation Time ◽  
Distributed Learning ◽  
Rates Of Convergence ◽  
Regression Problem ◽  
Data Set ◽  
Regularization Scheme ◽  
Original Algorithm ◽  
Indefinite Kernel

We investigate the distributed learning with coefficient-based regularization scheme under the framework of kernel regression methods. Compared with the classical kernel ridge regression (KRR), the algorithm under consideration does not require the kernel function to be positive semi-definite and hence provides a simple paradigm for designing indefinite kernel methods. The distributed learning approach partitions a massive data set into several disjoint data subsets, and then produces a global estimator by taking an average of the local estimator on each data subset. Easy exercisable partitions and performing algorithm on each subset in parallel lead to a substantial reduction in computation time versus the standard approach of performing the original algorithm on the entire samples. We establish the first mini-max optimal rates of convergence for distributed coefficient-based regularization scheme with indefinite kernels. We thus demonstrate that compared with distributed KRR, the concerned algorithm is more flexible and effective in regression problem for large-scale data sets.

scholarly journals Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches

2018 ◽  
Vol 2018 ◽  
pp. 1-13 ◽  
Author(s):  
Prabal Poudel ◽  
Alfredo Illanes ◽  
Debdoot Sheet ◽  
Michael Friebe
Keyword(s):  
Machine Learning ◽  
Active Contours ◽  
Computation Time ◽  
Machine Intelligence ◽  
Ease Of Use ◽  
Human Interaction ◽  
Ultrasound Images ◽  
Learning Approaches ◽  
Thyroid State ◽  
Advantages And Disadvantages

The thyroid is one of the largest endocrine glands in the human body, which is involved in several body mechanisms like controlling protein synthesis and the body's sensitivity to other hormones and use of energy sources. Hence, it is of prime importance to track the shape and size of thyroid over time in order to evaluate its state. Thyroid segmentation and volume computation are important tools that can be used for thyroid state tracking assessment. Most of the proposed approaches are not automatic and require long time to correctly segment the thyroid. In this work, we compare three different nonautomatic segmentation algorithms (i.e., active contours without edges, graph cut, and pixel-based classifier) in freehand three-dimensional ultrasound imaging in terms of accuracy, robustness, ease of use, level of human interaction required, and computation time. We figured out that these methods lack automation and machine intelligence and are not highly accurate. Hence, we implemented two machine learning approaches (i.e., random forest and convolutional neural network) to improve the accuracy of segmentation as well as provide automation. This comparative study intends to discuss and analyse the advantages and disadvantages of different algorithms. In the last step, the volume of the thyroid is computed using the segmentation results, and the performance analysis of all the algorithms is carried out by comparing the segmentation results with the ground truth.

scholarly journals The Trials and Tribulations of Assembling Large Medical Imaging Datasets for Machine Learning Applications

2021 ◽  
Author(s):  
Kirti Magudia ◽  
Christopher P. Bridge ◽  
Katherine P. Andriole ◽  
Michael H. Rosenthal
Keyword(s):  
Machine Learning ◽  
Medical Imaging ◽  
Data Storage ◽  
Large Scale ◽  
Large Datasets ◽  
Abdominal Ct ◽  
Institutional Review ◽  
Retrospective Clinical Study ◽  
Primary Investigator ◽  
Machine Learning Applications

AbstractWith vast interest in machine learning applications, more investigators are proposing to assemble large datasets for machine learning applications. We aim to delineate multiple possible roadblocks to exam retrieval that may present themselves and lead to significant time delays. This HIPAA-compliant, institutional review board–approved, retrospective clinical study required identification and retrieval of all outpatient and emergency patients undergoing abdominal and pelvic computed tomography (CT) at three affiliated hospitals in the year 2012. If a patient had multiple abdominal CT exams, the first exam was selected for retrieval (n=23,186). Our experience in attempting to retrieve 23,186 abdominal CT exams yielded 22,852 valid CT abdomen/pelvis exams and identified four major categories of challenges when retrieving large datasets: cohort selection and processing, retrieving DICOM exam files from PACS, data storage, and non-recoverable failures. The retrieval took 3 months of project time and at minimum 300 person-hours of time between the primary investigator (a radiologist), a data scientist, and a software engineer. Exam selection and retrieval may take significantly longer than planned. We share our experience so that other investigators can anticipate and plan for these challenges. We also hope to help institutions better understand the demands that may be placed on their infrastructure by large-scale medical imaging machine learning projects.

scholarly journals MS2AI: Automated repurposing of public peptide LC-MS data for machine learning applications

2021 ◽  
Author(s):  
Tobias Greisager Rehfeldt ◽  
Konrad Krawczyk ◽  
Mathias Bøgebjerg ◽  
Veit Schwämmle ◽  
Richard Röttger
Keyword(s):  
Machine Learning ◽  
Mass Spectrometry ◽  
Large Scale ◽  
Peptide Identification ◽  
Training Data ◽  
Supplementary Information ◽  
Large Sample Size ◽  
Raw Data ◽  
Machine Learning Applications ◽  
Rich Data

AbstractMotivationLiquid-chromatography mass-spectrometry (LC-MS) is the established standard for analyzing the proteome in biological samples by identification and quantification of thousands of proteins. Machine learning (ML) promises to considerably improve the analysis of the resulting data, however, there is yet to be any tool that mediates the path from raw data to modern ML applications. More specifically, ML applications are currently hampered by three major limitations: (1) absence of balanced training data with large sample size; (2) unclear definition of sufficiently information-rich data representations for e.g. peptide identification; (3) lack of benchmarking of ML methods on specific LC-MS problems.ResultsWe created the MS2AI pipeline that automates the process of gathering vast quantities of mass spectrometry (MS) data for large scale ML applications. The software retrieves raw data from either in-house sources or from the proteomics identifications database, PRIDE. Subsequently, the raw data is stored in a standardized format amenable for ML encompassing MS1/MS2 spectra and peptide identifications. This tool bridges the gap between MS and AI, and to this effect we also present an ML application in the form of a convolutional neural network for the identification of oxidized peptides.AvailabilityAn open source implementation of the software can be found freely available for non-commercial use at https://gitlab.com/roettgerlab/[email protected] informationSupplementary data are available at Bioinformatics online.

scholarly journals Large-scale comparative review and assessment of computational methods for anti-cancer peptide identification

2020 ◽  
Author(s):  
Xiao Liang ◽  
Fuyi Li ◽  
Jinxiang Chen ◽  
Junlong Li ◽  
Hao Wu ◽  
...  
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
State Of The Art ◽  
Machine Learning Algorithms ◽  
Accurate Identification ◽  
Advantages And Disadvantages ◽  
Comparative Review ◽  
Ensemble Strategy ◽  
Anti Cancer ◽  
Feature Encoding

Abstract Anti-cancer peptides (ACPs) are known as potential therapeutics for cancer. Due to their unique ability to target cancer cells without affecting healthy cells directly, they have been extensively studied. Many peptide-based drugs are currently evaluated in the preclinical and clinical trials. Accurate identification of ACPs has received considerable attention in recent years; as such, a number of machine learning-based methods for in silico identification of ACPs have been developed. These methods promote the research on the mechanism of ACPs therapeutics against cancer to some extent. There is a vast difference in these methods in terms of their training/testing datasets, machine learning algorithms, feature encoding schemes, feature selection methods and evaluation strategies used. Therefore, it is desirable to summarize the advantages and disadvantages of the existing methods, provide useful insights and suggestions for the development and improvement of novel computational tools to characterize and identify ACPs. With this in mind, we firstly comprehensively investigate 16 state-of-the-art predictors for ACPs in terms of their core algorithms, feature encoding schemes, performance evaluation metrics and webserver/software usability. Then, comprehensive performance assessment is conducted to evaluate the robustness and scalability of the existing predictors using a well-prepared benchmark dataset. We provide potential strategies for the model performance improvement. Moreover, we propose a novel ensemble learning framework, termed ACPredStackL, for the accurate identification of ACPs. ACPredStackL is developed based on the stacking ensemble strategy combined with SVM, Naïve Bayesian, lightGBM and KNN. Empirical benchmarking experiments against the state-of-the-art methods demonstrate that ACPredStackL achieves a comparative performance for predicting ACPs. The webserver and source code of ACPredStackL is freely available at http://bigdata.biocie.cn/ACPredStackL/ and https://github.com/liangxiaoq/ACPredStackL, respectively.

scholarly journals Machine learning method for tight-binding Hamiltonian parameterization from ab-initio band structure

2021 ◽  
Vol 7 (1) ◽  
Author(s):  
Zifeng Wang ◽  
Shizhuo Ye ◽  
Hao Wang ◽  
Jin He ◽  
Qijun Huang ◽  
...  
Keyword(s):  
Machine Learning ◽  
Ab Initio ◽  
Large Scale ◽  
Tight Binding ◽  
Real Space ◽  
Machine Learning Method ◽  
Learning Method ◽  
Energy Bands ◽  
Physical Problems ◽  
Machine Learning Applications

AbstractThe tight-binding (TB) method is an ideal candidate for determining electronic and transport properties for a large-scale system. It describes the system as real-space Hamiltonian matrices expressed on a manageable number of parameters, leading to substantially lower computational costs than the ab-initio methods. Since the whole system is defined by the parameterization scheme, the choice of the TB parameters decides the reliability of the TB calculations. The typical empirical TB method uses the TB parameters directly from the existing parameter sets, which hardly reproduces the desired electronic structures quantitatively without specific optimizations. It is thus not suitable for quantitative studies like the transport property calculations. The ab-initio TB method derives the TB parameters from the ab-initio results through the transformation of basis functions, which achieves much higher numerical accuracy. However, it assumes prior knowledge of the basis and may encompass truncation error. Here, a machine learning method for TB Hamiltonian parameterization is proposed, within which a neural network (NN) is introduced with its neurons acting as the TB matrix elements. This method can construct the empirical TB model that reproduces the given ab-initio energy bands with predefined accuracy, which provides a fast and convenient way for TB model construction and gives insights into machine learning applications in physical problems.

Sign in / Sign up

Export Citation Format

Share Document