scholarly journals Source-Finding for the Australian Square Kilometre Array Pathfinder

2012 ◽  
Vol 29 (3) ◽  
pp. 371-381 ◽  
Author(s):  
M. Whiting ◽  
B. Humphreys
Keyword(s):  
High Performance ◽  
Distributed Processing ◽  
Square Kilometre Array ◽  
Data Rates ◽  
Different Types ◽  
Automated Processing ◽  
Pipeline Processing ◽  
Science Community ◽  
Line Imaging ◽  
Performance Computing

AbstractThe Australian Square Kilometre Array Pathfinder (ASKAP) presents a number of challenges in the area of source finding and cataloguing. The data rates and image sizes are very large, and require automated processing in a high-performance computing environment. This requires development of new tools, that are able to operate in such an environment and can reliably handle large datasets. These tools must also be able to accommodate the different types of observations ASKAP will make: continuum imaging, spectral-line imaging, transient imaging. The ASKAP project has developed a source-finder known as selavy, built upon the duchamp source-finder. selavy incorporates a number of new features, which we describe here.Since distributed processing of large images and cubes will be essential, we describe the algorithms used to distribute the data, find an appropriate threshold and search to that threshold and form the final source catalogue. We describe the algorithm used to define a varying threshold that responds to the local, rather than global, noise conditions, and provide examples of its use. And we discuss the approach used to apply two-dimensional fits to detected sources, enabling more accurate parameterisation. These new features are compared for timing performance, where we show that their impact on the pipeline processing will be small, providing room for enhanced algorithms.We also discuss the development process for ASKAP source finding software. By the time of ASKAP operations, the ASKAP science community, through the Survey Science Projects, will have contributed important elements of the source finding pipeline, and the mechanisms in which this will be done are presented.

scholarly journals Big telescope, big data: towards exascale with the Square Kilometre Array

2020 ◽  
Vol 378 (2166) ◽  
pp. 20190060
Author(s):  
A. M. M. Scaife
Keyword(s):  
High Performance ◽  
Numerical Algorithms ◽  
Image Formation ◽  
Science Data ◽  
Square Kilometre Array ◽  
Huge Data ◽  
Data Rates ◽  
Data Processor ◽  
Optical Telescopes ◽  
Scale Data

Unlike optical telescopes, radio interferometers do not image the sky directly but require specialized image formation algorithms. For the Square Kilometre Array (SKA), the computational requirements of this image formation are extremely demanding due to the huge data rates produced by the telescope. This processing will be performed by the SKA Science Data Processor facilities and a network of SKA Regional Centres, which must not only deal with SKA-scale data volumes but also with stringent science-driven image fidelity requirements. This article is part of a discussion meeting issue ‘Numerical algorithms for high-performance computational science’.

scholarly journals Labkit: Labeling and Segmentation Toolkit for Big Image Data

2021 ◽  
Author(s):  
Matthias Arzt ◽  
Joran Deschamps ◽  
Christopher Schmied ◽  
Tobias Pietzsch ◽  
Deborah Schmidt ◽  
...  
Keyword(s):  
Graphics Processing Units ◽  
High Performance ◽  
Distributed Processing ◽  
Image Data ◽  
Microscopy Image ◽  
Processing Step ◽  
Graphics Processing ◽  
User Friendly ◽  
Automated Image Processing ◽  
Performance Computing

We present Labkit, a user-friendly Fiji plugin for the segmentation of microscopy image data. It offers easy to use manual and automated image segmentation routines that can be rapidly applied to single- and multi-channel images as well as to timelapse movies in 2D or 3D. Labkit is specifically designed to work efficiently on big image data and enables users of consumer laptops to conveniently work with multiple-terabyte images. This efficiency is achieved by using ImgLib2 and BigDataViewer as the foundation of our software. Furthermore, memory efficient and fast random forest based pixel classification inspired by the Waikato Environment for Knowledge Analysis (Weka) is implemented. Optionally we harness the power of graphics processing units (GPU) to gain additional runtime performance. Labkit is easy to install on virtually all laptops and workstations. Additionally, Labkit is compatible with high performance computing (HPC) clusters for distributed processing of big image data. The ability to use pixel classifiers trained in Labkit via the ImageJ macro language enables our users to integrate this functionality as a processing step in automated image processing workflows. Last but not least, Labkit comes with rich online resources such as tutorials and examples that will help users to familiarize themselves with available features and how to best use \Labkit in a number of practical real-world use-cases.

scholarly journals HPCI: A Perl module for writing cluster-portable bioinformatics pipelines

2018 ◽  
Author(s):  
John M Macdonald ◽  
Christopher M Lalansingh ◽  
Christopher I Cooper ◽  
Anqi Yang ◽  
Felix Lam ◽  
...  
Keyword(s):  
High Performance ◽  
Application Programming Interface ◽  
List Type ◽  
Software Module ◽  
Perl Module ◽  
Software Interface ◽  
Different Types ◽  
Application Programming ◽  
Computing Environments ◽  
Performance Computing

AbstractBackgroundMost biocomputing pipelines are run on clusters of computers. Each type of cluster has its own API (application programming interface). That API defines how a program that is to run on the cluster must request the submission, content and monitoring of jobs to be run on the cluster. Sometimes, it is desirable to run the same pipeline on different types of cluster. This can happen in situations including when:different labs are collaborating, but they do not use the same type of clustera pipeline is released to other labs as open source or commercial softwarea lab has access to multiple types of cluster, and wants to choose between them for scaling, cost or other purposesa lab is migrating their infrastructure from one cluster type to anotherduring testing or travelling, it is often desired to run on a single computerHowever, since each type of cluster has its own API, code that runs jobs on one type of cluster needs to be re-written if it is desired to run that application on a different type of cluster. To resolve this problem, we created a software module to generalize the submission of pipelines across computing environments, including local compute, clouds and clusters.ResultsHPCI (High Performance Computing Interface) is a Perl module that provides the interface to a standardized generic cluster.When the HPCI module is used, it accepts a parameter to specify the cluster type. The HPCI module uses this to load a driver HPCD∷<cluster>. This is used to translate the abstract HPCI interface to the specific software interface.Simply by changing the cluster parameter, the same pipeline can be run on a different type of cluster with no other changes.ConclusionThe HPCI module assists in writing Perl programs that can be run in different lab environments, with different site configuration requirements and different types of hardware clusters. Rather than having to re-write portions of the program, it is only necessary to change a configuration file.Using HPCI, an application can manage collections of jobs to be runs, specify ordering dependencies, detect success or failure of jobs run and allow automatic retry of failed jobs (allowing for the possibility of a changed configuration such as when the original attempt specified an inadequate memory allotment).

scholarly journals Integrating HPC into an agile and cloud-focused environment at CERN

2019 ◽  
Vol 214 ◽  
pp. 07025
Author(s):  
Pablo Llopis ◽  
Carolina Lindqvist ◽  
Nils Høimyr ◽  
Dan van der Ster ◽  
Philippe Ganz
Keyword(s):  
High Performance ◽  
Hadron Collider ◽  
Parallel Applications ◽  
Application Development ◽  
Shared Storage ◽  
Technology Sector ◽  
Processing Data ◽  
Different Types ◽  
Mpi Applications ◽  
Performance Computing

CERN’s batch and grid services are mainly focused on High Throughput computing (HTC) for processing data from the Large Hadron Collider (LHC) and other experiments. However, part of the user community requires High Performance Computing (HPC) for massively parallel applications across many cores on MPI-enabled infrastructure. This contribution addresses the implementation of HPC infrastructure at CERN for Lattice QCD application development, as well as for different types of simulations for the accelerator and technology sector at CERN. Our approach has been to integrate the HPC facilities as far as possible with the HTC services in our data centre, and to take advantage of an agile infrastructure for updates, configuration and deployment. The HPC cluster has been orchestrated with the OpenStack Ironic component, and is hence managed with the same tools as the CERN internal OpenStack cloud. Experience and benchmarks of MPI applications across Infiniband with shared storage on CephFS is discussed, as well the setup of the SLURM scheduler for HPC jobs with a provision for backfill of HTC workloads.

scholarly journals GPU Usage in ATLAS Reconstruction and Analysis

2020 ◽  
Vol 245 ◽  
pp. 05006
Author(s):  
Attila Krasznahorkay ◽  
Charles Leggett ◽  
Alaettin Serhan Mete ◽  
Scott Snyder ◽  
Vakho Tsulaia
Keyword(s):  
Data Model ◽  
Memory Management ◽  
High Performance ◽  
Event Data ◽  
Software Environment ◽  
The Past ◽  
Graphical Processing Units ◽  
Different Types ◽  
Graphical Processing ◽  
Performance Computing

With Graphical Processing Units (GPUs) and other kinds of accelerators becoming ever more accessible, High Performance Computing Centres all around the world using them ever more, ATLAS has to find the best way of making use of such accelerators in much of its computing. Tests with GPUs – mainly with CUDA – have been performed in the past in the experiment. At that time the conclusion was that it was not advantageous for the ATLAS offline and trigger software to invest time and money into GPUs. However as the usage of accelerators has become cheaper and simpler in recent years, their re-evaluation in ATLAS’s offline software is warranted. We show new results of using GPU accelerated calculations in ATLAS’s offline software environment using the ATLAS offline/analysis (xAOD) Event Data Model. We compare the performance and flexibility of a couple of the available GPU programming methods, and show how different memory management setups affect our ability to offload different types of calculations to a GPU efficiently.

scholarly journals Scientific Visualization in Astronomy: Towards the Petascale Astronomy Era

2011 ◽  
Vol 28 (2) ◽  
pp. 150-170 ◽  
Author(s):  
Amr Hassan ◽  
Christopher J. Fluke
Keyword(s):  
High Performance ◽  
Scientific Visualization ◽  
Distributed Processing ◽  
Critical Role ◽  
General Purpose ◽  
Workflow Systems ◽  
New Era ◽  
Data Representations ◽  
Advanced Interaction ◽  
Performance Computing

AbstractAstronomy is entering a new era of discovery, coincident with the establishment of new facilities for observation and simulation that will routinely generate petabytes of data. While an increasing reliance on automated data analysis is anticipated, a critical role will remain for visualization-based knowledge discovery. We have investigated scientific visualization applications in astronomy through an examination of the literature published during the last two decades. We identify the two most active fields for progress — visualization of large-N particle data and spectral data cubes—discuss open areas of research, and introduce a mapping between astronomical sources of data and data representations used in general-purpose visualization tools. We discuss contributions using high-performance computing architectures (e.g. distributed processing and GPUs), collaborative astronomy visualization, the use of workflow systems to store metadata about visualization parameters, and the use of advanced interaction devices. We examine a number of issues that may be limiting the spread of scientific visualization research in astronomy and identify six grand challenges for scientific visualization research in the Petascale Astronomy Era.

Design of an intermediate high voltage EM for 3-D studies of biological material and its integration with a system for remote access

1995 ◽  
Vol 53 ◽  
pp. 66-67 ◽  
Author(s):  
Mark H. Ellisman
Keyword(s):  
High Voltage ◽  
High Performance ◽  
Remote Access ◽  
3 Dimensional ◽  
Voltage Electron ◽  
Remote Operation ◽  
High Voltage Electron Microscope ◽  
La Jolla ◽  
Performance Computing ◽  
Advanced Version

The increased availability of High Performance Computing and Communications (HPCC) offers scientists and students the potential for effective remote interactive use of centralized, specialized, and expensive instrumentation and computers. Examples of instruments capable of remote operation that may be usefully controlled from a distance are increasing. Some in current use include telescopes, networks of remote geophysical sensing devices and more recently, the intermediate high voltage electron microscope developed at the San Diego Microscopy and Imaging Resource (SDMIR) in La Jolla. In this presentation the imaging capabilities of a specially designed JEOL 4000EX IVEM will be described. This instrument was developed mainly to facilitate the extraction of 3-dimensional information from thick sections. In addition, progress will be described on a project now underway to develop a more advanced version of the Telemicroscopy software we previously demonstrated as a tool to for providing remote access to this IVEM (Mercurio et al., 1992; Fan et al., 1992).

Sign in / Sign up

Export Citation Format

Share Document