scholarly journals Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

2020 ◽  
Vol 110 ◽  
pp. 440-452 ◽  
Author(s):  
S. Caíno-Lores ◽  
A. Lapin ◽  
J. Carretero ◽  
P. Kropf
2013 ◽  
Vol 14 (1) ◽  
pp. 51-61 ◽  
Author(s):  
Fabian Fischer ◽  
Johannes Fuchs ◽  
Florian Mansmann ◽  
Daniel A Keim

The enormous growth of data in the last decades led to a wide variety of different database technologies. Nowadays, we are capable of storing vast amounts of structured and unstructured data. To address the challenge of exploring and making sense out of big data using visual analytics, the tight integration of such backend services is needed. In this article, we introduce BANKSAFE, which was built for the VAST Challenge 2012 and won the outstanding comprehensive submission award. BANKSAFE is based on modern database technologies and is capable of visually analyzing vast amounts of monitoring data and security-related datasets of large-scale computer networks. To better describe and demonstrate the visualizations, we utilize the Visual Analytics Science and Technology (VAST) Challenge 2012 as case study. Additionally, we discuss lessons learned during the design and development of BANKSAFE, which are also applicable to other visual analytics applications for big data.


Author(s):  
O. Melet ◽  
D. Youssefi ◽  
C. L'Helguen ◽  
J. Michel ◽  
E. Sarrazin ◽  
...  

Abstract. Earth Observation (EO) remote sensing missions are producing an increasing volume of data due to higher spatial and spectral resolutions, and higher frequency of acquisitions. Thus, in order to prepare the future of image processing pipelines, CNES has carried out Research & Development studies related to the use of Big Data and Cloud technologies for image processing chains made. Since mid-2019, CNES in partnership with Airbus Defense & Space, has started a new High Resolution Optical EO mission dedicated to very high resolution 3D observation called CO3D (“Constellation Optique 3D”). To achieve those objectives, a new image processing pipeline prototype is being developed taking in consideration the lessons learned from the previous studies. The paper will introduce this new image processing pipeline, the processing paradigms used to take advantage of big data technologies and the results of production benchmarks at a large scale. The on-going works to optimize the processing pipeline and Cloud cluster will be also discussed.


2020 ◽  
Vol 29 (3S) ◽  
pp. 638-647 ◽  
Author(s):  
Janine F. J. Meijerink ◽  
Marieke Pronk ◽  
Sophia E. Kramer

Purpose The SUpport PRogram (SUPR) study was carried out in the context of a private academic partnership and is the first study to evaluate the long-term effects of a communication program (SUPR) for older hearing aid users and their communication partners on a large scale in a hearing aid dispensing setting. The purpose of this research note is to reflect on the lessons that we learned during the different development, implementation, and evaluation phases of the SUPR project. Procedure This research note describes the procedures that were followed during the different phases of the SUPR project and provides a critical discussion to describe the strengths and weaknesses of the approach taken. Conclusion This research note might provide researchers and intervention developers with useful insights as to how aural rehabilitation interventions, such as the SUPR, can be developed by incorporating the needs of the different stakeholders, evaluated by using a robust research design (including a large sample size and a longer term follow-up assessment), and implemented widely by collaborating with a private partner (hearing aid dispensing practice chain).


2020 ◽  
Author(s):  
Anusha Ampavathi ◽  
Vijaya Saradhi T

UNSTRUCTURED Big data and its approaches are generally helpful for healthcare and biomedical sectors for predicting the disease. For trivial symptoms, the difficulty is to meet the doctors at any time in the hospital. Thus, big data provides essential data regarding the diseases on the basis of the patient’s symptoms. For several medical organizations, disease prediction is important for making the best feasible health care decisions. Conversely, the conventional medical care model offers input as structured that requires more accurate and consistent prediction. This paper is planned to develop the multi-disease prediction using the improvised deep learning concept. Here, the different datasets pertain to “Diabetes, Hepatitis, lung cancer, liver tumor, heart disease, Parkinson’s disease, and Alzheimer’s disease”, from the benchmark UCI repository is gathered for conducting the experiment. The proposed model involves three phases (a) Data normalization (b) Weighted normalized feature extraction, and (c) prediction. Initially, the dataset is normalized in order to make the attribute's range at a certain level. Further, weighted feature extraction is performed, in which a weight function is multiplied with each attribute value for making large scale deviation. Here, the weight function is optimized using the combination of two meta-heuristic algorithms termed as Jaya Algorithm-based Multi-Verse Optimization algorithm (JA-MVO). The optimally extracted features are subjected to the hybrid deep learning algorithms like “Deep Belief Network (DBN) and Recurrent Neural Network (RNN)”. As a modification to hybrid deep learning architecture, the weight of both DBN and RNN is optimized using the same hybrid optimization algorithm. Further, the comparative evaluation of the proposed prediction over the existing models certifies its effectiveness through various performance measures.


2021 ◽  
Vol 51 (3) ◽  
pp. 9-16
Author(s):  
José Suárez-Varela ◽  
Miquel Ferriol-Galmés ◽  
Albert López ◽  
Paul Almasan ◽  
Guillermo Bernárdez ◽  
...  

During the last decade, Machine Learning (ML) has increasingly become a hot topic in the field of Computer Networks and is expected to be gradually adopted for a plethora of control, monitoring and management tasks in real-world deployments. This poses the need to count on new generations of students, researchers and practitioners with a solid background in ML applied to networks. During 2020, the International Telecommunication Union (ITU) has organized the "ITU AI/ML in 5G challenge", an open global competition that has introduced to a broad audience some of the current main challenges in ML for networks. This large-scale initiative has gathered 23 different challenges proposed by network operators, equipment manufacturers and academia, and has attracted a total of 1300+ participants from 60+ countries. This paper narrates our experience organizing one of the proposed challenges: the "Graph Neural Networking Challenge 2020". We describe the problem presented to participants, the tools and resources provided, some organization aspects and participation statistics, an outline of the top-3 awarded solutions, and a summary with some lessons learned during all this journey. As a result, this challenge leaves a curated set of educational resources openly available to anyone interested in the topic.


Sign in / Sign up

Export Citation Format

Share Document