scholarly journals Enabling the Big Earth Observation Data via Cloud Computing and DGGS: Opportunities and Challenges

2019 ◽  
Vol 12 (1) ◽  
pp. 62 ◽  
Author(s):  
Xiaochuang Yao ◽  
Guoqing Li ◽  
Junshi Xia ◽  
Jin Ben ◽  
Qianqian Cao ◽  
...  
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Earth Observation ◽  
Current Status ◽  
Observation Data ◽  
Computing Technology ◽  
Grid Systems ◽  
Data Framework ◽  
Earth Observation Data ◽  
Global Grid

In the era of big data, the explosive growth of Earth observation data and the rapid advancement in cloud computing technology make the global-oriented spatiotemporal data simulation possible. These dual developments also provide advantageous conditions for discrete global grid systems (DGGS). DGGS are designed to portray real-world phenomena by providing a spatiotemporal unified framework on a standard discrete geospatial data structure and theoretical support to address the challenges from big data storage, processing, and analysis to visualization and data sharing. In this paper, the trinity of big Earth observation data (BEOD), cloud computing, and DGGS is proposed, and based on this trinity theory, we explore the opportunities and challenges to handle BEOD from two aspects, namely, information technology and unified data framework. Our focus is on how cloud computing and DGGS can provide an excellent solution to enable big Earth observation data. Firstly, we describe the current status and data characteristics of Earth observation data, which indicate the arrival of the era of big data in the Earth observation domain. Subsequently, we review the cloud computing technology and DGGS framework, especially the works and contributions made in the field of BEOD, including spatial cloud computing, mainstream big data platform, DGGS standards, data models, and applications. From the aforementioned views of the general introduction, the research opportunities and challenges are enumerated and discussed, including EO data management, data fusion, and grid encoding, which are concerned with analysis models and processing performance of big Earth observation data with discrete global grid systems in the cloud environment.

scholarly journals FluxEngine: A Flexible Processing System for Calculating Atmosphere–Ocean Carbon Dioxide Gas Fluxes and Climatologies

2016 ◽  
Vol 33 (4) ◽  
pp. 741-756 ◽  
Author(s):  
Jamie D. Shutler ◽  
Peter E. Land ◽  
Jean-Francois Piolle ◽  
David K. Woolf ◽  
Lonneke Goddijn-Murphy ◽  
...  
Keyword(s):  
Carbon Dioxide ◽  
Cloud Computing ◽  
Processing System ◽  
Earth Observation ◽  
Observation Data ◽  
Large Spatial Scale ◽  
Process Indicator ◽  
Flux Calculation ◽  
Gas Fluxes ◽  
Earth Observation Data

AbstractThe air–sea flux of greenhouse gases [e.g., carbon dioxide (CO2)] is a critical part of the climate system and a major factor in the biogeochemical development of the oceans. More accurate and higher-resolution calculations of these gas fluxes are required if researchers are to fully understand and predict future climate. Satellite Earth observation is able to provide large spatial-scale datasets that can be used to study gas fluxes. However, the large storage requirements needed to host such data can restrict its use by the scientific community. Fortunately, the development of cloud computing can provide a solution. This paper describes an open-source air–sea CO2 flux processing toolbox called the “FluxEngine,” designed for use on a cloud-computing infrastructure. The toolbox allows users to easily generate global and regional air–sea CO2 flux data from model, in situ, and Earth observation data, and its air–sea gas flux calculation is user configurable. Its current installation on the Nephalae Cloud allows users to easily exploit more than 8 TB of climate-quality Earth observation data for the derivation of gas fluxes. The resultant netCDF data output files contain >20 data layers containing the various stages of the flux calculation along with process indicator layers to aid interpretation of the data. This paper describes the toolbox design, which verifies the air–sea CO2 flux calculations; demonstrates the use of the tools for studying global and shelf sea air–sea fluxes; and describes future developments.

scholarly journals Remote Sensing

2021 ◽  
pp. 49-61
Author(s):  
Miguel Ángel Esbrí
Keyword(s):  
Remote Sensing ◽  
Big Data ◽  
Information Access ◽  
Open Data ◽  
Earth Observation ◽  
Observation Data ◽  
Related Data ◽  
Data Formats ◽  
Earth Observation Data ◽  
Access Services

AbstractIn this chapter we present the concepts of remote sensing and Earth Observation and, explain why several of their characteristics (volume, variety and velocity) make us consider Earth Observation as Big Data. Thereafter, we discuss the most commonly open data formats used to store and share the data. The main sources of Earth Observation data are also described, with particular focus on the constellation of Sentinel satellites, Copernicus Hub and its six thematic services, as well as other private initiatives like the five Copernicus-related Data and Information Access Services and  Sentinel Hub. Next, we present an overview of representative software technologies for efficiently describing, storing, querying and accessing Earth Observation datasets. The chapter concludes with a summary of the Earth Observation datasets used in each DataBio pilot.

scholarly journals On-the-Fly Fusion of Remotely-Sensed Big Data Using an Elastic Computing Paradigm with a Containerized Spark Engine on Kubernetes

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 2971
Author(s):  
Wei Huang ◽  
Jianzhong Zhou ◽  
Dongying Zhang
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Satellite Image ◽  
Remotely Sensed ◽  
Observation Data ◽  
Fusion Model ◽  
Computing Paradigm ◽  
Big Picture ◽  
Elastic Computing ◽  
Earth Observation Data

Remotely-sensed satellite image fusion is indispensable for the generation of long-term gap-free Earth observation data. While cloud computing (CC) provides the big picture for RS big data (RSBD), the fundamental question of the efficient fusion of RSBD on CC platforms has not yet been settled. To this end, we propose a lightweight cloud-native framework for the elastic processing of RSBD in this study. With the scaling mechanisms provided by both the Infrastructure as a Service (IaaS) and Platform as a Services (PaaS) of CC, the Spark-on-Kubernetes operator model running in the framework can enhance the efficiency of Spark-based algorithms without considering bottlenecks such as task latency caused by an unbalanced workload, and can ease the burden to tune the performance parameters for their parallel algorithms. Internally, we propose a task scheduling mechanism (TSM) to dynamically change the Spark executor pods’ affinities to the computing hosts. The TSM learns the workload of a computing host. Learning from the ratio between the number of completed and failed tasks on a computing host, the TSM dispatches Spark executor pods to newer and less-overwhelmed computing hosts. In order to illustrate the advantage, we implement a parallel enhanced spatial and temporal adaptive reflectance fusion model (PESTARFM) to enable the efficient fusion of big RS images with a Spark aggregation function. We construct an OpenStack cloud computing environment to test the usability of the framework. According to the experiments, TSM can improve the performance of the PESTARFM using only PaaS scaling to about 11.7%. When using both the IaaS and PaaS scaling, the maximum performance gain with the TSM can be even greater than 13.6%. The fusion of such big Sentinel and PlanetScope images requires less than 4 min in the experimental environment.

scholarly journals Open Source Framework for Enabling HPC and Cloud Geoprocessing Services

2020 ◽  
Vol 12 (4) ◽  
pp. 61-76
Author(s):  
José Miguel Montañana ◽  
Paolo Marangino ◽  
Antonio Hervás
Keyword(s):  
Big Data ◽  
Open Source ◽  
Precision Agriculture ◽  
Data Analytics ◽  
Performance Metrics ◽  
Earth Observation ◽  
Observation Data ◽  
Open Source Framework ◽  
Earth Observation Data ◽  
Geoprocessing Services

Geoprocessing is a set of tools that can be used to efficiently address several pressing chal-lenges for the global economy ranging from agricultural productivity, the design of transport networks, to the prediction of climate change and natural disasters. This paper describes an Open Source Framework developed, within three European projects, for Ena-bling High-Performance Computing (HPC) and Cloud geoprocessing services applied to agricultural challenges. The main goals of the European Union projects EUXDAT (EUro-pean e-infrastructure for eXtreme Data Analytics in sustainable developmenT), CYBELE (fostering precision agriculture and livestock farming through secure access to large-scale HPC-enabled virtual industrial experimentation environment empowering scalable big data analytics), and EOPEN (opEn interOperable Platform for unified access and analysis of Earth observatioN data) are to enable the use of large HPC systems, as well as big data management, user-friendly access and visualization of results. In addition, these projects focus on the development of software frameworks, and fuse Earth-observation data, such as Copernicus data, with non-Earth-observation data, such as weather, environmental and social media information. In this paper, we describe the agroclimatic-zones pilot used to validate the framework. Finally, performance metrics collected during the execution (up to 182 times speedup with 256 MPI processes) of the pilot are presented.

Our goal is to establish the earth observation data in the business world Unser Ziel ist es, die Erdbeobachtungsdaten in der Geschäftswelt zu etablieren

GIS Business ◽  
2019 ◽  
Vol 12 (3) ◽  
pp. 12-14
Author(s):  
Eicher, A
Keyword(s):  
Earth Observation ◽  
Observation Data ◽  
Business World ◽  
The Earth ◽  
Earth Observation Data

Our goal is to establish the earth observation data in the business world Unser Ziel ist es, die Erdbeobachtungsdaten in der Geschäftswelt zu etablieren

Sign in / Sign up

Export Citation Format

Share Document