ESO Phase 3 automatic data validation: groovy-based tool to assure the compliance of the reduced data with the Science Data Product Standard

Abstract. The Multi-angle Imaging SpectroRadiometer (MISR) is one of the five instruments hosted on board the NASA Terra platform, launched on 18 December 1999. This instrument has been operational since 24 February 2000 and is still acquiring Earth observation data as of this writing. The primary mission of the MISR is to document the state and properties of the atmosphere, in particular the clouds and aerosols it contains, as well as the planetary surface, on the basis of 36 data channels collectively gathered by its nine cameras (pointing in different directions along the orbital track) in four spectral bands (blue, green, red and near-infrared). The radiometric camera-by-camera cloud mask (RCCM) is derived from the calibrated measurements at the nominal top of the atmosphere and is provided separately for each of the nine cameras. This RCCM data product is permanently archived at the NASA Atmospheric Science Data Center (ASDC) in Hampton, VA, USA, and is openly accessible (Diner et al., 1999b, and https://doi.org/10.5067/Terra/MISR/MIRCCM_L2.004). For various technical reasons described in this paper, this RCCM product exhibits missing data, even though an estimate of the clear or cloudy status of the environment at each individual observed location can be deduced from the available measurements. The aims of this paper are (1) to describe how to replace over 99 % of the missing values by estimates and (2) to briefly describe the software to replace missing RCCM values, which is openly available to the community from the GitHub website, https://github.com/mmverstraete/MISR\\ RCCM/ (last access: 12 March 2020), or https://doi.org/10.5281/ZENODO.3240017 (Verstraete, 2019e). Two additional sets of resources are also made available on the research data repository of GFZ Data Services in conjunction with this paper. The first set (A; Verstraete et al., 2020; https://doi.org/10.5880/fidgeo.2020.004) includes three items: (A1) a compressed archive, RCCM_Out.zip, containing all intermediary, final and ancillary outputs created while generating the figures of this paper; (A2) a user manual, RCCM_Out.pdf, describing how to install, uncompress and explore those files; and (A3) a separate input MISR data archive, RCCM_input_68050.zip, for Path 168, Orbit 68050. This latter archive is usable with (B), the second set (Verstraete and Vogt, 2020; https://doi.org/10.5880/fidgeo.2020.008), which includes (B1), a stand-alone, self-contained, executable version of the RCCM correction codes, RCCM_Soft_Win.zip, using the IDL Virtual Machine technology that does not require a paid IDL license, as well as (B2), a user manual, RCCM_Soft_Win.pdf, to explain how to install, uncompress and use this software.

Download Full-text

Flowmeter data validation and reconstruction methodology to provide the annual efficiency of a water transport network: the ATLL case study in Catalonia

Water Science & Technology Water Supply ◽

10.2166/ws.2013.203 ◽

2013 ◽

Vol 14 (2) ◽

pp. 337-346

Author(s):

J. Quevedo ◽

J. Pascual ◽

V. Puig ◽

J. Saludes ◽

R. Sarrate ◽

...

Keyword(s):

Water Transport ◽

Transport Network ◽

Information Storage ◽

Water Volume ◽

Data Validation ◽

Network Efficiency ◽

Data Set ◽

Flow Measurements ◽

Automatic Data ◽

Daily Data

The object of this paper is to provide a flowmeter data validation/reconstruction methodology that determines the annual economic efficiency of a water transport network. In this paper, the case of Aigües Ter Llobregat (ATLL) company, which manages 80% of the overall water transport network in Catalonia (Spain), will be used for illustrating purposes. Economic network efficiency is based on daily data set collected by the company using about 200 flowmeters of the network. Data collected using these sensors are used by remote control and information storage systems and they are stored in a relational database. All information provided by ATLL is analysed to detect inconsistent data using an automatic data validation method deployed in parallel with the network efficiency evaluation. As a result of the validation process, corrections of flow measurements and of billed water volume are introduced. Results from ATLL water transport network corresponding to year 2010 will be used to illustrate the approach proposed in this paper.

Download Full-text

An Overview of the Joint Polar Satellite System (JPSS) Science Data Product Calibration and Validation

Remote Sensing ◽

10.3390/rs8020139 ◽

2016 ◽

Vol 8 (2) ◽

pp. 139 ◽

Cited By ~ 18

Author(s):

Lihang Zhou ◽

Murty Divakarla ◽

Xingpin Liu

Keyword(s):

Satellite System ◽

Science Data ◽

Calibration And Validation ◽

Data Product ◽

Polar Satellite

Download Full-text

Clouds and the Earth's Radiant Energy System (CERES) instrument level 1 science data validation plan for geolocated radiances

10.1117/12.258093 ◽

1996 ◽

Cited By ~ 1

Author(s):

Robert B. Lee III ◽

Brooks A. Childers ◽

G. Louis Smith ◽

Jack Paden ◽

Dhirendra K. Pandey ◽

...

Keyword(s):

Radiant Energy ◽

Energy System ◽

Data Validation ◽

Science Data ◽

Validation Plan ◽

Level 1 ◽

Instrument Level

Download Full-text

Replacing Missing Values in the Standard MISR Radiometric Camera-by-Camera Cloud Mask (RCCM) Data Product

10.5194/essd-2019-77 ◽

2019 ◽

Author(s):

Michel M. Verstraete ◽

Linda A. Hunt ◽

Hugo De Lemos ◽

Larry Di Girolamo

Keyword(s):

Data Center ◽

Missing Values ◽

Near Infrared ◽

Observation Data ◽

Science Data ◽

Data Product ◽

Spectral Bands ◽

Data Products ◽

Earth Observation Data ◽

Cloud Mask

Abstract. The Multi-angle Imaging SpectroRadiometer (MISR) is one of the five instruments hosted on-board the NASA Terra platform, launched on 18 December 1999. This instrument has been operational since 24 February 2000 and is still acquiring Earth Observation data as of this writing. The primary missions of MISR are to document the state and properties of the atmosphere, and in particular the clouds and aerosols it contains, as well as the planetary surface, on the basis of 36 data channels gathered by each of its nine cameras (pointing in different directions along the orbital track) in four spectral bands (blue, green, red and near-infrared). The Radiometric Camera-by-Camera Cloud Mask (RCCM) is derived from the calibrated measurements at the nominal top of the atmosphere, and is provided separately for each of the nine cameras. This RCCM data product is permanently archived at the NASA Atmospheric Science Data Center (ASDC) in Langley, VA, USA and is openly accessible (Diner et al., 1999 and https://doi.org/10.5067/Terra/MISR/MIRCCM_L2.004). For various technical reasons described in this paper, this RCCM product exhibits missing data, even though an estimate of the clear or cloudy status of the environment at each individual observed location can be deduced from the available measurements. The aims of this paper are (1) to describe how to replace most missing values by estimates and (2) to briefly describe the software to process MISR RCCM data products, which is openly available to the community from the GitHub web site (https://github.com/mmverstraete or https://doi.org/10.5281/zenodo.3240018). Limited amounts of updated MISR RCCM data products are also archived in South Africa and can be made available upon request.

Download Full-text

Science Autonomy for Rover Subsurface Exploration of the Atacama Desert

AI Magazine ◽

10.1609/aimag.v35i4.2554 ◽

2014 ◽

Vol 35 (4) ◽

pp. 47-60 ◽

Cited By ~ 10

Author(s):

David Wettergreen ◽

Greydon Foil ◽

Michael Furlong ◽

David R. Thompson

Keyword(s):

Field Experiments ◽

Spatial Scales ◽

Sample Selection ◽

Atacama Desert ◽

Scientific Data ◽

Science Data ◽

Long Distance ◽

Automatic Data ◽

Limited Bandwidth

As planetary rovers expand their capabilities, traveling longer distances, deploying complex tools, and collecting voluminous scientific data, the requirements for intelligent guidance and control also grow. This, coupled with limited bandwidth and latencies, motivates onboard autonomy that ensures the quality of the science data return. Increasing quality of the data involves better sample selection, data validation, and data reduction. Robotic studies in Mars-like desert terrain have advanced autonomy for long distance exploration and seeded technologies for planetary rover missions. In these field experiments the remote science team uses a novel control strategy that intersperses preplanned activities with autonomous decision making. The robot performs automatic data collection, interpretation, and response at multiple spatial scales. Specific capabilities include instrument calibration, visual targeting of selected features, an onboard database of collected data, and a long range path planner that guides the robot using analysis of current surface and prior satellite data. Field experiments in the Atacama Desert of Chile over the past decade demonstrate these capabilities and illustrate current challenges and future directions.

Download Full-text