BD5: an open HDF5-based data format to represent quantitative biological dynamics data

BD5 is a new binary data format based on HDF5 (hierarchical data format version 5). It can be used for representing quantitative biological dynamics data obtained from bioimage informatics techniques and mechanobiological simulations. Biological Dynamics Markup Language (BDML) is an XML(Extensible Markup Language)-based open format that is also used to represent such data; however, it becomes difficult to access quantitative data in BDML files when the file size is large because parsing XML- based files requires large computational resources to first read the whole file sequentially into computer memory. BD5 enables fast random (i.e., direct) access to quantitative data on disk without parsing the entire file. Therefore, it allows practical reuse of data for understanding biological mechanisms underlying the dynamics.

NeuroChaT: A toolbox to analyse the dynamics of neuronal encoding in freely-behaving rodents in vivo

There is a dearth of freely-available, standardised open source analysis tools available for the analysis of neuronal signals recorded in vivo in the freely-behaving animal. In response, we have developed a freely-available, open-source toolbox, NeuroChaT (Neuron Characterisation Toolbox), specifically addressing this lacuna. Although we have particularly emphasised single unit analyses for spatial coding, NeuroChaT also characterises rhythmic properties of units and their dynamics associated with local field potential signals. NeuroChaT was developed using Python and facilitates a complete pipeline from automation of analysis to producing and managing publication-quality figures. Additionally, we have adopted a platform-independent format (Hierarchical Data Format version 5) for storing recorded and analysed data. By providing an easy-to-use software package, we aim to simplify the adoption of standardised analyses for behavioural neurophysiology and facilitate open data sharing and collaboration between laboratories.

Experimental Directory Structure (Exdir): An alternative to HDF5 without introducing a new file format

ABSTRACTNatural sciences generate an increasing amount of data in a wide range of formats developed by different research groups and commercial companies. At the same time there is a growing desire to share data along with publications in order to enable reproducible research. Open formats have publicly available specifications which facilitate data sharing and reproducible research. Hierarchical Data Format 5 (HDF5) is a popular open format widely used in neuroscience, often as a foundation for other, more specialized formats. However, drawbacks related to HDF5’s complex specification have initiated a discussion for an improved replacement. We propose a novel alternative, the Experimental Directory Structure (Exdir), an open standard for data storage in experimental pipelines which amends drawbacks associated with HDF5 while retaining its advantages. HDF5 stores data and metadata in a hierarchy within a complex binary file which, among other things, is not human-readable, not optimal for version control systems, and lacks support for storing raw data. Exdir, one the other hand, uses file system directories to represent the hierarchy, with metadata stored in human-readable YAML files, datasets stored in binary NumPy files, and raw data stored directly in subdirectories. Furthermore, storing data in multiple files makes it easier to track for version control systems. Exdir is not a file format in itself, but a standard for organizing files in a directory structure. Exdir uses the same abstractions as HDF5 and is compatible with the HDF5 Abstract Data Model. Several research groups are already using data stored in a directory hierarchy as an alternative to HDF5, but no common standard exists in the field. This complicates and limits the opportunity for data sharing and development of common tools for reading, writing, and analyzing data. Exdir facilitates improved data storage, data sharing, reproducible research, and novel insight from interdisciplinary collaboration. With the publication of Exdir, we invite the scientific community to join the development to create an open standard that will serve as many needs as possible and that will serve as a foundation for open access to and exchange of data.SIGNIFICANCE STATEMENTExperimental Directory Structure (Exdir) is a proposal to standardize a storage solution that has become an increasingly popular alternative to Hierarchical Data Format 5 (HDF5), namely to use directories to define a hierarchy, store data in binary files, and metadata in text files. While this strategy is deployed locally by several research groups, no common standard exists. We envision the establishment of such a standard and present Exdir to the community as a starting point.

eddy4R 0.2.0: a DevOps model for community-extensible processing and analysis of eddy-covariance data based on R, Git, Docker, and HDF5

Large differences in instrumentation, site setup, data format, and operating system stymie the adoption of a universal computational environment for processing and analyzing eddy-covariance (EC) data. This results in limited software applicability and extensibility in addition to often substantial inconsistencies in flux estimates. Addressing these concerns, this paper presents the systematic development of portable, reproducible, and extensible EC software achieved by adopting a development and systems operation (DevOps) approach. This software development model is used for the creation of the eddy4R family of EC code packages in the open-source R language for statistical computing. These packages are community developed, iterated via the Git distributed version control system, and wrapped into a portable and reproducible Docker filesystem that is independent of the underlying host operating system. The HDF5 hierarchical data format then provides a streamlined mechanism for highly compressed and fully self-documented data ingest and output. The usefulness of the DevOps approach was evaluated for three test applications. First, the resultant EC processing software was used to analyze standard flux tower data from the first EC instruments installed at a National Ecological Observatory (NEON) field site. Second, through an aircraft test application, we demonstrate the modular extensibility of eddy4R to analyze EC data from other platforms. Third, an intercomparison with commercial-grade software showed excellent agreement (R2 = 1.0 for CO2 flux). In conjunction with this study, a Docker image containing the first two eddy4R packages and an executable example workflow, as well as first NEON EC data products are released publicly. We conclude by describing the work remaining to arrive at the automated generation of science-grade EC fluxes and benefits to the science community at large. This software development model is applicable beyond EC and more generally builds the capacity to deploy complex algorithms developed by scientists in an efficient and scalable manner. In addition, modularity permits meeting project milestones while retaining extensibility with time.

OZONE VARIABILITY AND OZONE DEPLETING SUBSTANCES (ODS) IN INDONESIA BASED ON MLS-AURA DATA

Research and characterizing the ozone profiles and Ozone Depleting Substances (ODS) in Indonesia is a satellite data-based research activities. The aim of the study was to obtain the characteristics of ozone in Indonesia as well as the contribution of ODS to the variability of ozone. By performing a data inventory based on satellite data, analyze the pattern of annual, seasonal and perform linkage analysis of the contribution of ODS changes to the conditions of ozone. Daily data of vertical profiles of ozone and in the form of volume mixing ratio (vmr) with format HDF (Hierarchical Data Format) is extracted to the territory of Indonesia to take parameters as latitude, longitude, and concentration. Then converted to Excel format with the help of data processing software of MATLAB. Results obtained in the form of ozone characteristics in Indonesia, the percentage of contribution to the variability of ozone also contribution to the variability of ozone in Indonesia in several levels of height. By using Microwave Limb Sounders (MLS) AURA satellite data in the period of 2005 to 2013 characteristics of monthly vertical profiles of ozone in Indonesia has been obtained. The ODS studied were ClO and BrO. Peak of vertical profiles of ozone occurs at a pressure of 10 hPa or altitude of 25.9 km. ClO peak occurs at a pressure of 2.1 hPa or altitude of 30.6 km and BrO reached the peak at 14 hPa or altitude of 24.5 km. When ClO and BrO reach a maximum concentration at stratosphere then ozone molecules is potentially damaging or decrease in the stratosphere. Temporal variations of ozone showed decrease when ODS concentrations increased (particularly ClO and BrO). Linear regression of ozone with ozone showed a negative correlation coefficient which indicates there is a strong relationship between ozone concentrations decline in pressure of 14 hPa when BrO reach the maximum. Likewise for ClO which also showed a negative correlation with the decrease in ozone concentration. ClO contribution to the decreasing of ozone in Indonesia was marked by every addition of 0.01 ppb ClO will reduce ozone of 0.00583 ppm (5.83 ppb). While any increase of 0.01 ppb of BrO will decrease 0.03 ppb of ozone.

eddy4R: A community-extensible processing, analysis and modeling framework for eddy-covariance data based on R, Git, Docker and HDF5

This study presents the systematic development of an open-source, flexible and modular eddy-covariance (EC) data processing framework. This is achieved through adopting a Development and Systems Operation (DevOps) philosophy, building on the eddy4R family of EC code packages in the R Language for Statistical Computing as foundation. These packages are community-developed via the GitHub distributed version control system and wrapped into a portable and reproducible Docker filesystem that is independent of the underlying host operating system. The HDF5 hierarchical data format then provides a streamlined mechanism for highly compressed and fully self-documented data ingest and output. This framework is applicable beyond EC, and more generally builds the capacity to deploy complex algorithms developed by scientists in an efficient and scalable manner. In addition, modularity permits meeting project milestones while retaining extensibility with time. The efficiency and consistency of this framework is demonstrated in the form of three application examples. These include tower EC data from first instruments installed at a National Ecological Observatory (NEON) field site, aircraft flux measurements in combination with remote sensing data, as well as a software intercomparison. In conjunction with this study, the first two eddy4R packages and simple NEON EC data products are released publicly. While this proof-of-concept represents a significant advance, substantial work remains to arrive at the automated framework needed for the streaming generation of science-grade EC fluxes.

ANALISIS TEMPERATUR DAN UAP AIR BERBASIS SATELIT TERRA/AQUA (MODIS, LEVEL-2)

Terra and Aqua satellites that consist of multiple sensors including MODIS instruments, which is operated to detect the phenomena that exist on land, sea and atmosphere. Not a lot of data extracted especially for Indonesia region the associated with atmospheric data, because the product is still in the raw data (level-0). For data extraction of level-0 to level-2 needed software IMAPP (International MODIS/airs Processing Package) so displays some data atmospheric parameters including MOD 04 - Aerosol, MOD 05 - Total precipitable Water (Water Vapor), MOD 06 - Cloud, MOD 07 - Atmospheric Profiles, MOD 08 - gridded Atmospheric and MOD 35 in HDF4 format (Hierarchical Data Format-4) swath. This paper discussed only MOD07/MYD07 atmospheric profiles level-2 related parameters such as the temperature of the atmosphere at an altitude of 780 hPa and water vapor at a height of 700 hPa. This study aimed to analyze the phenomena in the atmosphere, based on extraction method Atmospheric Profiles in the resolution 1km, that consists of temperature and moisture level-2, in the format hdf4 daily swath into data daily and monthly grid in .dat format, in the period of December 2014, January, July, and August 2015, especially in the area of Indonesia. The comparison between the results of the extraction swath and grid data from Terra/Aqua MODIS, that parameter atmospheric for the temperature has R-sqare an average of 0.72 and water vapor 0.74, while the RMSE temperature and water vapor are 0.88 and 0.29. Abstrak Satelit Terra dan Aqua yang terdiri dari beberapa sensor diantaranya instrumen MODIS, yang dioperasikan untuk mendeteksi fenomena yang ada di darat, laut, dan atmosfer. Belum banyak data yang diekstrak khususnya untuk wilayah Indonesia yang terkait dengan data atmosfer, karena produk MODIS masih berupa data mentah (level-0). Untuk ekstraksi data dari level-0 menjadi level-2 dibutuhkan software International MODIS/AIRS Processing Package (IMAPP) sehingga menampilkan beberapa data parameter atmosfer diantaranya MOD 04 - Aerosol, MOD 05 - Total Precipitable Water (Water Vapor), MOD 06 - Cloud, MOD 07 - Atmospheric Profiles, MOD 08 - Gridded Atmospheric dan MOD 35 swath dalam format Hierarchical Data Format-4 (HDF4). Pada makalah ini yang dibahas hanya MOD07/MYD07 atmospheric profiles level-2 yang berkaitan dengan parameter atmosfer seperti temperatur pada ketinggian 780 hPa dan uap air pada ketinggian 700 hPa. Penelitian ini bertujuan untuk menganalisis hasil ekstraksi data Atmospheric Profiles dari format HDF4 swath harian menjadi data grided harian, bulanan dalam format .dat serta aplikasinya pada periode bulan Desember 2014, Januari, Juli, dan Agustus 2015, khususnya wilayah Indonesia dalam resolusi 1km yang terdiri dari temperatur dan uap air level-2. Perbandingan antara hasil ekstraksi data MODIS swath dan data MODIS grided Terra/Aqua untuk parameter temperatur atmosfer mempunyai R-sqare rata-rata 0.72 dan uap air 0.74, sedangkan RMSE untuk temperatur dan uap air sebesar 0.88 dan 0.29.