scholarly journals The Influence of Spectral Pretreatment on the Selection of Representative Calibration Samples for Soil Organic Matter Estimation Using Vis-NIR Reflectance Spectroscopy

2019 ◽  
Vol 11 (4) ◽  
pp. 450 ◽  
Author(s):  
Yi Liu ◽  
Yaolin Liu ◽  
Yiyun Chen ◽  
Yang Zhang ◽  
Tiezhu Shi ◽  
...  
Keyword(s):  
Organic Matter ◽  
Soil Organic Matter ◽  
Near Infrared ◽  
Sample Selection ◽  
Nir Spectroscopy ◽  
Selection Strategy ◽  
Least Squares Regression ◽  
Calibration Sets ◽  
Calibration Samples ◽  
Selection Of

In constructing models for predicting soil organic matter (SOM) by using visible and near-infrared (vis–NIR) spectroscopy, the selection of representative calibration samples is decisive. Few researchers have studied the inclusion of spectral pretreatments in the sample selection strategy. We collected 108 soil samples and applied six commonly used spectral pretreatments to preprocess soil spectra, namely, Savitzky–Golay (SG) smoothing, first derivative (FD), logarithmic function log(1/R), mean centering (MC), standard normal variate (SNV), and multiplicative scatter correction (MSC). Then, the Kennard–Stone (KS) strategy was used to select calibration samples based on the pretreated spectra, and the size of the calibration set varied from 10 samples to 86 samples (80% of the total samples). These calibration sets were employed to construct partial least squares regression models (PLSR) to predict SOM, and the built models were validated by a set of 21 samples (20% of the total samples). The results showed that 64−78% of the calibration sets selected by the inclusion of pretreatment demonstrated significantly better performance of SOM estimation. The average improved residual predictive deviations (ΔRPD) were 0.06, 0.13, 0.19, and 0.13 for FD, log(1/R), MSC, and SNV, respectively. Thus, we concluded that spectral pretreatment improves the sample selection strategy, and the degree of its influence varies with the size of the calibration set and the type of pretreatment.

Estimation of soil organic matter using proximal and satellite sensors after a wildfire in Mediterranean Croatia

2021 ◽  
Author(s):  
Iva Hrelja ◽  
Ivana Šestak ◽  
Igor Bogunović
Keyword(s):  
Organic Matter ◽  
Soil Organic Matter ◽  
Spectral Data ◽  
Near Infrared ◽  
Mean Squared Error ◽  
Nir Spectroscopy ◽  
Soil Samples ◽  
Least Squares Regression ◽  
Macro Scale ◽  
Sentinel 2

<p>Spectral data obtained from optical spaceborne sensors are being recognized as a valuable source of data that show promising results in assessing soil properties on medium and macro scale. Combining this technique with laboratory Visible-Near Infrared (VIS-NIR) spectroscopy methods can be an effective approach to perform robust research on plot scale to determine wildfire impact on soil organic matter (SOM) immediately after the fire. Therefore, the objective of this study was to assess the ability of Sentinel-2 superspectral data in estimating post-fire SOM content and comparison with the results acquired with laboratory VIS-NIR spectroscopy.</p><p>The study is performed in Mediterranean Croatia (44° 05’ N; 15° 22’ E; 72 m a.s.l.), on approximately 15 ha of fire affected mixed <em>Quercus ssp.</em> and <em>Juniperus ssp.</em> forest on Cambisols. A total of 80 soil samples (0-5 cm depth) were collected and geolocated on August 22<sup>nd</sup> 2019, two days after a medium to high severity wildfire. The samples were taken to the laboratory where soil organic carbon (SOC) content was determined via dry combustion method with a CHNS analyzer. SOM was subsequently calculated by using a conversion factor of 1.724. Laboratory soil spectral measurements were carried out using a portable spectroradiometer (350-1050 nm) on all collected soil samples. Two Sentinel-2 images were downloaded from ESAs Scientific Open Access Hub according to the closest dates of field sampling, namely August 31<sup>st</sup> and September 5<sup>th </sup>2019, each containing eight VIS-NIR and two SWIR (Short-Wave Infrared) bands which were extracted from bare soil pixels using SNAP software. Partial least squares regression (PLSR) model based on the pre-processed spectral data was used for SOM estimation on both datasets. Spectral reflectance data were used as predictors and SOM content was used as a response variable. The accuracy of the models was determined via Root Mean Squared Error of Prediction (RMSE<sub>p</sub>) and Ratio of Performance to Deviation (RPD) after full cross-validation of the calibration datasets.</p><p>The average post-fire SOM content was 9.63%, ranging from 5.46% minimum to 23.89% maximum. Models obtained from both datasets showed low RMSE<sub>p </sub>(Spectroscopy dataset RMSE<sub>p</sub> = 1.91; Sentinel-2 dataset RMSE<sub>p</sub> = 0.99). RPD values indicated very good predictions for both datasets (Spectrospcopy dataset RPD = 2.72; Sentinel-2 dataset RPD = 2.22). Laboratory spectroscopy method with higher spectral resolution provided more accurate results. Nonetheless, spaceborne method also showed promising results in the analysis and monitoring of SOM in post-burn period.</p><p><strong>Keywords:</strong> remote sensing, soil spectroscopy, wildfires, soil organic matter</p><p><strong>Acknowledgment: </strong>This work was supported by the Croatian Science Foundation through the project "Soil erosion and degradation in Croatia" (UIP-2017-05-7834) (SEDCRO). Aleksandra Perčin is acknowledged for her cooperation during the laboratory work.</p>

scholarly journals Exploring Appropriate Preprocessing Techniques for Hyperspectral Soil Organic Matter Content Estimation in Black Soil Area

2020 ◽  
Vol 12 (22) ◽  
pp. 3765
Author(s):  
Xitong Xu ◽  
Shengbo Chen ◽  
Zhengyuan Xu ◽  
Yan Yu ◽  
Sen Zhang ◽  
...  
Keyword(s):  
Organic Matter ◽  
Soil Organic Matter ◽  
Dimensionality Reduction ◽  
Near Infrared ◽  
Organic Matter Content ◽  
Nir Spectroscopy ◽  
Data Transformation ◽  
Black Soil ◽  
Locally Linear Embedding ◽  
Soil Area

Black soil in northeast China is gradually degraded and soil organic matter (SOM) content decreases at a rate of 0.5% per year because of the long-term cultivation. SOM content can be obtained rapidly by visible and near-infrared (Vis–NIR) spectroscopy. It is critical to select appropriate preprocessing techniques for SOM content estimation through Vis–NIR spectroscopy. This study explored three categories of preprocessing techniques to improve the accuracy of SOM content estimation in black soil area, and a total of 496 ground samples were collected from the typical black soil area at 0–15 cm in Hai Lun City, Heilongjiang Province, northeast of China. Three categories of preprocessing include denoising, data transformation and dimensionality reduction. For denoising, Svitzky-Golay filter (SGF), wavelet packet transform (WPT), multiplicative scatter correction (MSC), and none (N) were applied to spectrum of ground samples. For data transformation, fractional derivatives were allowed to vary from 0 to 2 with an increment of 0.2 at each step. For dimensionality reduction, multidimensional scaling (MDS) and locally linear embedding (LLE) were introduced and compared with principal component analysis (PCA), which was commonly used for dimensionality reduction of soil spectrum. After spectral pretreatments, a total of 132 partial least squares regression (PLSR) models were constructed for SOM content estimation. Results showed that SGF performed better than the other three denoising methods. Low-order derivatives can accentuate spectral features of soil for SOM content estimation; as the order increases from 0.8, the spectrum were more susceptible to spectral noise interferences. In most cases, 0.2–0.8 order derivatives exhibited the best estimation performance. Furthermore, PCA yielded the optimal predictability, the mean residual predictive deviation (RPD) and maximum RPD of the models using PCA were 1.79 and 2.60, respectively. The application of appropriate preprocessing techniques could improve the efficiency and accuracy of SOM content estimation, which is important for the protection of ecological and agricultural environment in black soil area.

Near-Infrared Spectroscopic Measurement of Organic Matter in Soil

2012 ◽  
Vol 499 ◽  
pp. 414-418
Author(s):  
Tao Pan ◽  
Zhen Tao Wu ◽  
Jie Mei Chen
Keyword(s):  
Organic Matter ◽  
Near Infrared ◽  
Rapid Determination ◽  
Nir Spectroscopy ◽  
Partial Least Square ◽  
Least Square ◽  
Modeling Prediction ◽  
Validation Set ◽  
Selection Of

Near-infrared (NIR) spectroscopy was successfully applied to chemical free and rapid determination of the organic matter in soil, and moving window partial least square (MWPLS) combining with Savitzky-Golay (SG) smoothing was used to the selection of NIR waveband. Thirty-five samples were randomly selected from all 97 collected soil samples as the validation set. The remaining 62 samples were divided into similar modeling calibration set (37 samples) and modeling prediction set (25 samples) based on partial least square cross-validation predictive bias (PLSPB). The selected waveband was 1896 nm to 2138 nm; the SG smoothing parameters and PLS factor OD, DP, NSP and F were 2, 6, 71 and 15, respectively; the modeling effect M-SEP and M-RPwere 0.219% and 0.944, respectively; the validating effect V-SEP and V-RPwere 0.243% and 0.878, respectively. The result provided a reliable NIR model and valuable references for designing specialized NIR instruments.

scholarly journals Quantitative Estimation of Organic Matter Content in Arid Soil Using Vis-NIR Spectroscopy Preprocessed by Fractional Derivative

2017 ◽  
Vol 2017 ◽  
pp. 1-9 ◽  
Author(s):  
Jingzhe Wang ◽  
Tashpolat Tiyip ◽  
Jianli Ding ◽  
Dong Zhang ◽  
Wei Liu ◽  
...  
Keyword(s):  
Organic Matter ◽  
Fractional Derivative ◽  
Near Infrared ◽  
Quantitative Estimation ◽  
Organic Matter Content ◽  
Nir Spectroscopy ◽  
Predictive Performance ◽  
Matter Content ◽  
Lake Basin ◽  
Least Squares Regression

Soil organic matter (SOM) content is an important index to measure the level of soil function and soil quality. However, conventional studies on estimation of SOM content concerned about the classic integer derivative of spectral data, while the fractional derivative information was ignored. In this research, a total of 103 soil samples were collected in the Ebinur Lake basin, Xinjiang Uighur Autonomous Region, China. After measuring the Vis-NIR (visible and near-infrared) spectroscopy and SOM content indoor, the raw reflectance and absorbance were treated by fractional derivative from 0 to 2nd order (order interval 0.2). Partial least squares regression (PLSR) was applied for model calibration, and five commonly used precision indices were used to assess the performance of these 22 models. The results showed that with the rise of order, these parameters showed the increasing or decreasing trends with vibration and reached the optimal values at the fractional order. A most robust model was calibrated based on 1.8 order derivative of R, with the lowest RMSEC (3.35 g kg−1) and RMSEP (2.70 g kg−1) and highest Rc2 (0.92), Rp2 (0.91), and RPD (3.42 > 3.0). This model had excellent predictive performance of estimating SOM content in the study area.

scholarly journals EXPLORE THE CHARACTER OF SOIL SPECTRAL REFLECTANCE RELATE TO THE SOIL ORGANIC MATTER CONTENT

2019 ◽  
Vol 8 (3) ◽  
pp. 214
Author(s):  
Sari Virgawati ◽  
Muhjidin Mawardi ◽  
Lilik Sutiarso ◽  
Sakae Shibusawa ◽  
Hendrik Segah ◽  
...  
Keyword(s):  
Organic Matter ◽  
Soil Organic Matter ◽  
Spectral Reflectance ◽  
Near Infrared ◽  
Diffuse Reflectance Spectroscopy ◽  
Low Cost ◽  
Organic Matter Content ◽  
Nir Spectroscopy ◽  
Matter Content ◽  
Strong Negative Correlation

ABSTRACTThe visible and near-infrared (Vis-NIR) diffuse reflectance spectroscopy has emerged as a rapid and low-cost tool for extensive investigation of soil properties. The objective of this research was to explore how significant the relationship between the soil spectral reflectance and soil organic matter (SOM) content. Some soil samples in Yogyakarta were taken for SOM content and spectroscopy measurement. The SOM was analyzed using Walkley and Black method, while the spectral reflectance was determined using ASD Field-spectrophotometer by scanned the sample with Vis-NIR spectrum. Pearson’s coefficient showed that there was a strong negative correlation between SOM and soil spectral of certain wavelengths. Soil with less organic matter content performed high reflectance. Keywords: Soil organic matter; Vis-NIR spectroscopy; soil reflectance; Pearson’s correlation coefficient.

On the selection of samples for multivariate regression analysis: application to near-infrared (NIR) calibration models for the prediction of pulp yield in Eucalyptus nitens

2008 ◽  
Vol 38 (10) ◽  
pp. 2626-2634 ◽  
Author(s):  
Christian R. Mora ◽  
Laurence R. Schimleck
Keyword(s):  
Spectral Data ◽  
Computer Aided Design ◽  
Near Infrared ◽  
Cross Validation ◽  
Sample Selection ◽  
Pulp Yield ◽  
Eucalyptus Nitens ◽  
Calibration Models ◽  
Calibration Sets ◽  
Measured Property

The effects of using reduced calibration sets on the development of near-infrared (NIR) calibration models for the prediction of kraft pulp yield in Eucalyptus nitens (Dean & Maiden) Maiden trees were explored. Three selection techniques based on NIR spectral data (CADEX (computer-aided design of experiments), DUPLEX, and SELECT algorithms) and one selection method based on a measured property (RANKING algorithm) were used for analysis and compared against a model using all data. The effect of using calibration sets of different sizes was also evaluated. All sample-selection methods resulted in models of similar performance compared with the model fitted using all samples. For calibration purposes, RANKING selection resulted in models with the lowest errors of cross-validation, followed by the DUPLEX, CADEX, and SELECT methods. In terms of validation, the SELECT and CADEX methods resulted in lower errors of prediction compared with the DUPLEX and RANKING algorithms. In general, cross-validation and prediction errors decreased as the number of calibration samples increased. These results show that it is possible to obtain adequate NIR calibration models with a reduced number of samples allowing the remaining samples to be used for model validation and that sample selection based on NIR spectral data alone is as successful as selection based on a measured property.

Prediction of Stilbene Content from Heartwood Increment Cores of Scots Pine Using near Infrared Spectroscopy Methodology

2016 ◽  
Vol 24 (6) ◽  
pp. 517-528 ◽  
Author(s):  
Susanna Pulkka ◽  
Vincent Segura ◽  
Anni Harju ◽  
Tarja Tapanila ◽  
Johanna Tanner ◽  
...  
Keyword(s):  
Scots Pine ◽  
High Throughput ◽  
Near Infrared ◽  
Nir Spectroscopy ◽  
Tree Breeding ◽  
Decay Resistance ◽  
Increment Core ◽  
Automatic Selection ◽  
Non Destructive ◽  
Selection Of

High-throughput and non-destructive methods for quantifying the content of the stilbene compounds of Scots pine ( Pinus sylvestris L.) heartwood are needed in the breeding for decay resistance of heartwood timber. In this study, near infrared (NIR) spectroscopy calibrations were developed for a large collection of solid heartwood increment core samples in order to predict the amount of the stilbene pinosylvin (PS), its monomethyl ether (PSM) and their sum (STB). The resulting models presented quite accurate predictions in an independent validation set with R2V values ranging between 0.79 and 0.91. The accuracy of the models strongly depended on the chemical being calibrated, with the lowest accuracy for PS, intermediate accuracy for PSM and highest accuracy for STB. The effect of collecting one, two or more (up to five) spectra per sample on the calibration models was studied and it was found that averaging multiple spectra yielded better accuracy as it may account for the heterogeneity of wood along the increment core within and between rings. Several statistical pretreatments of the spectra were tested and an automatic selection of wavenumbers prior to calibration. Without the automatic selection of wavenumbers, a first derivative of normalised spectra yielded the best accuracies, whereas after the automatic selection of wavenumbers, no particular statistical pretreatment appeared to yield better results than any other. Finally, the automatic selection of wavenumbers slightly improved the accuracy of the models for all traits. These results demonstrate the potential of NIR spectroscopy as a high-throughput and non-destructive phenotyping technique in tree breeding for the improvement of decay resistance in heartwood timber.

Ability of Vis-PIR spectroscopy to monitor changes in organic carbon of loamy soils at two depths

2021 ◽  
Author(s):  
Hayfa Zayani ◽  
Youssef Fouad ◽  
Didier Michot ◽  
Zeineb Kassouk ◽  
Zohra Lili-Chabaane ◽  
...  
Keyword(s):  
Organic Carbon ◽  
Near Infrared ◽  
Full Range ◽  
Nir Spectroscopy ◽  
Soil Samples ◽  
Calibration Data ◽  
Least Squares Regression ◽  
Data Set ◽  
Soil Carbon Stocks ◽  
The Mean

<p>Visible-Near Infrared (Vis-NIR) spectroscopy has proven its efficiency in predicting several soil properties such as soil organic carbon (SOC) content. In this preliminary study, we explored the ability of Vis-NIR to assess the temporal evolution of SOC content. Soil samples were collected in a watershed (ORE AgrHys), located in Brittany (Western France). Two sampling campaigns were carried out 5 years apart: in 2013, 198 soil samples were collected respectively at two depths (0-15 and 15-25 cm) over an area of 1200 ha including different land use and land cover; in 2018, 111 sampling points out of 198 of 2013 were selected and soil samples were collected from the same two depths. Whole samples were analyzed for their SOC content and were scanned for their reflectance spectrum. Spectral information was acquired from samples sieved at 2 mm fraction and oven dried at 40°C, 24h prior to spectra acquisition, with a full range Vis-NIR spectroradiometer ASD Fieldspec®3. Data set of 2013 was used to calibrate the SOC content prediction model by the mean of Partial Least Squares Regression (PLSR). Data set of 2018 was therefore used as test set. Our results showed that the variation ∆SOC<sub>obs</sub><sub></sub>obtained from observed values in 2013 and 2018 (∆SOC<sub>obs</sub> = Observed SOC (2018) - Observed SOC (2013)) is ranging from 0.1 to 25.9 g/kg. Moreover, our results showed that the prediction performance of the calibrated model was improved by including 11 spectra of 2018 in the 2013 calibration data set (R²= 0.87, RMSE = 5.1 g/kg and RPD = 1.92). Furthermore, the comparison of predicted and observed ∆SOC between 2018 and 2013 showed that 69% of the variations were of the same sign, either positive or negative. For the remaining 31%, the variations were of opposite signs but concerned mainly samples for which ∆SOCobs is less than 1,5 g/kg. These results reveal that Vis-NIR spectroscopy was potentially appropriate to detect variations of SOC content and are encouraging to further explore Vis-NIR spectroscopy to detect changes in soil carbon stocks.</p>

Review: The evolution of chemometrics coupled with near infrared spectroscopy for fruit quality evaluation

2022 ◽  
pp. 096703352110572
Author(s):  
Nicholas T Anderson ◽  
Kerry B Walsh
Keyword(s):  
Near Infrared ◽  
Nir Spectroscopy ◽  
Short Wave ◽  
Least Squares Regression ◽  
Machine Learning Methods ◽  
Transmission Geometry ◽  
Modelling Techniques ◽  
Test Sets ◽  
Model Robustness

Short wave near infrared (NIR) spectroscopy operated in a partial or full transmission geometry and a point spectroscopy mode has been increasingly adopted for evaluation of quality of intact fruit, both on-tree and on-packing lines. The evolution in hardware has been paralleled by an evolution in the modelling techniques employed. This review documents the range of spectral pre-treatments and modelling techniques employed for this application. Over the last three decades, there has been a shift from use of multiple linear regression to partial least squares regression. Attention to model robustness across seasons and instruments has driven a shift to machine learning methods such as artificial neural networks and deep learning in recent years, with this shift enabled by the availability of large and diverse training and test sets.

scholarly journals Deep Learning Application for Predicting Soil Organic Matter Content by VIS-NIR Spectroscopy

2019 ◽  
Vol 2019 ◽  
pp. 1-11 ◽  
Author(s):  
Zhe Xu ◽  
Xiaomin Zhao ◽  
Xi Guo ◽  
Jiaxin Guo
Keyword(s):  
Organic Matter ◽  
Deep Learning ◽  
Infrared Spectroscopy ◽  
Soil Organic Matter ◽  
Near Infrared Spectroscopy ◽  
Spectral Reflectance ◽  
Near Infrared ◽  
Spectral Variation ◽  
Full Spectrum ◽  
Reflectance Data

Deep learning is characterized by its strong ability of data feature extraction. This method can provide unique advantages when applying it to visible and near-infrared spectroscopy for predicting soil organic matter (SOM) content in those cases where the SOM content is negatively correlated with the spectral reflectance of soil. This study relied on the SOM content data of 248 red soil samples and their spectral reflectance data of 400–2450 nm in Fengxin County, Jiangxi Province (China) to meet three objectives. First, a multilayer perceptron and two convolutional neural networks (LeNet5 and DenseNet10) were used to predict the SOM content based on spectral variation and variable selection, and the outcomes were compared with that from the traditional back-propagation neural network (BPN). Second, the four methods were applied to full-spectrum modeling to test the difference to selected feature variables. Finally, the potential of direct modeling was evaluated using spectral reflectance data without any spectral variation. The results of prediction accuracy showed that deep learning performed better at predicting the SOM content than did the traditional BPN. Based on full-spectrum data, deep learning was able to obtain more feature information, thus achieving better and more stable results (i.e., similar average accuracy and far lower standard deviation) than those obtained through variable selection. DenseNet achieved the best prediction result, with a coefficient of determination (R2) = 0.892 ± 0.004 and a ratio of performance to deviation (RPD) = 3.053 ± 0.056 in validation. Based on DenseNet, the application of spectral reflectance data (without spectral variation) produced robust results for application-level purposes (validation R2 = 0.853 ± 0.007 and validation RPD = 2.639 ± 0.056). In conclusion, deep learning provides an effective approach to predict the SOM content by visible and near-infrared spectroscopy and DenseNet is a promising method for reducing the amount of data preprocessing.

Sign in / Sign up

Export Citation Format

Share Document