scholarly journals An Efficient Algorithm for Ill-Conditioned Separable Nonlinear Least Squares

2021 ◽  
Vol 2021 ◽  
pp. 1-8
Author(s):  
Jiayan Wang ◽  
Lanlan Guo ◽  
Zongmin Li ◽  
Xueqin Wang ◽  
Zhengqing Fu
Keyword(s):  
Least Squares ◽  
Rbf Neural Network ◽  
Solution Process ◽  
Nonlinear Least Squares ◽  
Projection Algorithm ◽  
Tikhonov Regularization Method ◽  
Model Parameters ◽  
Nonlinear Parameters ◽  
Schmidt Orthogonalization ◽  
Quantitative Indicators

For separable nonlinear least squares models, a variable projection algorithm based on matrix factorization is studied, and the ill-conditioning of the model parameters is considered in the specific solution process of the model. When the linear parameters are estimated, the Tikhonov regularization method is used to solve the ill-conditioned problems. When the nonlinear parameters are estimated, the QR decomposition, Gram–Schmidt orthogonalization decomposition, and SVD are applied in the Jacobian matrix. These methods are then compared with the method in which the variables are not separated. Numerical experiments are performed using RBF neural network data, and the experimental results are analyzed in terms of both qualitative and quantitative indicators. The results show that the proposed algorithms are effective and robust.

scholarly journals An Improved Tikhonov-Regularized Variable Projection Algorithm for Separable Nonlinear Least Squares

Axioms ◽  
2021 ◽  
Vol 10 (3) ◽  
pp. 196
Author(s):  
Hua Guo ◽  
Guolin Liu ◽  
Luyao Wang
Keyword(s):  
Least Squares ◽  
Tikhonov Regularization ◽  
Regularization Method ◽  
Nonlinear Least Squares ◽  
Singular Values ◽  
Exponential Model ◽  
Singular Value ◽  
Projection Algorithm ◽  
Tikhonov Regularization Method ◽  
Variable Projection

In this work, we investigate the ill-conditioned problem of a separable, nonlinear least squares model by using the variable projection method. Based on the truncated singular value decomposition method and the Tikhonov regularization method, we propose an improved Tikhonov regularization method, which neither discards small singular values, nor treats all singular value corrections. By fitting the Mackey–Glass time series in an exponential model, we compare the three regularization methods, and the numerically simulated results indicate that the improved regularization method is more effective at reducing the mean square error of the solution and increasing the accuracy of unknowns.

scholarly journals A Method for Solving LiDAR Waveform Decomposition Parameters Based on a Variable Projection Algorithm

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-13
Author(s):  
Ke Wang ◽  
Guolin Liu ◽  
Qiuxiang Tao ◽  
Luyao Wang ◽  
Yang Chen
Keyword(s):  
Parameter Estimation ◽  
Least Squares ◽  
Gaussian Function ◽  
Nonlinear Least Squares ◽  
Projection Algorithm ◽  
Nonlinear Parameters ◽  
Least Squares Problem ◽  
Variable Projection ◽  
Nonlinear Least Squares Problem ◽  
Lidar Waveform

Light detection and ranging (LiDAR) is commonly used to create high-resolution maps; however, the efficiency and convergence of parameter estimation are difficult. To address this issue, we evaluated the structural characteristics of received LiDAR signals by decomposing them into Gaussian functions and applied the variable projection algorithm of the separable nonlinear least-squares problem to the process of waveform fitting. First, using a variable projection algorithm, we separated the linear (amplitude) and nonlinear (center position and width) parameters in the Gaussian function model; the linear parameters are expressed with nonlinear parameters by the function. Thereafter, the optimal estimation of the characteristic parameters of the Gaussian function components was transformed into a least-squares problem only comprising nonlinear parameters. Finally, the Levenberg–Marquardt algorithm was used to solve these nonlinear parameters, whereas the linear parameters were calculated simultaneously in each iteration, and the estimation results satisfying the nonlinear least-square criterion were obtained. Five groups of waveform decomposition simulation data and ICESat/GLAS satellite LiDAR waveform data were used for the parameter estimation experiments. During the experiments, for the same accuracy, the separable nonlinear least-squares optimization method required fewer iterations and lesser calculation time than the traditional method of not separating parameters; the maximum number of iterations was reached before the traditional method converged to the optimal estimate. The method of separating variables only required 14 iterations to obtain the optimal estimate, reducing the computational time from 1128 s to 130 s. Therefore, the application of the separable nonlinear least-squares problem can improve the calculation efficiency and convergence speed of the parameter solution process. It can also provide a new method for parameter estimation in the Gaussian model for LiDAR waveform decomposition.

scholarly journals Quantifying soil hydraulic properties and their uncertainties by modified GLUE method

2017 ◽  
Vol 31 (3) ◽  
pp. 433-445
Author(s):  
Yifan Yan ◽  
Jianli Liu ◽  
Jiabao Zhang ◽  
Xiaopeng Li ◽  
Yongchao Zhao
Keyword(s):  
Confidence Interval ◽  
Least Squares ◽  
Water Content ◽  
Soil Water ◽  
Nonlinear Least Squares ◽  
Uncertainty Estimation ◽  
Model Parameters ◽  
Retention Curve ◽  
Least Squares Algorithm ◽  
Soil Hydraulic

AbstractNonlinear least squares algorithm is commonly used to fit the evaporation experiment data and to obtain the ‘optimal’ soil hydraulic model parameters. But the major defects of nonlinear least squares algorithm include non-uniqueness of the solution to inverse problems and its inability to quantify uncertainties associated with the simulation model. In this study, it is clarified by applying retention curve and a modified generalised likelihood uncertainty estimation method to model calibration. Results show that nonlinear least squares gives good fits to soil water retention curve and unsaturated water conductivity based on data observed by Wind method. And meanwhile, the application of generalised likelihood uncertainty estimation clearly demonstrates that a much wider range of parameters can fit the observations well. Using the ‘optimal’ solution to predict soil water content and conductivity is very risky. Whereas, 95% confidence interval generated by generalised likelihood uncertainty estimation quantifies well the uncertainty of the observed data. With a decrease of water content, the maximum of nash and sutcliffe value generated by generalised likelihood uncertainty estimation performs better and better than the counterpart of nonlinear least squares. 95% confidence interval quantifies well the uncertainties and provides preliminary sensitivities of parameters.

Integrating Parameter Estimation Solutions From the Time and Time-Scale Domains

2009 ◽  
Author(s):  
James R. McCusker ◽  
Kourosh Danai
Keyword(s):  
Parameter Estimation ◽  
Least Squares ◽  
Time Scale ◽  
Prediction Error ◽  
Nonlinear Least Squares ◽  
Integrated Approach ◽  
Parameter Estimates ◽  
Model Parameters ◽  
Single Model ◽  
Iterative Estimation

A method of parameter estimation was recently introduced that separately estimates each parameter of the dynamic model [1]. In this method, regions coined as parameter signatures, are identified in the time-scale domain wherein the prediction error can be attributed to the error of a single model parameter. Based on these single-parameter associations, individual model parameters can then be estimated for iterative estimation. Relative to nonlinear least squares, the proposed Parameter Signature Isolation Method (PARSIM) has two distinct attributes. One attribute of PARSIM is to leave the estimation of a parameter dormant when a parameter signature cannot be extracted for it. Another attribute is independence from the contour of the prediction error. The first attribute could cause erroneous parameter estimates, when the parameters are not adapted continually. The second attribute, on the other hand, can provide a safeguard against local minima entrapments. These attributes motivate integrating PARSIM with a method, like nonlinear least-squares, that is less prone to dormancy of parameter estimates. The paper demonstrates the merit of the proposed integrated approach in application to a difficult estimation problem.

scholarly journals Efficient Parameters Estimation Method for the Separable Nonlinear Least Squares Problem

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-16 ◽  
Author(s):  
Ke Wang ◽  
Guolin Liu ◽  
Qiuxiang Tao ◽  
Min Zhai
Keyword(s):  
Least Squares ◽  
Least Squares Method ◽  
Nonlinear Least Squares ◽  
Coefficient Matrix ◽  
Nonlinear Parameters ◽  
Nonlinear Least Squares Problem ◽  
Ill Posed ◽  
Characteristic Values ◽  
Parameter Separation ◽  
Fitting Problem

In this work, we combine the special structure of the separable nonlinear least squares problem with a variable projection algorithm based on singular value decomposition to separate linear and nonlinear parameters. Then, we propose finding the nonlinear parameters using the Levenberg–Marquart (LM) algorithm and either solve the linear parameters using the least squares method directly or by using an iteration method that corrects the characteristic values based on the L-curve, according to whether or not the nonlinear function coefficient matrix is ill posed. To prove the feasibility of the proposed method, we compared its performance on three examples with that of the LM method without parameter separation. The results show that (1) the parameter separation method reduces the number of iterations and improves computational efficiency by reducing the parameter dimensions and (2) when the coefficient matrix of the linear parameters is well-posed, using the least squares method to solve the fitting problem provides the highest fitting accuracy. When the coefficient matrix is ill posed, the method of correcting characteristic values based on the L-curve provides the most accurate solution to the fitting problem.

scholarly journals Uncertainty Quantification in Machine Learning and Nonlinear Least Squares Regression Models

2021 ◽  
Author(s):  
Ni Zhan ◽  
John Kitchin
Keyword(s):  
Machine Learning ◽  
Least Squares ◽  
Uncertainty Quantification ◽  
Automatic Differentiation ◽  
Nonlinear Least Squares ◽  
Delta Method ◽  
Training Data ◽  
Model Parameters ◽  
Uncertainty Measure ◽  
Least Squares Regression

Machine learning (ML) models are valuable research tools for making accurate predictions. However, ML models often unreliably extrapolate outside their training data. We propose an uncertainty quantification method for ML models (and generally for other nonlinear models) with parameters trained by least squares regression. The uncertainty measure is based on the multiparameter delta method from statistics, which gives the standard error of the prediction. The uncertainty measure requires the gradient of the model prediction and the Hessian of the loss function, both with respect to model parameters. Both the gradient and Hessian can be readily obtained from most ML software frameworks by automatic differentiation. We show that the uncertainty measure is larger for input space regions that are not part of the training data. Therefore this method can be used to identify extrapolation and to aid in selecting training data or assessing model reliability.

Mixing efficiency as estimated by nonlinear least squares

1983 ◽  
Vol 10 (4) ◽  
pp. 703-712
Author(s):  
David T. Chapman
Keyword(s):  
Least Squares ◽  
Dispersion Model ◽  
Nonlinear Least Squares ◽  
Statistical Technique ◽  
Sum Of Squares ◽  
Axial Dispersion ◽  
Mixing Efficiency ◽  
Parameter Estimates ◽  
Model Parameters ◽  
Washout Curves

The suitability of a statistical technique known as nonlinear least squares for use in estimating mixing coefficients was evaluated by fitting models to residence time distribution curves. The washout curves were generated by adding slug inputs of tracers to three different reactors. Each of the reactors, used to treat wastewaters, was a different size and represented a different degree of mixing.Three models, described in the paper, were examined for use in conjuction with the nonlinear least squares technique. They included the axial dispersion, N-tanks-in-series, and Cholette–Cloutier models. The form of the equation for the axial dispersion model depends on the boundary conditions for the reactor being studied. For reactors which cannot be classified as "open" vessels, the required analytical solutions either do not exist or are not suitable for use with the nonlinear least squares technique.Mixing coefficients for the N-tanks and Chollette–Cloutier models were obtained from the tracer washout curves for the three reactors. The residual sum of squares based on nonlinear least squares estimates for the model parameters was compared with the sum of squares obtained using more conventional methods for estimating the parameters. The existence of trailing tails on the tracer curves resulted in misleading parameter estimates for the two models using conventional techniques. Keywords: mixing, least squares, tracer, dispersion, short-circuiting, deadspace.

scholarly journals Tikhonov Regularized Variable Projection Algorithms for Separable Nonlinear Least Squares Problems

Complexity ◽  
2019 ◽  
Vol 2019 ◽  
pp. 1-9 ◽  
Author(s):  
Zhengqing Fu ◽  
Lanlan Guo
Keyword(s):  
Parameter Estimation ◽  
Least Squares ◽  
Nonlinear Least Squares ◽  
Real Data ◽  
Estimation Methods ◽  
Linear Parameter ◽  
Nonlinear Parameters ◽  
Marquardt Algorithm ◽  
Tikhonov Regularisation ◽  
Parameter Estimation Methods

This paper considers the classical separable nonlinear least squares problem. Such problems can be expressed as a linear combination of nonlinear functions, and both linear and nonlinear parameters are to be estimated. Among the existing results, ill-conditioned problems are less often considered. Hence, this paper focuses on an algorithm for ill-conditioned problems. In the proposed linear parameter estimation process, the sensitivity of the model to disturbance is reduced using Tikhonov regularisation. The Levenberg–Marquardt algorithm is used to estimate the nonlinear parameters. The Jacobian matrix required by LM is calculated by the Golub and Pereyra, Kaufman, and Ruano methods. Combining the nonlinear and linear parameter estimation methods, three estimation models are obtained and the feasibility and stability of the model estimation are demonstrated. The model is validated by simulation data and real data. The experimental results also illustrate the feasibility and stability of the model.

Periodicity analysis and a model structure for consumer behavior on hotel online search interest in the US

2017 ◽  
Vol 29 (5) ◽  
pp. 1486-1500 ◽  
Author(s):  
Juan Liu ◽  
Xue Li ◽  
Ya Guo
Keyword(s):  
Consumer Behavior ◽  
Least Squares ◽  
Nonlinear Least Squares ◽  
Model Parameters ◽  
Model Structure ◽  
Online Search ◽  
Content Type ◽  
The Usa ◽  
Frequency Components ◽  
Least Squares Algorithm

Purpose This paper aims to analyze and model consumer behavior on hotel online search interest in the USA. Design/methodology/approach Discrete Fourier transform was used to analyze the periodicity of hotel search behavior in the USA by using Google Trends data. Based on the obtained frequency components, a model structure was proposed to describe the search interest. A separable nonlinear least squares algorithm was developed to fit the data. Findings It was found that the major dynamics of the search interest was composed of nine frequency components. The developed separable nonlinear least squares algorithm significantly reduced the number of model parameters that needed to be estimated. The fitting results indicated that the model structure could fit the data well (average error 0.575 per cent). Practical implications Knowledge of consumer behavior on online search is critical to marketing decision because search engine has become an important tool for customers to find hotels. This work is thus very useful to marketing strategy. Originality/value This research is the first work on analyzing and modeling consumer behavior on hotel online search interest.

Sign in / Sign up

Export Citation Format

Share Document