scholarly journals Analytical Derivation of Nonlinear Spectral Effects and 1/f Scaling Artifact in Signal Processing of Real-World Data

2017 ◽  
Vol 29 (7) ◽  
pp. 2004-2020 ◽  
Author(s):  
Claudia Lainscsek ◽  
Lyle E. Muller ◽  
Aaron L. Sampson ◽  
Terrence J. Sejnowski
Keyword(s):  
Signal Processing ◽  
Real World ◽  
Time Series Data ◽  
Time Window ◽  
Low Frequency ◽  
Series Data ◽  
Infinite Length ◽  
Scaling Exponent ◽  
Sampled Data ◽  
Frequency Components

In estimating the frequency spectrum of real-world time series data, we must violate the assumption of infinite-length, orthogonal components in the Fourier basis. While it is widely known that care must be taken with discretely sampled data to avoid aliasing of high frequencies, less attention is given to the influence of low frequencies with period below the sampling time window. Here, we derive an analytic expression for the side-lobe attenuation of signal components in the frequency domain representation. This expression allows us to detail the influence of individual frequency components throughout the spectrum. The first consequence is that the presence of low-frequency components introduces a 1/f[Formula: see text] component across the power spectrum, with a scaling exponent of [Formula: see text]. This scaling artifact could be composed of diffuse low-frequency components, which can render it difficult to detect a priori. Further, treatment of the signal with standard digital signal processing techniques cannot easily remove this scaling component. While several theoretical models have been introduced to explain the ubiquitous 1/f[Formula: see text] scaling component in neuroscientific data, we conjecture here that some experimental observations could be the result of such data analysis procedures.

scholarly journals An Interpretable Early Dynamic Sequential Predictor for Sepsis-Induced Coagulopathy Progression in the Real-World Using Machine Learning

2021 ◽  
Vol 8 ◽  
Author(s):  
Ruixia Cui ◽  
Wenbo Hua ◽  
Kai Qu ◽  
Heran Yang ◽  
Yingmu Tong ◽  
...  
Keyword(s):  
Machine Learning ◽  
Real World ◽  
Time Series Data ◽  
Time Window ◽  
Medical Center ◽  
Characteristic Curve ◽  
Series Data ◽  
Gradient Boosting ◽  
Early Management ◽  
Extreme Gradient Boosting

Sepsis-associated coagulation dysfunction greatly increases the mortality of sepsis. Irregular clinical time-series data remains a major challenge for AI medical applications. To early detect and manage sepsis-induced coagulopathy (SIC) and sepsis-associated disseminated intravascular coagulation (DIC), we developed an interpretable real-time sequential warning model toward real-world irregular data. Eight machine learning models including novel algorithms were devised to detect SIC and sepsis-associated DIC 8n (1 ≤ n ≤ 6) hours prior to its onset. Models were developed on Xi'an Jiaotong University Medical College (XJTUMC) and verified on Beth Israel Deaconess Medical Center (BIDMC). A total of 12,154 SIC and 7,878 International Society on Thrombosis and Haemostasis (ISTH) overt-DIC labels were annotated according to the SIC and ISTH overt-DIC scoring systems in train set. The area under the receiver operating characteristic curve (AUROC) were used as model evaluation metrics. The eXtreme Gradient Boosting (XGBoost) model can predict SIC and sepsis-associated DIC events up to 48 h earlier with an AUROC of 0.929 and 0.910, respectively, and even reached 0.973 and 0.955 at 8 h earlier, achieving the highest performance to date. The novel ODE-RNN model achieved continuous prediction at arbitrary time points, and with an AUROC of 0.962 and 0.936 for SIC and DIC predicted 8 h earlier, respectively. In conclusion, our model can predict the sepsis-associated SIC and DIC onset up to 48 h in advance, which helps maximize the time window for early management by physicians.

Data Preprocessing

2018 ◽  
Author(s):  
Ray Huffaker ◽  
Marco Bittelli ◽  
Rodolfo Rosa
Keyword(s):  
Stochastic Process ◽  
Time Series ◽  
Signal Processing ◽  
Real World ◽  
Null Hypothesis ◽  
Time Series Data ◽  
Nonlinear Behavior ◽  
Surrogate Data ◽  
Series Data ◽  
Linear Behavior

Successful reconstruction of a shadow attractor provides preliminary empirical evidence that a signal isolated from observed time series data may be generated by deterministic dynamics. However, because we cannot reasonably expect signal processing to purge the signal of all noise in practice, and because noisy linear behavior can be visually indistinguishable from nonlinear behavior, the possibility remains that noticeable regularity detected in a shadow attractor may be fortuitously reconstructed from data generated by a linear-stochastic process. This chapter investigates how we can test this null hypothesis using surrogate data testing. The combination of a noticeably regular shadow attractor, along with strong statistical rejection of fortuitous regularity, increases the probability that observed data are generated by deterministic real-world dynamics.

scholarly journals Reply to comment by Vallée et al. on “Earthquake-induced prompt gravity signals identified in dense array data in Japan”

2019 ◽  
Vol 71 (1) ◽  
Author(s):  
Masaya Kimura ◽  
Nobuki Kame ◽  
Shingo Watada ◽  
Makiko Ohtani ◽  
Akito Araya ◽  
...  
Keyword(s):  
Signal Processing ◽  
Time Window ◽  
Statistical Significance ◽  
Low Frequency ◽  
Maximum Amplitude ◽  
P Wave ◽  
Full Account ◽  
Frequency Components ◽  
Input Acceleration ◽  
Broadband Seismograms

AbstractDensity perturbations accompanying seismic waves are expected to generate prompt gravity perturbations preceding the arrival of P-waves. Vallée et al. (Science 358:1164–1168, 2017, https://doi.org/10.1126/science.aao0746) reported the detection of such pre-P-wave signals in broadband seismograms during the 2011 Tohoku-oki earthquake. Kimura et al. (Earth Planets Space 71:27, 2019, https://doi.org/10.1186/s40623-019-1006-x) considered that their detection involved some uncertain points, including a concern regarding their signal processing procedure. Specifically, to remove the instrumental response, Vallée et al. (2017) applied acausal deconvolution to the seismograms truncated at the P-wave arrivals. Generally, acausal deconvolution produces artifacts at the edge of the time window. However, they did not present quantitative assessment whether the detected signals were artifacts due to the signal processing. To avoid this concern, Kimura et al. (2019) employed another procedure that eliminated acausal processes, resulting in the detection of a pre-P-wave signal with a statistical significance of 7σ in stacked broadband seismograms. Subsequently, Vallée et al. (Earth Planets Space 71:51, 2019, https://doi.org/10.1186/s40623-019-1030-x) commented that the procedure employed by Kimura et al. (2019) for the signal detection was inappropriate because it dismissed the low-frequency components of data. Although we admit the loss of low-frequency components in the data in Kimura et al. (2019), Vallée et al. (2019) have not yet provided a full account of the validity of their own procedure. Here, we assessed the validity of the procedure employed by Vallée et al. (2017) by quantitatively evaluating the magnitude of the acausal artifacts. First, we investigated how the input acceleration waveform, having an ideal signal-like shape, was distorted by their procedure. Their acausal deconvolution indeed generated a large-amplitude terminal artifact; however, it was removed by the causal band-pass filtering performed after the deconvolution and consequently became negligible. Next, we constrained the maximum amplitude of the artifact due to the noise in a seismogram and showed that it was sufficiently small compared to the reported signal amplitudes. These results suggest that the signal waveforms seen after their procedure were not artifacts but were representing the input acceleration with sufficient accuracy. Namely, their procedure well functions as a detection method for pre-P-wave signals. In the context of this validation, we replied to the comments of Vallée et al. (2019).

Theoretical Consideration on the Application of Continuous Wavelet Transform to Time-Series Processing for Magnetotelluric Data

2020 ◽  
Author(s):  
Hiroki Ogawa ◽  
Yuki Hama ◽  
Koichi Asamori ◽  
Takumi Ueda
Keyword(s):  
Time Series ◽  
Fourier Transform ◽  
Data Processing ◽  
Time Series Data ◽  
Series Data ◽  
Short Time Fourier Transform ◽  
Numerical Errors ◽  
Spectral Transform ◽  
Frequency Components ◽  
Short Time

Abstract In the magnetotelluric (MT) method, the responses of the natural electromagnetic fields are evaluated by transforming time-series data into spectral data and calculating the apparent resistivity and phase. The continuous wavelet transform (CWT) can be an alternative to the short-time Fourier transform, and the applicability of CWT to MT data has been reported. There are, however, few cases of considering the effect of numerical errors derived from spectral transform on MT data processing. In general, it is desirable to adopt a window function narrow in the time domain for higher-frequency components and one in the frequency domain for lower-frequency components. In conducting the short-time Fourier transform, because the size of the window function is fixed unless the time-series data are decimated, there might be difference between the calculated MT responses and the true ones due to the numerical errors. Meanwhile, CWT can strike a balance between the resolution of the time and frequency domains by magnifying or reducing the wavelet, according to the value of frequency. Although the types of wavelet functions and their parameters influence the resolution of time and frequency, those calculation settings of CWT are often determined empirically. In this study, focusing on the frequency band between 0.001 Hz and 10 Hz, we demonstrated the superiority of utilizing CWT in MT data processing and determined its proper calculation settings in terms of restraining the numerical errors caused by the spectral transform of time-series data. The results obtained with the short-time Fourier transform accompanied with gradual decimation of the time-series data, called cascade decimation, were compared with those of CWT. The shape of the wavelet was changed by using different types of wavelet functions or their parameters, and the respective results of data processing were compared. Through these experiments, this study indicates that CWT with the complex Morlet function with its wavelet parameter k set to 6 ≤ k < 10 will be effective in restraining the numerical errors caused by the spectral transform.

scholarly journals Solitary potential structures observed in the magnetosheath by the Cluster spacecraft

2003 ◽  
Vol 10 (1/2) ◽  
pp. 3-11 ◽  
Author(s):  
J. S. Pickett ◽  
J. D. Menietti ◽  
D. A. Gurnett ◽  
B. Tsurutani ◽  
P. M. Kintner ◽  
...  
Keyword(s):  
Magnetic Field ◽  
Time Series Data ◽  
Broad Band ◽  
Low Frequency ◽  
Bow Shock ◽  
Local Magnetic Field ◽  
Series Data ◽  
The Individual ◽  
Almost All ◽  
The Magnetic Field

Abstract. Bipolar pulses of ~ 25-100 µs in duration have been observed in the wave electric field data obtained by the Wideband plasma wave instrument on the Cluster spacecraft in the dayside magnetosheath. These pulses are similar in almost all respects to those observed on several spacecraft over the last few years. They represent solitary potential structures, and in this case, electron phase space holes. When the time series data containing the bipolar pulses on Cluster are transformed to the frequency domain by a windowed FFT, the pulses appear as typical broad-band features, extending from the low-frequency cutoff of the bandpass filter, ~ 1 kHz, up to as great as 20-40 kHz in some cases, with decreasing intensity as the frequency increases. The upper frequency cutoff of the broad band is an indication of the individual pulse durations (1/f). The solitary potential structures are detected when the local magnetic field is contained primarily in the spin plane, indicating that they propagate along the magnetic field. Their frequency extent and intensity seem to increase as the angle between the directions of the magnetic field and the plasma flow decreases from 90°. Of major significance is the finding that the overall profile of the broad-band features observed simultaneously by two Cluster spacecraft, separated by a distance of over 750 km, are strikingly similar in terms of onset times, frequency extent, intensity, and termination. This implies that the generation region of the solitary potential structures observed in the magnetosheath near the bow shock is very large and may be located at or near the bow shock, or be connected with the bow shock in some way.

Estimation of Most Probable Maximum From Short-Duration or Undersampled Time-Series Data

2015 ◽  
Author(s):  
Puneet Agarwal ◽  
William Walker ◽  
Kenneth Bhalla
Keyword(s):  
Time Series ◽  
Gaussian Process ◽  
Short Duration ◽  
Time Series Data ◽  
Extreme Value ◽  
Series Data ◽  
Sampled Data ◽  
Zero Crossing ◽  
Short Time ◽  
Undersampled Data

The most probable maximum (MPM) is the extreme value statistic commonly used in the offshore industry. The extreme value of vessel motions, structural response, and environment are often expressed using the MPM. For a Gaussian process, the MPM is a function of the root-mean square and the zero-crossing rate of the process. Accurate estimates of the MPM may be obtained in frequency domain from spectral moments of the known power spectral density. If the MPM is to be estimated from the time-series of a random process, either from measurements or from simulations, the time series data should be of long enough duration, sampled at an adequate rate, and have an ensemble of multiple realizations. This is not the case when measured data is recorded for an insufficient duration, or one wants to make decisions (requiring an estimate of the MPM) in real-time based on observing the data only for a short duration. Sometimes, the instrumentation system may not be properly designed to measure the dynamic vessel motions with a fine sampling rate, or it may be a legacy instrumentation system. The question then becomes whether the short-duration and/or the undersampled data is useful at all, or if some useful information (i.e., an estimate of MPM) can be extracted, and if yes, what is the accuracy and uncertainty of such estimates. In this paper, a procedure for estimation of the MPM from the short-time maxima, i.e., the maximum value from a time series of short duration (say, 10 or 30 minutes), is presented. For this purpose pitch data is simulated from the vessel RAOs (response amplitude operators). Factors to convert the short-time maxima to the MPM are computed for various non-exceedance levels. It is shown that the factors estimated from simulation can also be obtained from the theory of extremes of a Gaussian process. Afterwards, estimation of the MPM from the short-time maxima is explored for an undersampled process; however, undersampled data must not be used and only the adequately sampled data should be utilized. It is found that the undersampled data can be somewhat useful and factors to convert the short-time maxima to the MPM can be derived for an associated non-exceedance level. However, compared to the adequately sampled data, the factors for the undersampled data are less useful since they depend on more variables and have more uncertainty. While the vessel pitch data was the focus of this paper, the results and conclusions are valid for any adequately sampled narrow-banded Gaussian process.

scholarly journals An Approach for Reconstruction of Realistic Economic Data Based on Frequency Characteristics between IMFs

2020 ◽  
Vol 2020 ◽  
pp. 1-10 ◽  
Author(s):  
Hao Du ◽  
Hao Gong ◽  
Suyue Han ◽  
Peng Zheng ◽  
Bin Liu ◽  
...  
Keyword(s):  
Time Series ◽  
Data Analysis ◽  
Time Series Data ◽  
Low Frequency ◽  
Test Method ◽  
Series Data ◽  
Permutation Entropy ◽  
Intrinsic Mode Functions ◽  
Economic Data ◽  
Index Method

Reconstruction of realistic economic data often causes social economists to analyze the underlying driving factors in time-series data or to study volatility. The intrinsic complexity of time-series data interests and attracts social economists. This paper proposes the bilateral permutation entropy (BPE) index method to solve the problem based on partly ensemble empirical mode decomposition (PEEMD), which was proposed as a novel data analysis method for nonlinear and nonstationary time series compared with the T-test method. First, PEEMD is extended to the case of gold price analysis in this paper for decomposition into several independent intrinsic mode functions (IMFs), from high to low frequency. Second, IMFs comprise three parts, including a high-frequency part, low-frequency part, and the whole trend based on a fine-to-coarse reconstruction by the BPE index method and the T-test method. Then, this paper conducts a correlation analysis on the basis of the reconstructed data and the related affected macroeconomic factors, including global gold production, world crude oil prices, and world inflation. Finally, the BPE index method is evidently a vitally significant technique for time-series data analysis in terms of reconstructed IMFs to obtain realistic data.

Analysis of frequency components in time series data

1987 ◽  
Vol 22 (1) ◽  
pp. 79-87 ◽  
Author(s):  
Zehava Frostig ◽  
Ron D. Frostig
Keyword(s):  
Time Series ◽  
Time Series Data ◽  
Series Data ◽  
Frequency Components

scholarly journals Time-Series Causality with Missing Data

2021 ◽  
Vol 6 (1) ◽  
pp. 1-4
Author(s):  
Bo Yuan Chang ◽  
Mohamed A. Naiel ◽  
Steven Wardell ◽  
Stan Kleinikkink ◽  
John S. Zelek
Keyword(s):  
Time Series ◽  
Missing Data ◽  
Missing Values ◽  
Time Series Data ◽  
Multivariate Time Series ◽  
Gaussian Process Regression ◽  
Series Data ◽  
Causal Relationships ◽  
Sampled Data ◽  
The Past

Over the past years, researchers have proposed various methods to discover causal relationships among time-series data as well as algorithms to fill in missing entries in time-series data. Little to no work has been done in combining the two strategies for the purpose of learning causal relationships using unevenly sampled multivariate time-series data. In this paper, we examine how the causal parameters learnt from unevenly sampled data (with missing entries) deviates from the parameters learnt using the evenly sampled data (without missing entries). However, to obtain the causal relationship from a given time-series requires evenly sampled data, which suggests filling the missing data values before obtaining the causal parameters. Therefore, the proposed method is based on applying a Gaussian Process Regression (GPR) model for missing data recovery, followed by several pairwise Granger causality equations in Vector Autoregssive form to fit the recovered data and obtain the causal parameters. Experimental results show that the causal parameters generated by using GPR data filling offers much lower RMSE than the dummy model (fill with last seen entry) under all missing values percentage, suggesting that GPR data filling can better preserve the causal relationships when compared with dummy data filling, thus should be considered when dealing with unevenly sampled time-series causality learning.

scholarly journals Roadmap on signal processing for next generation measurement systems

2021 ◽  
Vol 33 (1) ◽  
pp. 012002
Author(s):  
Dimitris K Iakovidis ◽  
Melanie Ooi ◽  
Ye Chow Kuang ◽  
Serge Demidenko ◽  
Alexandr Shestakov ◽  
...  
Keyword(s):  
Signal Processing ◽  
Time Series Data ◽  
Research Field ◽  
Series Data ◽  
Next Generation ◽  
Research Attention ◽  
Measurement Systems ◽  
Scientific Disciplines ◽  
Wide Range ◽  
Future Challenges

Abstract Signal processing is a fundamental component of almost any sensor-enabled system, with a wide range of applications across different scientific disciplines. Time series data, images, and video sequences comprise representative forms of signals that can be enhanced and analysed for information extraction and quantification. The recent advances in artificial intelligence and machine learning are shifting the research attention towards intelligent, data-driven, signal processing. This roadmap presents a critical overview of the state-of-the-art methods and applications aiming to highlight future challenges and research opportunities towards next generation measurement systems. It covers a broad spectrum of topics ranging from basic to industrial research, organized in concise thematic sections that reflect the trends and the impacts of current and future developments per research field. Furthermore, it offers guidance to researchers and funding agencies in identifying new prospects.

Sign in / Sign up

Export Citation Format

Share Document