Thirty years of focus on individual variability and the dynamics of processes

We fully endorse Arocha’s (2021) thesis about the fundamental importance of studying variability in real, observable processes and agree with his critique of the standard practice of psychological research. However, we regret that Arocha’s article does not acknowledge a rich body of research that has been around for almost three decades and that does exactly what Arocha recommends. This research is based on the theory of complex dynamic systems. We discuss its main implications for a research focus on concrete psychological processes, as they occur in individual cases (including real interacting groups). Variability over time is used as a main source of information about the nature of the underlying processes. Various examples of empirical studies, model building, and process-oriented methodology are discussed, and Arocha’s examples of perceptual control theory (PCT) and observation-oriented modeling (OOM) are put in the perspective of the complex dynamic systems approach, which is fully compatible with scientific realism as advocated by Arocha.

Scientific realism and the issue of variability in behavior

The purpose of this article is to present a critical analysis of current research practices in the study of behavior from the point of view of scientific realism. Although the so-called “replication crisis” observed in the psychological and health sciences has led to various proposals for improving research quality, most of those proposals take the standard linear input–output approach for granted, where behavioral variability is seen as the result of uncontrolled random variables hiding the true input–output relations. Aggregate data and the computation of sample statistics are used to estimate population parameters, the true reality behind appearances. In this paper, I offer a different interpretation: variability is a fact of behavior necessary for successful performance, not the result of some unknown variables randomly affecting individual outputs. Research models that take individual behavior with all of its complexity as the real thing, can help us overcome the limitations of the standard approach to research. As an illustration, I also describe two approaches to behavioral investigations that do not rely on standard statistical analysis for producing genuine knowledge: perceptual control theory and observation-oriented modeling.

Ordinal Pattern Analysis in Comparative Psychology: A Flexible Alternative to Null Hypothesis Significance Testing Using an Observation Oriented Modeling Paradigm

The data of comparative psychology generally differ from the majority of data collected within mainstream psychology in several key respects – most notably in the diversity of forms of measurement and fewer number of subjects. We believe null hypothesis significance testing may not be the most appropriate method of analysis for comparative psychology for these reasons. Comparative psychology has a rich history of performing several analyses on a few subjects due to a philosophical interest in individual subject behavior, along with group assessments. Since first being published in 2011, Observation Oriented Modeling has successfully been used to analyze individual subjects’ responses from honey bees, horses, humans, and rattlesnakes. Observation Oriented Modeling is highly flexible and has allowed comparative researchers to perform a variety of assessments comparable to null hypothesis significance testing’s T-Tests, One-way ANOVA, and Repeated-Measures ANOVA while producing easily-interpretable and, most importantly, relevant results. This paper describes the diverse manners in which comparative psychologists can assess individual and group performances without concerns of statistical assumptions and limitations that complicate assessments when employing Null Hypothesis Significance Testing.

Beyond p-values: Utilizing Multiple Estimates to Evaluate Evidence

Null hypothesis significance testing is frequently cited as a threat to the validity and reproducibility of the social sciences. While many individuals suggest we should focus on altering the *p*-value at which we deem an effect significant, we believe this suggestion is short-sighted. Alternative procedures (i.e., Bayesian analyses and Observation Oriented Modeling) can be more powerful and meaningful to our discipline. However, these methodologies are less frequently utilized and are rarely discussed in combination with NHST. Herein, we compare the possible interpretations of three analyses (ANOVA, Bayes Factor, and an Ordinal Pattern Analysis) in various data environments using a simulation study. The simulation generated 20000 unique datasets which varied sample size (*N*s of 10, 30, 100, 500, 1000), and effect sizes (*d*s of 0.10, 0.20, 0.05, 0.80). Through this simulation, we find that changing the threshold at which *p*-values are considered significant has little to no effect on conclusions. Further, we find that evaluating multiple estimates as evidence of an effect can allow for a more robust and nuanced report of findings. These findings suggest the need to redefine evidentiary value and reporting practices.

Observation Oriented Modelling revised from a Statistical Point of View

Observation Oriented Modeling was proposed to overcome some of the problems in the application of statistical inference methods in the behavioral sciences. In this paper, we refine one part of this approach and show how it is connected to methods that are well known in statistical learning. Precisely, we argue that the Moore-Penrose pseudo inverse is superior to the initial solution from a statistical point of view. With this we also show that Observation Oriented Modeling can indeed be appropriate for some tasks in the analysis of observed data. To provide a practical example, we demonstrate the revised method by analyzing the effect of mindfulness training on attentional processes.

Observation-Oriented Modeling: Going Beyond “Is It All a Matter of Chance”?

An alternative to null hypothesis significance testing is presented and discussed. This approach, referred to as observation-oriented modeling, is centered on model building in an effort to explicate the structures and processes believed to generate a set of observations. In terms of analysis, this novel approach complements traditional methods based on means, variances, and covariances with methods of pattern detection and analysis. Using data from a previously published study by Shoda et al., the basic tenets and methods of observation-oriented modeling are demonstrated and compared with traditional methods, particularly with regard to null hypothesis significance testing.