The Effect of Person Misfit on Item Parameter Estimation and Classification Accuracy: A Simulation Study

Often, important decisions regarding accountability and placement of students in performance categories are made on the basis of test scores generated from tests, therefore, it is important to evaluate the validity of the inferences derived from test results. One of the threats to the validity of such inferences is aberrant responding. Several person fit indices were developed to detect aberrant responding on educational and psychological tests. The majority of the person fit literature has been focused on creating and evaluating new indices. The aim of this study was to assess the effect of aberrant responding on the accuracy of estimated item parameters and refining estimations by using person fit statistics by means of simulation. Our results showed that the presence of aberrant response patterns created bias in the both b and a parameters at the item level and affected the classification of students, particularly high-performing students, into performance categories regardless of whether aberrant response patterns were present in the data or were removed. The results differed by test length and the percentage of students with aberrant response patterns. Practical and theoretical implications are discussed.

Identifying and Classifying Aberrant Response Patterns Through Functional Data Analysis

We propose new methods for identifying and classifying aberrant response patterns (ARPs) by means of functional data analysis. These methods take the person response function (PRF) of an individual and compare it with the pattern that would correspond to a generic individual of the same ability according to the item-person response surface. ARPs correspond to atypical difference functions. The ARP classification is done with functional data clustering applied to the PRFs identified as ARP. We apply these methods to two sets of simulated data (the first is used to illustrate the ARP identification methods and the second demonstrates classification of the response patterns flagged as ARP) and a real data set (a Grade 12 science assessment test, SAT, with 32 items answered by 600 examinees). For comparative purposes, ARPs are also identified with three nonparametric person-fit indices (Ht, Modified Caution Index, and ZU3). Our results indicate that the ARP detection ability of one of our proposed methods is comparable to that of person-fit indices. Moreover, the proposed classification methods enable ARP associated with either spuriously low or spuriously high scores to be distinguished.

Aberrant Response Patterns on the Jordanian Version of Otis–Lennon Test and their Impact on the Accuracy of Estimating a Person's Ability and Information Function

This study aimed at investigating the aberrant response patterns and their impacts on the Jordanian version Otis- Lennon as well as the accuracy in the estimation of a person's' ability and information function test. To achieve this goal, the Jordanian version of the Ability Test primary II level form K was administrated to 568 first-grade male and female students of Ajloun district public schools during 2016/2017. The Lz person fit index and the three-parameter logistic model were used to analyze students' responses to test items to assess the person ability, information function test, and to detect aberrant response patterns. The results revealed that the response patterns of 56 students were aberrant based on Lz index; also the results showed that the factors responsible for the presence of this aberrant response were: guessing, cheating, laziness, and exponential creatively. The results indicated that when the aberrant patterns were excluded, both the accuracy person's estimating ability and the information function test had significantly increased at different ability levels.

Detection of Aberrant Response Patterns and their Effect on Dimensionality

Two indices were developed for measuring the degree of conformity or consistency of an individual examinee’s response pattern on a set of items. The first, called the norm conformity index (NCI), measures the proximity of the pattern to a baseline pattern in which all 0’s precede all l’s when the items are arranged in some prescribed order. The second, called the individual consistency index (ICI), measures the extent to which an individual’s response pattern remains invariant when he or she responds to several waves of parallel items. We show how these two indices, used in conjunction with test scores, can spot aberrant response patterns of students requiring detailed error diagnosis and remedial treatment.