The Effect of Item Exposure Control Methods on Measurement Precision and Test Security under Different Measurement Conditions in Computerized Adaptive Testing

The Hetter and Sympson (1997 ; 1985 ) method is a method of probabilistic item-exposure control in computerized adaptive testing. Setting its control parameters to admissible values requires an iterative process of computer simulations that has been found to be time consuming, particularly if the parameters have to be set conditional on a realistic set of values for the examinees’ ability parameter. Formal properties of the method are identified that help us explain why this iterative process can be slow and does not guarantee admissibility. In addition, some alternatives to the SH method are introduced. The behavior of these alternatives was estimated for an adaptive test from an item pool from the Law School Admission Test (LSAT). Two of the alternatives showed attractive behavior and converged smoothly to admissibility for all items in a relatively small number of iteration steps.

Download Full-text

A Dynamic Stratification Method for Improving Trait Estimation in Computerized Adaptive Testing Under Item Exposure Control

Applied Psychological Measurement ◽

10.1177/0146621619843820 ◽

2019 ◽

Vol 44 (3) ◽

pp. 182-196

Author(s):

Jyun-Hong Chen ◽

Hsiu-Yi Chao ◽

Shu-Ying Chen

Keyword(s):

Computerized Adaptive Testing ◽

Item Difficulty ◽

Adaptive Testing ◽

Item Selection ◽

Exposure Control ◽

Item Exposure ◽

Stratification Method ◽

High Discrimination ◽

Item Exposure Control ◽

Trait Estimation

When computerized adaptive testing (CAT) is under stringent item exposure control, the precision of trait estimation will substantially decrease. A new item selection method, the dynamic Stratification method based on Dominance Curves (SDC), which is aimed at improving trait estimation, is proposed to mitigate this problem. The objective function of the SDC in item selection is to maximize the sum of test information for all examinees rather than maximizing item information for individual examinees at a single-item administration, as in conventional CAT. To achieve this objective, the SDC uses dominance curves to stratify an item pool into strata with the number being equal to the test length to precisely and accurately increase the quality of the administered items as the test progresses, reducing the likelihood that a high-discrimination item will be administered to an examinee whose ability is not close to the item difficulty. Furthermore, the SDC incorporates a dynamic process for on-the-fly item–stratum adjustment to optimize the use of quality items. Simulation studies were conducted to investigate the performance of the SDC in CAT under item exposure control at different levels of severity. According to the results, the SDC can efficiently improve trait estimation in CAT through greater precision and more accurate trait estimation than those generated by other methods (e.g., the maximum Fisher information method) in most conditions.

Download Full-text

Constraining Item Exposure in Computerized Adaptive Testing With Shadow Tests

Journal of Educational and Behavioral Statistics ◽

10.3102/10769986029003273 ◽

2004 ◽

Vol 29 (3) ◽

pp. 273-291 ◽

Cited By ~ 57

Author(s):

Wim J van der Linden ◽

Bernard P Veldkamp

Keyword(s):

Computerized Adaptive Testing ◽

Mean Squared Error ◽

Adaptive Testing ◽

Law School ◽

Exposure Control ◽

Require Time ◽

Item Exposure ◽

Squared Error ◽

Item Exposure Control ◽

Admission Test

Item-exposure control in computerized adaptive testing is implemented by imposing item-ineligibility constraints on the assembly process of the shadow tests. The method resembles Sympson and Hetter’s (1985) method of item-exposure control in that the decisions to impose the constraints are probabilistic. The method does not, however, require time-consuming simulation studies to set values for control parameters before the operational use of the test. Instead, it can set the probabilities of item ineligibility adaptively during the test using the actual item-exposure rates. An empirical study using an item pool from the Law School Admission Test showed that application of the method yielded perfect control of the item-exposure rates and had negligible impact on the bias and mean-squared error functions of the ability estimator.

Download Full-text

Binary Restrictive Threshold Method for Item Exposure Control in Cognitive Diagnostic Computerized Adaptive Testing

Frontiers in Psychology ◽

10.3389/fpsyg.2021.517155 ◽

2021 ◽

Vol 12 ◽

Author(s):

Xiaojian Sun ◽

Yizhu Gao ◽

Tao Xin ◽

Naiqing Song

Keyword(s):

Classification Accuracy ◽

Measurement Accuracy ◽

Computerized Adaptive Testing ◽

Critical Issue ◽

Adaptive Testing ◽

Exposure Control ◽

Threshold Method ◽

Item Exposure ◽

Item Exposure Control ◽

Better Than

Although classification accuracy is a critical issue in cognitive diagnostic computerized adaptive testing, attention has increasingly shifted to item exposure control to ensure test security. In this study, we developed the binary restrictive threshold (BRT) method to balance measurement accuracy and item exposure. In addition, a simulation study was conducted to evaluate its performance. The results indicated that the BRT method performed better than the restrictive progressive (RP) and stratified dynamic binary searching (SDBS) approaches but worse than the restrictive threshold (RT) method in terms of classification accuracy. With respect to item exposure control, the BRT method exhibited noticeably stronger performance compared with the RT method, even though its performance was not as high as that of the RP and SDBS methods.

Download Full-text