Multidimensional CAT Item Selection Methods for Domain Scores and Composite Scores With Item Exposure Control and Content Constraints

2014 ◽  
Vol 51 (1) ◽  
pp. 18-38 ◽  
Author(s):  
Lihua Yao
Keyword(s):  
Item Selection ◽  
Exposure Control ◽  
Selection Methods ◽  
Item Exposure ◽  
Item Exposure Control ◽  
Composite Scores ◽  
Domain Scores ◽  
Multidimensional Cat

Computerized Classification Testing Under the One-Parameter Logistic Response Model With Ability-Based Guessing

2011 ◽  
Vol 71 (6) ◽  
pp. 925-941 ◽  
Author(s):  
Wen-Chung Wang ◽  
Sheng-Yun Huang
Keyword(s):  
Fisher Information ◽  
Item Selection ◽  
Ratio Test ◽  
Exposure Rate ◽  
Exposure Control ◽  
Progressive Method ◽  
Selection Methods ◽  
Item Exposure ◽  
Item Exposure Control ◽  
The One

The one-parameter logistic model with ability-based guessing (1PL-AG) has been recently developed to account for effect of ability on guessing behavior in multiple-choice items. In this study, the authors developed algorithms for computerized classification testing under the 1PL-AG and conducted a series of simulations to evaluate their performances. Four item selection methods (the Fisher information, the Fisher information with a posterior distribution, the progressive method, and the adjusted progressive method) and two termination criteria (the ability confidence interval [ACI] method and the sequential probability ratio test [SPRT]) were developed. In addition, the Sympson–Hetter online method with freeze (SHOF) was implemented for item exposure control. Major results include the following: (a) when no item exposure control was made, all the four item selection methods yielded very similar correct classification rates, but the Fisher information method had the worst item bank usage and the highest item exposure rate; (b) SHOF can successfully maintain the item exposure rate at a prespecified level, without compromising substantial accuracy and efficiency in classification; (c) once SHOF was implemented, all the four methods performed almost identically; (d) ACI appeared to be slightly more efficient than SPRT; and (e) in general, a higher weight of ability in guessing led to a slightly higher accuracy and efficiency, and a lower forced classification rate.

scholarly journals A Dynamic Stratification Method for Improving Trait Estimation in Computerized Adaptive Testing Under Item Exposure Control

2019 ◽  
Vol 44 (3) ◽  
pp. 182-196
Author(s):  
Jyun-Hong Chen ◽  
Hsiu-Yi Chao ◽  
Shu-Ying Chen
Keyword(s):  
Computerized Adaptive Testing ◽  
Item Difficulty ◽  
Adaptive Testing ◽  
Item Selection ◽  
Exposure Control ◽  
Item Exposure ◽  
Stratification Method ◽  
High Discrimination ◽  
Item Exposure Control ◽  
Trait Estimation

When computerized adaptive testing (CAT) is under stringent item exposure control, the precision of trait estimation will substantially decrease. A new item selection method, the dynamic Stratification method based on Dominance Curves (SDC), which is aimed at improving trait estimation, is proposed to mitigate this problem. The objective function of the SDC in item selection is to maximize the sum of test information for all examinees rather than maximizing item information for individual examinees at a single-item administration, as in conventional CAT. To achieve this objective, the SDC uses dominance curves to stratify an item pool into strata with the number being equal to the test length to precisely and accurately increase the quality of the administered items as the test progresses, reducing the likelihood that a high-discrimination item will be administered to an examinee whose ability is not close to the item difficulty. Furthermore, the SDC incorporates a dynamic process for on-the-fly item–stratum adjustment to optimize the use of quality items. Simulation studies were conducted to investigate the performance of the SDC in CAT under item exposure control at different levels of severity. According to the results, the SDC can efficiently improve trait estimation in CAT through greater precision and more accurate trait estimation than those generated by other methods (e.g., the maximum Fisher information method) in most conditions.

scholarly journals Components of the item selection algorithm in computerized adaptive testing

2018 ◽  
Vol 15 ◽  
pp. 7 ◽  
Author(s):  
Kyung (Chris) Tyek Han
Keyword(s):  
Computerized Adaptive Testing ◽  
Selection Criterion ◽  
Information Criterion ◽  
Item Selection ◽  
Exposure Control ◽  
Selection Algorithm ◽  
High Stakes ◽  
Item Exposure ◽  
Content Balancing ◽  
Item Exposure Control

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the astratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the KullbackLeibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

scholarly journals New Item Selection Method Accommodating Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing: Maximum Deviation and Maximum Limitation Global Discrimination Indexes

2021 ◽  
Vol 12 ◽  
Author(s):  
Junjie Li ◽  
Lihua Ma ◽  
Pingfei Zeng ◽  
Chunhua Kang
Keyword(s):  
Classification Accuracy ◽  
Computerized Adaptive Testing ◽  
Selection Method ◽  
Discrimination Index ◽  
Item Selection ◽  
Maximum Deviation ◽  
Exposure Control ◽  
Item Exposure ◽  
Item Exposure Control ◽  
Attribute Classification

Maximum deviation global discrimination index (MDGDI) is a new item selection method for cognitive diagnostic computerized adaptive testing that allows for attribute coverage balance. We developed the maximum limitation global discrimination index (MLGDI) from MDGDI, which allows for both attribute coverage balance and item exposure control. MLGDI can realize the attribute coverage balance and exposure control of the item. Our simulation study aimed to evaluate the performance of our new method against maximum global discrimination index (GDI), modified maximum GDI (MMGDI), standardized weighted deviation GDI (SWDGDI), and constraint progressive with SWDGDI (CP_SWDGDI). The results indicated that (1a) under the condition of realizing the attribute coverage balance, MDGDI had the highest attribute classification accuracy; (1b) when the selection strategy accommodated the practical constraints of the attribute coverage balance and item exposure control, MLGDI had the highest attribute classification accuracy; (2) adding the item exposure control mechanism to the item selection method reduces the classification accuracy of the attributes of the item selection method; and (3) compared with GDI, MMGDI, SWDGDI, CP_SWDGDI, and MDGDI, MLGDI can better achieve the attribute-coverage requirement, control item exposure rate, and attribute correct classification rate.

Incorporating randomness in the Fisher information for improving item-exposure control in CATs

2008 ◽  
Vol 61 (2) ◽  
pp. 493-513 ◽  
Author(s):  
Juan Ramón Barrada ◽  
Julio Olea ◽  
Vicente Ponsoda ◽  
Francisco José Abad
Keyword(s):  
Fisher Information ◽  
Exposure Control ◽  
Item Exposure ◽  
Item Exposure Control

scholarly journals Some Alternatives to Sympson-Hetter Item-Exposure Control in Computerized Adaptive Testing

2003 ◽  
Vol 28 (3) ◽  
pp. 249-265 ◽  
Author(s):  
Wim J. van der Linden
Keyword(s):  
Iterative Process ◽  
Computerized Adaptive Testing ◽  
Adaptive Testing ◽  
Law School ◽  
Exposure Control ◽  
Item Exposure ◽  
Item Exposure Control ◽  
Admission Test ◽  
Formal Properties ◽  
Ability Parameter

The Hetter and Sympson (1997 ; 1985 ) method is a method of probabilistic item-exposure control in computerized adaptive testing. Setting its control parameters to admissible values requires an iterative process of computer simulations that has been found to be time consuming, particularly if the parameters have to be set conditional on a realistic set of values for the examinees’ ability parameter. Formal properties of the method are identified that help us explain why this iterative process can be slow and does not guarantee admissibility. In addition, some alternatives to the SH method are introduced. The behavior of these alternatives was estimated for an adaptive test from an item pool from the Law School Admission Test (LSAT). Two of the alternatives showed attractive behavior and converged smoothly to admissibility for all items in a relatively small number of iteration steps.

Sign in / Sign up

Export Citation Format

Share Document