Motivations of assessment item writers in medical programs: a qualitative study

Background The challenge of generating sufficient quality items for medical student examinations is a common experience for medical program coordinators. Faculty development strategies are commonly used, but there is little research on the factors influencing medical educators to engage in item writing. To assist with designing evidence-based strategies to improve engagement, we conducted an interview study informed by self-determination theory (SDT) to understand educators’ motivations to write items. Methods We conducted 11 semi-structured interviews with educators in an established medical program. Interviews were transcribed verbatim and underwent open coding and thematic analysis. Results Major themes included; responsibility for item writing and item writer motivations, barriers and enablers; perceptions of the level of content expertise required to write items; and differences in the writing process between clinicians and non-clinicians. Conclusions Our findings suggest that flexible item writing training, strengthening of peer review processes and institutional improvements such as improved communication of expectations, allocation of time for item writing and pairing new writers with experienced writers for mentorship could enhance writer engagement.

ITEM ANALYSIS AND PEER-REVIEW EVALUATION OF SPECIFIC HEALTH PROBLEMS AND APPLIED RESEARCH BLOCK EXAMINATION

Background: Assessment and evaluation for students is an essential component of teaching and learning process. Item analysis is the technique of collecting, summarizing, and using students’ response data to assess the quality of the Multiple Choice Question (MCQ) test by measuring indices of difficulty and discrimination, also distracter efficiency. Peer review practices improve quality of assessment validity in evaluating student performance.Method: We analyzed 150 student’s responses for 100 MCQs in Block Examination for its difficulty index (p), discrimination index (D) and distractor efficiency (DE) using Microsoft excel formula. The Correlation of p and D was analyzed using Spearman correlation test by SPSS 23.0. The result was analyzed to evaluate the peer-review strategy.Results: The median of difficulty index (p) was 54% or within the range of excellent level (p 40-60%) and the mean of discrimination index (D) was 0.24 which is reasonably good. There were 7 items with excellent p (40–60%) and excellent D (≥0.4). Nineteen of items had excellent discrimination index (D≥0.4). However,there were 9 items with negative discrimination index and 30 items with poor discrimination index, which should be fully revised. Forty-two of items had 4 functioning distracters (DE 0%) which suggested the teacher to be more precise and carefully creating the distracters.Conclusion: Based on item analysis, there were items to be fully revised. For better test quality, feedback and suggestions for the item writer should also be performed as a part of peer-review process on the basis of item analysis.

Motivating assessment item writers in medical programs: A qualitative study

Background The challenge of generating sufficient quality items for medical student examinations is a common experience for medical program coordinators. Faculty development strategies are commonly used, but there is little research on the factors influencing medical educators to engage in item writing. To assist with designing evidence-based strategies to improve engagement, we conducted an interview study informed by self-determination theory (SDT) to understand educators’ motivations to write items.Methods We conducted 11 semi-structured interviews with educators in an established medical program. Interviews were transcribed verbatim and underwent open coding and thematic analysis.Results Major themes included; responsibility for item writing and item writer motivations, barriers and enablers; perceptions of the level of content expertise required to write items; and differences in the writing process between clinicians and non-clinicians.Conclusions Our findings suggest that flexible item writing training, strengthening of peer review processes and institutional improvements such as improved communication of expectations, allocation of time for item writing and pairing new writers with experienced writers for mentorship could enhance writer engagement.