Selecting Mathematical Tasks for Assessing Student’s Understanding: Pre-Service Teachers’ Sensitivity to and Adaptive Use of Diagnostic Task Potential in Simulated Diagnostic One-To-One Interviews

Teachers’ diagnostic competences are regarded as highly important for classroom assessment and teacher decision making. Prior conceptualizations of diagnostic competences as judgement accuracy have been extended to include a wider understanding of what constitutes a diagnosis; novel models of teachers’ diagnostic competences explicitly include the diagnostic process as the core of diagnosing. In this context, domain-general and mathematics-specific research emphasizes the importance of tasks used to elicit student cognition. However, the role of (mathematical) tasks in diagnostic processes has not yet attracted much systematic empirical research interest. In particular, it is currently unclear whether teachers consider diagnostic task potential when selecting tasks for diagnostic interviews and how this relationship is shaped by their professional knowledge. This study focuses on pre-service mathematics teachers’ selection of tasks during one-to-one diagnostic interviews in live simulations. Each participant worked on two 30 mins interviews in the role of a teacher, diagnosing a student’s mathematical understanding of decimal fractions. The participants’ professional knowledge was measured afterward. Trained assistants played simulated students, who portrayed one of four student case profiles, each having different mathematical (mis-)conceptions of decimal fractions. For the interview, participants could select tasks from a set of 45 tasks with different diagnostic task potentials. Two aspects of task selection during the diagnostic processes were analyzed: participants’ sensitivity to the diagnostic potential, which was reflected in higher odds for selecting tasks with high potential than tasks with low potential, and the adaptive use of diagnostic task potential, which was reflected in task selection influenced by a task’s diagnostic potential in combination with previously collected information about the student’s understanding. The results show that participants vary in their sensitivity to diagnostic task potential, but not in their adaptive use. Moreover, participants’ content knowledge had a significant effect on their sensitivity. However, the effects of pedagogical content and pedagogical knowledge did not reach significance. The results highlight that pre-service teachers require further support to effectively attend to diagnostic task potential. Simulations were used for assessment purposes in this study, and they appear promising for this purpose because they allow for the creation of authentic yet controlled situations.

Investigating Pre-Service Biology Teachers’ Diagnostic Competences: Relationships between Professional Knowledge, Diagnostic Activities, and Diagnostic Accuracy

Teachers’ diagnostic competences are essential with respect to student achievement, classroom assessment, and instructional quality. Important components of diagnostic competences are teachers’ professional knowledge including content knowledge (CK), pedagogical knowledge (PK), and pedagogical content knowledge (PCK), their diagnostic activities as a specification of situation-specific skills, and diagnostic accuracy. Accuracy is determined by comparing a teacher’s observation of classroom incidents with subject-specific challenges to be identified from scripted instructional situations. To approximate diagnostic situations close to real-life, the assessment of science teachers’ diagnostic competences requires a situated context that was provided through videotaped classroom situations in this study. We investigated the relationship between professional knowledge (PCK, CK, PK) of 186 pre-service biology teachers, their diagnostic activities, and diagnostic accuracy measured with the video-based assessment tool DiKoBi Assess. Results of path analyses utilizing Rasch measures showed that both PCK and PK were statistically significantly related to pre-service teachers’ diagnostic activities. Additionally, biology teachers’ PCK was positively related to diagnostic accuracy. Considering higher effect sizes of PCK compared to PK, the findings support previous findings indicating the importance of PCK, thus demonstrating its importance in the context of subject-specific diagnosis as well.

Assessment of Diagnostic Competences With Standardized Patients Versus Virtual Patients: Experimental Study in the Context of History Taking (Preprint)

BACKGROUND Standardized patients (SPs) have been one of the popular assessment methods in clinical teaching for decades, although they are resource intensive. Nowadays, simulated virtual patients (VPs) are increasingly used because they are permanently available and fully scalable to a large audience. However, empirical studies comparing the differential effects of these assessment methods are lacking. Similarly, the relationships between key variables associated with diagnostic competences (ie, diagnostic accuracy and evidence generation) in these assessment methods still require further research. OBJECTIVE The aim of this study is to compare perceived authenticity, cognitive load, and diagnostic competences in performance-based assessment using SPs and VPs. This study also aims to examine the relationships of perceived authenticity, cognitive load, and quality of evidence generation with diagnostic accuracy. METHODS We conducted an experimental study with 86 medical students (mean 26.03 years, SD 4.71) focusing on history taking in dyspnea cases. Participants solved three cases with SPs and three cases with VPs in this repeated measures study. After each case, students provided a diagnosis and rated perceived authenticity and cognitive load. The provided diagnosis was scored in terms of diagnostic accuracy; the questions asked by the medical students were rated with respect to their quality of evidence generation. In addition to regular null hypothesis testing, this study used equivalence testing to investigate the absence of meaningful effects. RESULTS Perceived authenticity (1-tailed <i>t</i>₈₁=11.12; <i>P</i>&lt;.001) was higher for SPs than for VPs. The correlation between diagnostic accuracy and perceived authenticity was very small (<i>r</i>=0.05) and neither equivalent (<i>P</i>=.09) nor statistically significant (<i>P</i>=.32). Cognitive load was equivalent in both assessment methods (<i>t</i>₈₂=2.81; <i>P</i>=.003). Intrinsic cognitive load (1-tailed <i>r</i>=−0.30; <i>P</i>=.003) and extraneous load (1-tailed <i>r</i>=−0.29; <i>P</i>=.003) correlated negatively with the combined score for diagnostic accuracy. The quality of evidence generation was positively related to diagnostic accuracy for VPs (1-tailed <i>r</i>=0.38; <i>P</i>&lt;.001); this finding did not hold for SPs (1-tailed <i>r</i>=0.05; <i>P</i>=.32). Comparing both assessment methods with each other, diagnostic accuracy was higher for SPs than for VPs (2-tailed <i>t</i>₈₅=2.49; <i>P</i>=.01). CONCLUSIONS The results on perceived authenticity demonstrate that learners experience SPs as more authentic than VPs. As higher amounts of intrinsic and extraneous cognitive loads are detrimental to performance, both types of cognitive load must be monitored and manipulated systematically in the assessment. Diagnostic accuracy was higher for SPs than for VPs, which could potentially negatively affect students’ grades with VPs. We identify and discuss possible reasons for this performance difference between both assessment methods.

The importance of teacher’s diagnostic competences for students’ development

The article focuses on the issue of teacher’s diagnostic competence. In order to emphasize the importance of the issue and to encourage scientifi c discussion, the value of the contemporary teacher’s pedagogical competences was pointed out, among which diagnostic competence was highlighted. Then the importance of diagnostic competence in working with the student and the consequences of its lack, which is the student’s marginalization, were determined. The conclusions indicate the implications for educational practice.

Delphi study to determine the key qualities consultant histopathologists look for in their trainees

AimsA Delphi study to triangulate and determine the relative importance of the key qualities of trainees identified from qualitative interviews that sought to understand how consultant histopathologists determine diagnostic competences in trainees.MethodsTwelve participants were purposively chosen for the Delphi to form an expert panel of relevant stakeholders. Participants were asked to score and rank the items presented to them.ResultsA total of 22 out of 27 of the key qualities of trainees (items) reached ‘consensus in’ after round 2 suggesting participants were able to agree that the majority of the items identified in the qualitative interviews were important to diagnostic competence. Five items reached ‘no consensus’. Participants did not suggest any additional items. Participants particularly valued qualities of reflection and professionalism and trainees who understood the process of reaching a diagnosis and how their pathological report could impact on patient care.ConclusionsThis study has triangulated findings from our qualitative interviews and show that consultants value a wide variety of qualities when determining diagnostic competence in their trainees. The judgement is complex and is therefore best assessed longitudinally and on a number of cases, so consultants can look for consistency of both approach to diagnosis and of trainee behaviour.