scholarly journals P122: Narrative assessment of emergency medicine learners: What should we keep as we move to competency-based assessment?

CJEM ◽  
2019 ◽  
Vol 21 (S1) ◽  
pp. S108
Author(s):  
S. Segeren ◽  
L. Shepherd ◽  
R. Pack
Keyword(s):  
Emergency Medicine ◽  
Contextual Information ◽  
Management Plan ◽  
Professional Activities ◽  
Narrative Assessment ◽  
Canmeds Role ◽  
Competency Based ◽  
Qualitative Methodologies ◽  
Iterative Coding ◽  
Narrative Comments

Introduction: For many years, Emergency Medicine (EM) educators have used narrative comments to assess their learners on each shift, either in isolation or combined with some type of Likert scale ranking. Competency based medical education (CBME), soon to be fully implemented throughout Canadian EM educational programs, encourages this type of frequent low-stakes narrative assessment. It is important to understand what information is currently garnered from existing narrative assessments in order to successfully and smoothly transition to the CBME system. The purpose of this study was to explore how one Canadian undergraduate EM program's narrative assessment comments mapped to two competency frameworks: one traditional CanMEDS-based and one competency-based, built on entrustable professional activities (EPAs). Methods: A qualitative and quantitative content analysis of 1925 retrospective, narrative assessments was conducted for the 2015/2016 and 2016/2017 academic years. The unprompted comments were mapped to the Royal College CanMEDS framework and the Association of Faculties of Medicine of Canada EPA Framework. Using an iterative coding process as per accepted qualitative methodologies, additional codes were generated to classify comments and identify themes that were not captured by either framework. Results: 93% and 85% of the unprompted narrative assessments contained comments that mapped to at least one CanMEDS role or EPA competency, respectively. The most common CanMEDS role commented upon was Medical Expert (86%), followed by Communicator, Collaborator and Scholar (all at 23%). The most common EPA competency mentioned related to history and physical findings (62%) followed by management plan (33%), and differential diagnosis (33%). However, 75% of narrative comments contained within the assessments, included ideas that did not fall into either framework but were repeated with frequency to suggest importance. The experiential characteristics of working with a learner were commented upon by 22% of preceptors. Other unmapped themes included contextual information, generalities and platitudes, and directed feedback for next steps to improve. Conclusion: While much of the currently captured data can be mapped to established frameworks, important information for both learner and assessor may be lost by limiting comments to the competencies described within a particular framework, suggesting caution when transitioning to a CBME assessment program.

Assessment of Entrustable Professional Activities Using an Online Simulation Platform During Transition to Emergency Medicine Residency: A Mixed Methods Pilot Study (Preprint)

2021 ◽  
Author(s):  
Cynthia R Peng ◽  
Kimberly A Schertzer ◽  
Holly A Caretta-Weyer ◽  
Stefanie S Sebok-Syer ◽  
William Lu ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Pilot Study ◽  
Final Diagnosis ◽  
Diagnostic Assessment ◽  
Entrustable Professional Activities ◽  
Simulation Platform ◽  
Professional Activities ◽  
Clinical Encounters ◽  
Critical Action ◽  
Competency Based

BACKGROUND The 13 Core Entrustable Professional Activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education. Five of these EPAs (EPA2: prioritizing differential, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for online assessment. OBJECTIVE For this pilot study, we created a web-based simulation platform for diagnostic assessment of these EPAs and examined its feasibility and acceptability. METHODS Four simulation cases underwent three rounds of consensus panels and pilot testing. Incoming emergency medicine interns (n=15) completed all cases, and up to 4 “look for” statements, which encompassed specific EPAs, were generated for each participant: 1) performing harmful or missing actions, 2) narrow differential or wrong final diagnosis, 3) having errors in documentation, and 4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis. RESULTS All participants had at least 1 missing critical action and 40% participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of assessments (53%). Other errors included choosing the incorrect documentation (40%) and indiscriminately applying oxygen (60%). The themes to the interviews included: psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface. CONCLUSIONS This study demonstrates the feasibility and acceptability of this platform for diagnostic assessment of specific EPAs. This approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides this feedback in a manner appreciated by residency leadership, and informs individualized learning plans.

scholarly journals P101: The development of entrustable professional activity reference cards to support the implementation of Competence by Design in emergency medicine

CJEM ◽  
2019 ◽  
Vol 21 (S1) ◽  
pp. S100
Author(s):  
E. Stoneham ◽  
L. Witt ◽  
Q. Paterson ◽  
L. Martin ◽  
B. Thoma
Keyword(s):  
Emergency Medicine ◽  
Low Cost ◽  
Intervention Mapping ◽  
Assessment System ◽  
Professional Activity ◽  
Residency Programs ◽  
Professional Activities ◽  
Competency Based ◽  
Curriculum Map ◽  
User Friendly

Innovation Concept: Competence by Design (CBD) was implemented nationally for Emergency Medicine (EM) residents beginning training in 2018. One challenge is the need to introduce residents to Entrustable Professional Activities (EPAs) that are assessed across numerous clinical rotations. The Royal College's resources detail these requirements, but do not map them to specific rotations or present them in a succinct format. This is problematic as trainees are less likely to succeed when expectations are unclear. We identified a need to create practical resources that residents can use at the bedside. Methods: We followed an intervention mapping framework to design two practical, user-friendly, low-cost, aesthetically pleasing resources that could be used by residents and observers at the bedside to facilitate competency-based assessment. Curriculum, Tool or Material: First, we designed a set of rotation- and stage-specific EPA reference cards for the use of residents and observers at the bedside. These cards list EPAs and clinical presentations likely to be encountered during various stages of training and on certain rotations. Second, we developed a curriculum board to organize the EPA reference cards by stage based upon our program's curriculum map. The curriculum board allows residents to view the program's curriculum map and the EPAs associated with each clinical rotation at a glance. It also contains hooks to hang and store extra cards in an organized manner. Conclusion: We believe that these practical and inexpensive tools facilitated our residency program's transition to competency-based EPA assessments. Anecdotally, the residents are using the cards and completing the suggested rotation-specific EPAs. We hope that the reference cards and curriculum board will be successfully incorporated into other residency programs to facilitate the introduction of their EPA-based CBD assessment system.

The competency-based medical education evolution of Canadian emergency medicine specialist training

CJEM ◽  
2019 ◽  
Vol 22 (1) ◽  
pp. 95-102 ◽  
Author(s):  
Jonathan Sherbino ◽  
Glen Bandiera ◽  
Ken Doyle ◽  
Jason R. Frank ◽  
Brian R. Holroyd ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Medical Education ◽  
Residency Education ◽  
Multiple Sources ◽  
High Stakes ◽  
Competency Based Medical Education ◽  
Professional Activities ◽  
Local Resources ◽  
Education Design ◽  
Competency Based

ABSTRACTCanadian specialist emergency medicine (EM) residency training is undergoing the most significant transformation in its history. This article describes the rationale, process, and redesign of EM competency-based medical education. The rationale for this evolution in residency education includes 1) improved public trust by increasing transparency of the quality and rigour of residency education, 2) improved fiscal accountability to government and institutions regarding specialist EM training, 3) improved assessment systems to replace poor functioning end-of-rotation assessment reports and overemphasis on high-stakes, end-of-training examinations, and 4) and tailored learning for residents to address individualized needs. A working group with geographic and stakeholder representation convened over a 2-year period. A consensus process for decision-making was used. Four key design features of the new residency education design include 1) specialty EM-specific outcomes to be achieved in residency; 2) designation of four progressive stages of training, linked to required learning experiences and entrustable professional activities to be achieved at each stage; 3) tailored learning that provides residency programs and learner flexibility to adapt to local resources and learner needs; and 4) programmatic assessment that emphasizes systematic, longitudinal assessments from multiple sources, and sampling sentinel abilities. Required future study includes a program evaluation of this complex education intervention to ensure that intended outcomes are achieved and unintended outcomes are identified.

scholarly journals LO72: Spotting potential opportunities for teachable moments

CJEM ◽  
2020 ◽  
Vol 22 (S1) ◽  
pp. S33-S34
Author(s):  
H. Alrimawi ◽  
MB BCh ◽  
S. Sample ◽  
T. Chan
Keyword(s):  
Emergency Medicine ◽  
Post Intervention ◽  
Historical Cohort ◽  
Competency Based Medical Education ◽  
Professional Activities ◽  
Significant Difference ◽  
Local Program ◽  
Competency Based ◽  
Student’S T ◽  
Case Based

Introduction: With the transition of Emergency Medicine into competency based medical education (CBME), entrustable professional activities (EPAs) are used to evaluate residents on performed clinical duties. This study aimed to determine if implementing a case-based orientation, designed to increase recognition of available EPAs, into CBME orientation would help residents increase the number of EPAs completed. Methods: We designed an intervention consisting of clinical cases that were reviewed by national EPA experts who identified which EPAs could be assessed from each case. A case-based session was incorporated into the 2019 CBME orientation for The McMaster Emergency Medicine Program. Postgraduate Year (PGY)1 residents read the cases and discussed which EPAs could be obtained with PGY2/faculty facilitators. The number of EPAs completed in the first two blocks of PGY1 was determined from local program data and Student's t-test was used to compare averages between cohorts. Results: We analyzed data from 22 trainees (7 in 2017, 8 in 2018, and 7 in 2019). In the first two blocks of PGY1, the intervention cohort (2019) had a significantly higher average number of EPAs completed per trainee (47.4 [SD 11.8]) than the historical cohort (25.3 [SD 6.7]) (p < 0.001) (Cohen's d = 2.3). No significant difference existed in the number EPAs obtained between the 2017/2018 cohorts, with averages of 24.3 [SD 6.8] and 26.1 [SD 7.0] per trainee respectively (p = 0.6). Conclusion: Implementation of a case-based orientation led by CBME-experienced facilitators nearly doubled the EPA acquisition rate of our PGY1s. The consistent EPA acquisition by the 2017/2018 cohorts suggest that the post-intervention increase was not solely due to developed familiarity with the CBME curriculum.

scholarly journals Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project

2020 ◽  
Author(s):  
Robert Carey ◽  
Grayson Wilson ◽  
Venkat Bandi ◽  
Debajyoti Mondal ◽  
Lynsey Martin ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Focus Groups ◽  
Residency Program ◽  
Research Process ◽  
Theory Approach ◽  
Assessment Data ◽  
Design Based Research ◽  
Professional Activities ◽  
Competency Based ◽  
The University

Background: Canadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents’ CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents’ needs. Methods: We utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements. Results: Resident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs. Conclusions: Our design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.

scholarly journals An analysis of core EPAs reveals a gap between curricular expectations and medical school graduates’ self-perceived level of competence

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Adrian Marty ◽  
Sonia Frick ◽  
Heidi Bruderer Enzler ◽  
Sabine Zundel
Keyword(s):  
Medical Education ◽  
Online Survey ◽  
Management Plan ◽  
Practical Skills ◽  
Professional Activities ◽  
Emergency Situations ◽  
New Curriculum ◽  
Competency Based ◽  
University Of Zurich ◽  
The University

Abstract Background Entrustable Professional Activities (EPAs) are being implemented worldwide as a means to promote competency-based medical education. In Switzerland, the new EPA-based curriculum for undergraduate medical education will be implemented in 2021. The aim of our study was to analyze the perceived, self-reported competence of graduates in 2019. The data represent a pre-implementation baseline and will provide guidance for curriculum developers. Methods Two hundred eighty-one graduates of the Master of Human Medicine program of the University of Zurich who had passed the Federal Licensing Exam in September 2019 were invited to complete an online survey. They were asked to rate their needed level of supervision (“observe only”, “direct, proactive supervision”, “indirect, reactive supervision”) for 46 selected EPAs. We compared the perceived competence with the expected competence of the new curriculum. Results The response rate was 54%. The need for supervision expressed by graduates varied considerably by EPA. The proportion of graduates rating themselves at expected level was high for “history taking”, “physical examination” “and documentation”; medium for “prioritizing differential diagnoses”, “interpreting results” and “developing and communicating a management plan”; low for “practical skills”; and very low for EPAs related to “urgent and emergency care”. Conclusions Currently, there are significant gaps between the expectations of curriculum developers and the perceived competences of students. This is most obvious for practical skills and emergency situations. The new curriculum will either need to fill this gap or expectations might need to be revised.

scholarly journals Developing a dashboard for faculty development in competency-based training programs: a design-based research project

2021 ◽  
Author(s):  
Yusuf Yilmaz ◽  
Robert Carey ◽  
Teresa Chan ◽  
Venkat Bandi ◽  
Shisong Wang ◽  
...  
Keyword(s):  
Faculty Development ◽  
Training Programs ◽  
Research Process ◽  
Assessment Data ◽  
Design Based Research ◽  
Program Leaders ◽  
Professional Activities ◽  
Narrative Assessment ◽  
Competency Based ◽  
The University

Background: Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires frequent assessments of entrustable professional activities (EPAs). Faculty struggle to provide helpful feedback and assign appropriate entrustment scores. CBME faculty development initiatives rarely incorporate teaching metrics. Dashboards could be used to visualize faculty assessment data to support faculty development. Methods: Using a design-based research process, we identified faculty development needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. Data was collected within the emergency medicine residency program at the University of Saskatchewan through interviews with program leaders, faculty development experts, and faculty participating in development sessions. Two investigators thematically analyzed interview transcripts to identify faculty needs that were audited by a third investigator. The needs were described using representative quotes and the dashboard elements designed to address them. Results: Between July 1, 2019 and December 11, 2020 we conducted 15 interviews with nine participants (two program leaders, three faculty development experts, and four faculty members). Three needs emerged as themes from the analysis: analysis of assessments, contextualization of assessments, and accessible reporting. We addressed these needs by designing an accessible dashboard to present contextualized quantitative and narrative assessment data for each faculty member. Conclusions: We identified faculty development needs related to EPA assessments and designed dashboard elements to meet them. The resulting dashboard was used for faculty development sessions. This work will inform the development of CBME assessment dashboards for faculty.

scholarly journals Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Severin Pinilla ◽  
Alexandra Kyrou ◽  
Stefan Klöppel ◽  
Werner Strik ◽  
Christoph Nissen ◽  
...  
Keyword(s):  
Clinical Encounter ◽  
Entrustable Professional Activities ◽  
Written Feedback ◽  
Professional Activities ◽  
Quality Feedback ◽  
Observational Prospective Cohort Study ◽  
Competency Based ◽  
Patient’S History ◽  
Indirect Supervision ◽  
Clerkship Rotation

Abstract Background Entrustable professional activities (EPAs) in competency-based, undergraduate medical education (UME) have led to new formative workplace-based assessments (WBA) using entrustment-supervision scales in clerkships. We conducted an observational, prospective cohort study to explore the usefulness of a WBA designed to assess core EPAs in a psychiatry clerkship. Methods We analyzed changes in self-entrustment ratings of students and the supervisors’ ratings per EPA. Timing and frequencies of learner-initiated WBAs based on a prospective entrustment-supervision scale and resultant narrative feedback were analyzed quantitatively and qualitatively. Predictors for indirect supervision levels were explored via regression analysis, and narrative feedback was coded using thematic content analysis. Students evaluated the WBA after each clerkship rotation. Results EPA 1 (“Take a patient’s history”), EPA 2 (“Assess physical & mental status”) and EPA 8 (“Document & present a clinical encounter”) were most frequently used for learner-initiated WBAs throughout the clerkship rotations in a sample of 83 students. Clinical residents signed off on the majority of the WBAs (71%). EPAs 1, 2, and 8 showed the largest increases in self-entrustment and received most of the indirect supervision level ratings. We found a moderate, positive correlation between self-entrusted supervision levels at the end of the clerkship and the number of documented entrustment-supervision ratings per EPA (p < 0.0001). The number of entrustment ratings explained 6.5% of the variance in the supervisors’ ratings for EPA 1. Narrative feedback was documented for 79% (n = 214) of the WBAs. Most narratives addressed the Medical Expert role (77%, n = 208) and used reinforcement (59%, n = 161) as a feedback strategy. Students perceived the feedback as beneficial. Conclusions Using formative WBAs with an entrustment-supervision scale and prompts for written feedback facilitated targeted, high-quality feedback and effectively supported students’ development toward self-entrusted, indirect supervision levels.

scholarly journals Preparing for CBME: How often are faculty observing residents?

2020 ◽  
Author(s):  
Sheenagh J K George ◽  
Sarah Manos ◽  
Kenny K Wong
Keyword(s):  
Direct Observation ◽  
Assessment Tools ◽  
Inpatient Setting ◽  
Institutional Research ◽  
Professional Activities ◽  
Inpatient Settings ◽  
Competency Based ◽  
Resident Assessment ◽  
Outpatient Settings ◽  
Common Barriers

Abstract Background The Royal College of Physicians and Surgeons of Canada officially launched ‘Competence by Design’ in July 2017, moving from time-based to outcomes-based training. Transitioning to competency-based medical education (CBME) necessitates change in resident assessment. A greater frequency of resident observation will likely be required to adequately assess whether entrustable professional activities have been achieved. Purpose Characterize faculty and resident experiences of direct observation in a single paediatric residency program, pre-CBME implementation. Qualitatively describe participants’ perceived barriers and incentives to participating in direct observation. Methods Surveys were sent to paediatric residents and faculty asking for demographics, the frequency of resident observation during an average 4-week rotation, perceived ideal frequency of observation, and factors influencing observation frequency. Descriptive data were analyzed. Institutional research ethics board approval was received. Results The response rate was 54% (34/68 faculty and 16/25 residents). When asked the MAXIMUM frequency FACULTY observed a resident take a history, perform a physical examination, or deliver a plan, the median faculty reply was 1, 2, and 3, for outpatient settings and 0, 1, and 2, for inpatient settings. The median RESIDENT reply was 2, 4, and 10 for outpatient settings and 1, 2, and 20 for inpatient settings. When asked the MINIMUM frequency for each domain, the median FACULTY and RESIDENT reply was 0, except for delivering a plan in the inpatient setting. Faculty reported observing seniors delivering the plan more frequently than junior residents. Faculty and resident median replies for how frequently residents should be observed for each domain were the same, three to four, three to four, and five to six times. Four per cent of faculty reported regularly scheduling observations, and 77% of residents regularly ask to be observed. The most common barriers to observation were too many patients to see and both faculty and residents were seeing patients at the same time. Most faculty and resident responders felt that observation frequency could be improved if scheduled at the start of the rotation; faculty were provided a better tool for assessment; and if residents asked to be observed. Conclusions This study provides baseline data on how infrequent faculty observation is occurring and at a frequency lower than what faculty and residents feel is necessary. The time needed for observation competes with clinical service demands, but better scheduling strategies and assessment tools may help.

scholarly journals A search for entrustable professional activities for the 1st year pathology postgraduate trainees

2018 ◽  
Vol 10 (01) ◽  
pp. 026-030 ◽  
Author(s):  
Pooja Sharma ◽  
Nadeem Tanveer ◽  
Aditi Goyal
Keyword(s):  
Medical Education ◽  
Training Program ◽  
Paradigm Shift ◽  
Entrustable Professional Activities ◽  
Average Score ◽  
Professional Activities ◽  
The Past ◽  
Cumulative Score ◽  
Competency Based ◽  
The Right

Abstract INTRODUCTION: During the past decade, there has been a paradigm shift in medical education from the problem-based learning to competency-based training. This has forced a rethink on the way we evaluate the residents and finally give them the right to handle patients independently. This study makes the first attempt towards designing competency-based training program for pathology residents by formulating the entrustable professional activities (EPAs) for the 1st year pathology residents. MATERIALS AND METHODS: A questionnaire comprising 18 potential EPAs in histopathology and 12 potential EPAs in cytology were circulated among the residents of Pathology Department. The respondents were asked to grade the EPAs on a scale of 0–4 based on how important they considered that activity as EPA. The cumulative score of each EPA was divided by the number of respondents to arrive at the average score. The EPAs with an average score of 3 or more qualified to be shortlisted as consensus EPAs. RESULTS: Five activities each of histopathology and cytopathology had an average score of 3 or above and were shortlisted as EPAs for the 1st year pathology postgraduates. Each of these was also mapped to their respective competencies. CONCLUSION: There is an urgent need to restructure the postgraduate pathology curriculum in line with competency-based training. This study is the first step in this direction.

Sign in / Sign up

Export Citation Format

Share Document