The critical role of direct observation in entrustment decisions

Background: Entrustment decisions may be retrospective (based on past experiences with a trainee) or real-time (based on direct observation). We investigated judgments of entrustment based on assessor prior knowledge of candidates and based on systematic direct observation, conducted in an objective structured clinical exam (OSCE). Methods: Sixteen faculty examiners provided 287 retrospective and real-time entrustment ratings of 16 cardiology trainees during OSCE stations in 2019 and 2020. Reliability and validity of these ratings were assessed by comparing correlations across stations as a measure of reliability, differences across postgraduate years as an index of construct validity, correlation to standardized in-training exam (ITE) as a measure of criterion validity, and reclassification of entrustment as a measure of consequential validity. Results: Both retrospective and real-time assessments were highly reliable (all intra-class correlations >0.86). Both increased with year of postgraduate training. Real-time entrustment ratings were significantly correlated with standardized ITE scores; retrospective ratings were not. Real-time ratings explained 37% (2019) and 46% (2020) of variance in examination scores vs. 21% (2019) and 7% (2020) for retrospective ratings. Direct observation resulted in a different level of entrustment compared with retrospective ratings in 44% of cases (p = <0.001). Conclusions: Ratings based on direct observation made unique contributions to entrustment decisions.

Procedural sensitivities of effect sizes for single-case designs with directly observed behavioral outcome measures

A wide variety of effect size indices have been proposed for quantifying the magnitude of treatment effects in single-case designs. Commonly used measures include parametric indices such as the standardized mean difference, as well as non-overlap measures such as the percentage of non-overlapping data, improvement rate difference, and non-overlap of all pairs. Currently, little is known about the properties of these indices when applied to behavioral data collected by systematic direct observation, even though systematic direct observation is the most common method for outcome measurement in single-case research. This study uses Monte Carlo simulation to investigate the properties of several widely used single-case effect size measures when applied to systematic direct observation data. Results indicate that the magnitude of the non-overlap measures and of the standardized mean difference can be strongly influenced by procedural details of the study's design, which is a significant limitation to using these indices as effect sizes for meta-analysis of single-case designs. A less widely used parametric index, the log-response ratio, has the advantage of being insensitive to sample size and observation session length, although its magnitude is influenced by the use of partial interval recording.

The Effectiveness of Daily Behavior Report Cards for Children With ADHD: A Meta-Analysis

Objective: This meta-analysis examined group-design studies investigating the effectiveness of Daily Behavior Report Cards (DBRC) as a school-based intervention to manage the classroom behavior of students with ADHD. Methods: A search of three article databases (PsycINFO, ERIC and Medline) identified seven group design evaluations of DBRC interventions. This meta-analysis included a total of 272 participants, with an average age of 7.9 years old. Three of the studies compared a control group to a treatment group with randomized group assignment, one study compared a control group to three treatment groups, two studies compared pre-and post-treatment scores in the same group, and one study compared pre-and post-treatment results of two intervention groups without random assignment. Dependent measures for these studies were teacher ratings (n = 5) and systematic direct observation of student academic and social behaviour (n = 2). Standardized mean differences ( Hedge’s g) were calculated to obtain a pooled effect size using fixed effects. Results: DBRCs were associated with reductions teacher-rated ADHD symptoms, with a Hedge’s g of 0.36 (95% CI: 0.12-0.60, z=2.93, p ≤ .005) with low heterogeneity (Q-value: 2.40, I2= 0.00). This result excluded two studies that used observational coding instead of standardized tests to evaluate the effects of the intervention. A moderator analysis indicated that the effect size for systematic direct observation was large ( Hedge’s G = 1.05[95% CI: 0.66-1.44, z=5.25, p ≤ .00]), with very high heterogeneity (Q-value: 46.34, I2: 93.53). A second moderator analysis found differences in the effects of DBRCs for comorbid externalizing symptoms with an overall effect size of 0.34 (95%CI: -0.04-0.72, z=1.76 p =0.08) with high heterogeneity (Q-value: 3.98, I2: 74.85). Conclusions: DBRCs effectively reduce the frequency and severity of ADHD symptoms in classroom settings. Additionally, they have a significant effect on co-occuring externalizing behaviors. It appears that systematic direct observation may be a more sensitive measure of treatment effects compared to teacher ratings of ADHD symptoms.

Brief Report: A Preliminary Investigation of a Mindful Breathing Intervention to Increase Academic Engagement in an Alternative Educational Setting

The current study used a multiple-baseline across subjects design to examine the effect of a daily, audio-delivered, mindful breathing intervention on adolescents with emotional and behavioral difficulties attending an alternative educational setting. Academic engagement served as the primary dependent variable, with disruptive behavior, feasibility, and acceptability serving as secondary dependent variables. Four students entered the baseline phase; however, only two students entered the intervention phase. Results indicated that participants displayed increases in academic engagement as measured by both direct behavior rating and systematic direct observation that were maintained at 6-week follow-up. Decreases in disruptive behavior were also observed. Because the effects were not replicated three times, changes cannot be directly attributed to the intervention. In addition, teachers and students perceived the intervention to be both feasible and acceptable. Implications of the findings are discussed.

Maternal perceptions on Well Child Care

The aim was to analyze well child nursing care (WCC) and to distinguish if the care is procedure or user centered. The concepts of the nursing work process and the micro-politics of health work supported this qualitative study. Systematic direct observation of 87 WCC consultations was accomplished at one Family Medicine Unit and semistructured interviews were held with 25 mothers who attended WCC consultations with their child. Data saturation and understanding of the meaning were the criteria used to determine the number of observations and interviews. Thematic analysis was applied. The activity was focused on procedures, which cannot be considered WHCC. The mothers value comprehensive care and request information on the growth and development of their children. Educative topic should be addressed and trust relations should be established with a view to user-centered care delivery.