Inter-rater Reliability in Clinical Assessments: Do Examiner Pairings Influence Candidate Ratings?

BackgroundThe reliability of clinical assessments is known to vary considerably with inter-rater reliability a key contributor. Many of the mechanisms that contribute to inter-rater reliability however remain largely unexplained and unclear. While research in other fields suggests personality of raters can impact ratings, studies looking at personality factors in clinical assessments are few. Many schools use the approach of pairing examiners in clinical assessments and asking them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score. Could personality factors have an impact? Methods: A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate’s observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality with data collected by questionnaire. A purposeful sample of doctors who examine in the Final Medical examination at our institution was recruited. Results: Variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12). 75% of examiners (N=9) scored below average for neuroticism and 75% also scored high or very high for extroversion. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score. Conclusions: While the variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12), the reliability statistics for both assessments were comparable. However, using paired examiners resulted in a more accurate and robust score than simply averaging two independent examiners scores. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score. These findings could have implications for the organisation and administration of clinical assessments. Further studies with larger numbers of participants might establish if personality testing before choosing examiner pairs should be adopted.

Examiner Variability in Clinical Assessments: Do Examiner Pairings Influence Candidate Ratings?

Background The reliability of clinical assessments is known to vary considerably and inter-examiner variability is a key contributor. This may result in significant differences in scores between comparable candidates, a serious challenge in medical education. An approach frequently adopted to avoid this and improve reliability is to pair examiners and ask them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score.Methods A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate’s observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality. Demographic and personality data was collected by questionnaire. A purposeful sample of medical doctors who examine in the Final Medical examination at our institution was recruited.Results Variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12). 75% of examiners (N=9) scored below average for neuroticism and 75% also scored high or very high for extroversion. Two thirds scored high or very high for conscientiousness. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score.Conclusions While the variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12), the reliability statistics for both assessments were comparable. Using paired examiners resulted in a more accurate and robust score than simply averaging two independent examiners scores. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score. These findings could have implications for the organisation and administration of clinical assessments. Further studies with larger numbers of participants might establish if personality testing before choosing examiner pairs could be utilised to help pair examiners and improve examiner variability.

On the brink of universalism: the Emergency Hospital Services in Second World War Northern Ireland

This chapter examines the medical responses to the outbreak of the Second World War in Northern Ireland with an emphasis on Belfast. It focuses on the emergence of the Emergency Medical Service (EMS), established throughout the United Kingdom in response to the anticipation of likely air-raid casualties. Pre- Second World War hospital services in Belfast were piecemeal, lacking integration and provided by varying independent bodies including voluntary, municipal and poor law authorities. This chapter argues that the EMS brought a degree of integration previously unknown in Northern Irish health organisation and administration. This new found integration of war time medical services greatly influenced the ‘post war reconstruction’ and ‘planning’ of health. The chapter examines Northern Irish contexts and suggests that Irish and Northern Irish health care systems began to dramatically diverge during wartime. It also examines the relationship between Belfast and London’s Ministry of Health, and the challenges of devolved healthcare. In addition, the chapter examines the public health responses to the 1941 Belfast Blitz, and the overall effectiveness of wartime health services.

The local church as a non-governmental organisation in the fight against poverty: A historical overview of Bethulie 1933–1935

Poverty is one of the greatest threats to society. In South Africa it is also one of the biggest challenges. This article starts with the challenges put to society by Mr Trevor Manuel at the Carnegie 3 conference. It then explores the possibility of if and how the church can act as a non-governmental organisation in the fight against poverty. A historical overview of the actions of Rev. E.P. Groenewald, during the drought of 1933–1934 in the Dutch Reformed Church Bethulie, serves as a case study of how the church can make a difference. It, however, also illustrates the many pitfalls on this challenging road. The article comes to the conclusion that the main challenge of the church in the fight against poverty is to act as a non-governmental organisation, which transforms values and assists society with good organisation and administration.