Summative evaluation with a full-scale patient simulator: Challenges and adaptations

In this paper we outline considerations that went into designing and executing a full-scale simulator-based summative evaluation of four different display configurations for presenting information about anesthetized patients to an anesthesiologist. Although patient simulators appear to provide a “natural laboratory” for evaluating medical device innovations and equipment interface concepts, the software underlying patient simulators can be unequal to the challenges posed by the need for good representation of patient physiology and good experimental control. Moreover, the opportunities that full-scale patient simulators can offer for completely interactive, event-driven scenarios can present problems for experimental control and can promote participant hypervigilance. We describe the design of our experimental scenarios, the challenges our scenarios posed for simulator software and how we overcame those challenges, the design of a distractor task, and the methodology used to ensure we collected behavioral data sensitive to the manipulations of interest. Our adaptations in the face of challenges posed by the full-scale simulator context let us design an experiment that was highly informative about the advantages and disadvantages of the display configurations of interest.