scholarly journals Ambiguity-Free Optical–Inertial Tracking for Augmented Reality Headsets

Sensors ◽  
2020 ◽  
Vol 20 (5) ◽  
pp. 1444 ◽  
Author(s):  
Fabrizio Cutolo ◽  
Virginia Mamone ◽  
Nicola Carbonaro ◽  
Vincenzo Ferrari ◽  
Alessandro Tognetti
Keyword(s):  
Augmented Reality ◽  
Optical Tracking ◽  
Tracking Data ◽  
Tracking Accuracy ◽  
Short Baseline ◽  
Head Mounted Display ◽  
Lighting Conditions ◽  
Mobile Graphics ◽  
Custom Made ◽  
Manual Tasks

The increasing capability of computing power and mobile graphics has made possible the release of self-contained augmented reality (AR) headsets featuring efficient head-anchored tracking solutions. Ego motion estimation based on well-established infrared tracking of markers ensures sufficient accuracy and robustness. Unfortunately, wearable visible-light stereo cameras with short baseline and operating under uncontrolled lighting conditions suffer from tracking failures and ambiguities in pose estimation. To improve the accuracy of optical self-tracking and its resiliency to marker occlusions, degraded camera calibrations, and inconsistent lighting, in this work we propose a sensor fusion approach based on Kalman filtering that integrates optical tracking data with inertial tracking data when computing motion correlation. In order to measure improvements in AR overlay accuracy, experiments are performed with a custom-made AR headset designed for supporting complex manual tasks performed under direct vision. Experimental results show that the proposed solution improves the head-mounted display (HMD) tracking accuracy by one third and improves the robustness by also capturing the orientation of the target scene when some of the markers are occluded and when the optical tracking yields unstable and/or ambiguous results due to the limitations of using head-anchored stereo tracking cameras under uncontrollable lighting conditions.

scholarly journals Augmented Reality-Assisted Craniotomy for Parasagittal and Convexity En Plaque Meningiomas and Custom-Made Cranio-Plasty: A Preliminary Laboratory Report

2021 ◽  
Vol 18 (19) ◽  
pp. 9955
Author(s):  
Nicola Montemurro ◽  
Sara Condino ◽  
Nadia Cattari ◽  
Renzo D’Amato ◽  
Vincenzo Ferrari ◽  
...  
Keyword(s):  
Augmented Reality ◽  
Surgical Simulation ◽  
Cranial Vault ◽  
Bone Flap ◽  
Patient Specific ◽  
Real Patient ◽  
Head Mounted Display ◽  
Laboratory Report ◽  
Custom Made ◽  
Wearable Augmented Reality

Background: This report discusses the utility of a wearable augmented reality platform in neurosurgery for parasagittal and convexity en plaque meningiomas with bone flap removal and custom-made cranioplasty. Methods: A real patient with en plaque cranial vault meningioma with diffuse and extensive dural involvement, extracranial extension into the calvarium, and homogeneous contrast enhancement on gadolinium-enhanced T1-weighted MRI, was selected for this case study. A patient-specific manikin was designed starting with the segmentation of the patient’s preoperative MRI images to simulate a craniotomy procedure. Surgical planning was performed according to the segmented anatomy, and customized bone flaps were designed accordingly. During the surgical simulation stage, the VOSTARS head-mounted display was used to accurately display the planned craniotomy trajectory over the manikin skull. The precision of the craniotomy was assessed based on the evaluation of previously prepared custom-made bone flaps. Results: A bone flap with a radius 0.5 mm smaller than the radius of an ideal craniotomy fitted perfectly over the performed craniotomy, demonstrating an error of less than ±1 mm in the task execution. The results of this laboratory-based experiment suggest that the proposed augmented reality platform helps in simulating convexity en plaque meningioma resection and custom-made cranioplasty, as carefully planned in the preoperative phase. Conclusions: Augmented reality head-mounted displays have the potential to be a useful adjunct in tumor surgical resection, cranial vault lesion craniotomy and also skull base surgery, but more study with large series is needed.

scholarly journals Holographic patient tracking after bed movement for augmented reality neuronavigation using a head-mounted display

2021 ◽  
Author(s):  
T. Fick ◽  
J.A.M. van Doormaal ◽  
E.W. Hoving ◽  
L. Regli ◽  
T.P.C. van Doormaal
Keyword(s):  
Augmented Reality ◽  
Anatomical Landmarks ◽  
Registration Accuracy ◽  
Patient Tracking ◽  
First Report ◽  
Head Mounted Display ◽  
Reference Array ◽  
Custom Made ◽  
Before And After ◽  
Neuronavigation System

Abstract Background Holographic neuronavigation has several potential advantages compared to conventional neuronavigation systems. We present the first report of a holographic neuronavigation system with patient-to-image registration and patient tracking with a reference array using an augmented reality head-mounted display (AR-HMD). Methods Three patients undergoing an intracranial neurosurgical procedure were included in this pilot study. The relevant anatomy was first segmented in 3D and then uploaded as holographic scene in our custom neuronavigation software. Registration was performed using point-based matching using anatomical landmarks. We measured the fiducial registration error (FRE) as the outcome measure for registration accuracy. A custom-made reference array with QR codes was integrated in the neurosurgical setup and used for patient tracking after bed movement. Results Six registrations were performed with a mean FRE of 8.5 mm. Patient tracking was achieved with no visual difference between the registration before and after movement. Conclusions This first report shows a proof of principle of intraoperative patient tracking using a standalone holographic neuronavigation system. The navigation accuracy should be further optimized to be clinically applicable. However, it is likely that this technology will be incorporated in future neurosurgical workflows because the system improves spatial anatomical understanding for the surgeon.

scholarly journals In Situ Visualization for 3D Ultrasound-Guided Interventions with Augmented Reality Headset

2021 ◽  
Vol 8 (10) ◽  
pp. 131
Author(s):  
Nadia Cattari ◽  
Sara Condino ◽  
Fabrizio Cutolo ◽  
Mauro Ferrari ◽  
Vincenzo Ferrari
Keyword(s):  
Augmented Reality ◽  
Ultrasound Imaging ◽  
High Precision ◽  
Soft Tissues ◽  
Imaging System ◽  
3D Ultrasound ◽  
Integrated System ◽  
Head Mounted Display ◽  
Manual Tasks

Augmented Reality (AR) headsets have become the most ergonomic and efficient visualization devices to support complex manual tasks performed under direct vision. Their ability to provide hands-free interaction with the augmented scene makes them perfect for manual procedures such as surgery. This study demonstrates the reliability of an AR head-mounted display (HMD), conceived for surgical guidance, in navigating in-depth high-precision manual tasks guided by a 3D ultrasound imaging system. The integration between the AR visualization system and the ultrasound imaging system provides the surgeon with real-time intra-operative information on unexposed soft tissues that are spatially registered with the surrounding anatomic structures. The efficacy of the AR guiding system was quantitatively assessed with an in vitro study simulating a biopsy intervention aimed at determining the level of accuracy achievable. In the experiments, 10 subjects were asked to perform the biopsy on four spherical lesions of decreasing sizes (10, 7, 5, and 3 mm). The experimental results showed that 80% of the subjects were able to successfully perform the biopsy on the 5 mm lesion, with a 2.5 mm system accuracy. The results confirmed that the proposed integrated system can be used for navigation during in-depth high-precision manual tasks.

Augmented Reality Procedure Assistance System for Operator Training and Simulation

2020 ◽  
Vol 64 (1) ◽  
pp. 1176-1180
Author(s):  
Eugene Hayden ◽  
Kang Wang ◽  
Chengjie Wu ◽  
Shi Cao
Keyword(s):  
Augmented Reality ◽  
Situation Awareness ◽  
Mental Workload ◽  
Field Of View ◽  
Task Completion ◽  
Operator Training ◽  
Head Mounted Display ◽  
Subject Design ◽  
Procedural Tasks ◽  
Speed Accuracy

This study explores the design, implementation, and evaluation of an Augmented Reality (AR) prototype that assists novice operators in performing procedural tasks in simulator environments. The prototype uses an optical see-through head-mounted display (OST HMD) in conjunction with a simulator display to supplement sequences of interactive visual and attention-guiding cues to the operator’s field of view. We used a 2x2 within-subject design to test two conditions: with/without AR-cues, each condition had a voice assistant and two procedural tasks (preflight and landing). An experiment examined twenty-six novice operators. The results demonstrated that augmented reality had benefits in terms of improved situation awareness and accuracy, however, it yielded longer task completion time by creating a speed-accuracy trade-off effect in favour of accuracy. No significant effect on mental workload is found. The results suggest that augmented reality systems have the potential to be used by a wider audience of operators.

scholarly journals Augmented Reality, Mixed Reality, and Hybrid Approach in Healthcare Simulation: A Systematic Review

2021 ◽  
Vol 11 (5) ◽  
pp. 2338
Author(s):  
Rosanna Maria Viglialoro ◽  
Sara Condino ◽  
Giuseppe Turini ◽  
Marina Carbone ◽  
Vincenzo Ferrari ◽  
...  
Keyword(s):  
Augmented Reality ◽  
Mixed Reality ◽  
Ethical Issues ◽  
Medical Training ◽  
Hybrid Approach ◽  
Dental Surgery ◽  
Feasibility Assessment ◽  
Hybrid Approaches ◽  
Custom Made ◽  
Improved Performance

Simulation-based medical training is considered an effective tool to acquire/refine technical skills, mitigating the ethical issues of Halsted’s model. This review aims at evaluating the literature on medical simulation techniques based on augmented reality (AR), mixed reality (MR), and hybrid approaches. The research identified 23 articles that meet the inclusion criteria: 43% combine two approaches (MR and hybrid), 22% combine all three, 26% employ only the hybrid approach, and 9% apply only the MR approach. Among the studies reviewed, 22% use commercial simulators, whereas 78% describe custom-made simulators. Each simulator is classified according to its target clinical application: training of surgical tasks (e.g., specific tasks for training in neurosurgery, abdominal surgery, orthopedic surgery, dental surgery, otorhinolaryngological surgery, or also generic tasks such as palpation) and education in medicine (e.g., anatomy learning). Additionally, the review assesses the complexity, reusability, and realism of the physical replicas, as well as the portability of the simulators. Finally, we describe whether and how the simulators have been validated. The review highlights that most of the studies do not have a significant sample size and that they include only a feasibility assessment and preliminary validation; thus, further research is needed to validate existing simulators and to verify whether improvements in performance on a simulated scenario translate into improved performance on real patients.

scholarly journals Using Brain Activity Patterns to Differentiate Real and Virtual Attended Targets during Augmented Reality Scenarios

Information ◽  
2021 ◽  
Vol 12 (6) ◽  
pp. 226
Author(s):  
Lisa-Marie Vortmann ◽  
Leonid Schwenke ◽  
Felix Putze
Keyword(s):  
Augmented Reality ◽  
Eye Tracking ◽  
Brain Activity ◽  
Activity Patterns ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Neural Net ◽  
Dependent Manner ◽  
Tracking Data ◽  
Average Accuracy

Augmented reality is the fusion of virtual components and our real surroundings. The simultaneous visibility of generated and natural objects often requires users to direct their selective attention to a specific target that is either real or virtual. In this study, we investigated whether this target is real or virtual by using machine learning techniques to classify electroencephalographic (EEG) and eye tracking data collected in augmented reality scenarios. A shallow convolutional neural net classified 3 second EEG data windows from 20 participants in a person-dependent manner with an average accuracy above 70% if the testing data and training data came from different trials. This accuracy could be significantly increased to 77% using a multimodal late fusion approach that included the recorded eye tracking data. Person-independent EEG classification was possible above chance level for 6 out of 20 participants. Thus, the reliability of such a brain–computer interface is high enough for it to be treated as a useful input mechanism for augmented reality applications.

scholarly journals Fooling the size–weight illusion—Using augmented reality to eliminate the effect of size on perceptions of heaviness and sensorimotor prediction

2021 ◽  
Author(s):  
Nina Rohrbach ◽  
Joachim Hermsdörfer ◽  
Lisa-Marie Huber ◽  
Annika Thierfelder ◽  
Gavin Buckingham
Keyword(s):  
Augmented Reality ◽  
Visual Cues ◽  
Equal Mass ◽  
Small Object ◽  
Verbal Reports ◽  
Fully Integrated ◽  
Head Mounted Display ◽  
Lift Off ◽  
Cognitive Knowledge ◽  
Weight Illusion

AbstractAugmented reality, whereby computer-generated images are overlaid onto the physical environment, is becoming significant part of the world of education and training. Little is known, however, about how these external images are treated by the sensorimotor system of the user – are they fully integrated into the external environmental cues, or largely ignored by low-level perceptual and motor processes? Here, we examined this question in the context of the size–weight illusion (SWI). Thirty-two participants repeatedly lifted and reported the heaviness of two cubes of unequal volume but equal mass in alternation. Half of the participants saw semi-transparent equally sized holographic cubes superimposed onto the physical cubes through a head-mounted display. Fingertip force rates were measured prior to lift-off to determine how the holograms influenced sensorimotor prediction, while verbal reports of heaviness after each lift indicated how the holographic size cues influenced the SWI. As expected, participants who lifted without augmented visual cues lifted the large object at a higher rate of force than the small object on early lifts and experienced a robust SWI across all trials. In contrast, participants who lifted the (apparently equal-sized) augmented cubes used similar force rates for each object. Furthermore, they experienced no SWI during the first lifts of the objects, with a SWI developing over repeated trials. These results indicate that holographic cues initially dominate physical cues and cognitive knowledge, but are dismissed when conflicting with cues from other senses.

scholarly journals High Precision Optical Tracking System Based on near Infrared Trinocular Stereo Vision

Sensors ◽  
2021 ◽  
Vol 21 (7) ◽  
pp. 2528
Author(s):  
Songlin Bi ◽  
Yonggang Gu ◽  
Jiaqi Zou ◽  
Lianpo Wang ◽  
Chao Zhai ◽  
...  
Keyword(s):  
High Precision ◽  
Stereo Vision ◽  
Near Infrared ◽  
Nearest Neighbor ◽  
Tracking System ◽  
Optical Tracking ◽  
Tracking Accuracy ◽  
Optical Tracking System ◽  
Dynamic Tracking ◽  
Trinocular Stereo

A high precision optical tracking system (OTS) based on near infrared (NIR) trinocular stereo vision (TSV) is presented in this paper. Compared with the traditional OTS on the basis of binocular stereo vision (BSV), hardware and software are improved. In the hardware aspect, a NIR TSV platform is built, and a new active tool is designed. Imaging markers of the tool are uniform and complete with large measurement angle (>60°). In the software aspect, the deployment of extra camera brings high computational complexity. To reduce the computational burden, a fast nearest neighbor feature point extraction algorithm (FNNF) is proposed. The proposed method increases the speed of feature points extraction by hundreds of times over the traditional pixel-by-pixel searching method. The modified NIR multi-camera calibration method and 3D reconstruction algorithm further improve the tracking accuracy. Experimental results show that the calibration accuracy of the NIR camera can reach 0.02%, positioning accuracy of markers can reach 0.0240 mm, and dynamic tracking accuracy can reach 0.0938 mm. OTS can be adopted in high-precision dynamic tracking.

Visual Coherence in Mixed Reality: A Systematic Enquiry

2017 ◽  
Vol 26 (1) ◽  
pp. 16-41 ◽  
Author(s):  
Jonny Collins ◽  
Holger Regenbrecht ◽  
Tobias Langlotz
Keyword(s):  
Augmented Reality ◽  
Mixed Reality ◽  
Focus Of Attention ◽  
User Experiences ◽  
Head Mounted Display ◽  
Virtual And Augmented Reality

Virtual and augmented reality, and other forms of mixed reality (MR), have become a focus of attention for companies and researchers. Before they can become successful in the market and in society, those MR systems must be able to deliver a convincing, novel experience for the users. By definition, the experience of mixed reality relies on the perceptually successful blending of reality and virtuality. Any MR system has to provide a sensory, in particular visually coherent, set of stimuli. Therefore, issues with visual coherence, that is, a discontinued experience of a MR environment, must be avoided. While it is very easy for a user to detect issues with visual coherence, it is very difficult to design and implement a system for coherence. This article presents a framework and exemplary implementation of a systematic enquiry into issues with visual coherence and possible solutions to address those issues. The focus is set on head-mounted display-based systems, notwithstanding its applicability to other types of MR systems. Our framework, together with a systematic discussion of tangible issues and solutions for visual coherence, aims at guiding developers of mixed reality systems for better and more effective user experiences.

Sign in / Sign up

Export Citation Format

Share Document