scholarly journals An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion

Sensors ◽  
2019 ◽  
Vol 19 (20) ◽  
pp. 4586 ◽  
Author(s):  
Chunxu Li ◽  
Ashraf Fahmy ◽  
Johann Sienz
Keyword(s):  
Kalman Filter ◽  
Augmented Reality ◽  
Sensor Fusion ◽  
Kalman Filtering ◽  
Control Process ◽  
Human Robot Interaction ◽  
Post Processing ◽  
Robot Interaction ◽  
Kinect Camera ◽  
Interaction Interface

In this paper, the application of Augmented Reality (AR) for the control and adjustment of robots has been developed, with the aim of making interaction and adjustment of robots easier and more accurate from a remote location. A LeapMotion sensor based controller has been investigated to track the movement of the operator hands. The data from the controller allows gestures and the position of the hand palm’s central point to be detected and tracked. A Kinect V2 camera is able to measure the corresponding motion velocities in x, y, z directions after our investigated post-processing algorithm is fulfilled. Unreal Engine 4 is used to create an AR environment for the user to monitor the control process immersively. Kalman filtering (KF) algorithm is employed to fuse the position signals from the LeapMotion sensor with the velocity signals from the Kinect camera sensor, respectively. The fused/optimal data are sent to teleoperate a Baxter robot in real-time by User Datagram Protocol (UDP). Several experiments have been conducted to test the validation of the proposed method.

scholarly journals Projection Surfaces Detection and Image Correction for Mobile Robots in HRI

2017 ◽  
Vol 2017 ◽  
pp. 1-16
Author(s):  
Enrique Fernández-Rodicio ◽  
Víctor González-Pacheco ◽  
José Carlos Castillo ◽  
Álvaro Castro-González ◽  
María Malfaz ◽  
...  
Keyword(s):  
Mobile Robot ◽  
Vertical Plane ◽  
Vertical Surface ◽  
Human Robot Interaction ◽  
Image Correction ◽  
Robot Interaction ◽  
Kinect Camera ◽  
Fitting Algorithm ◽  
The Right ◽  
Large Groups

Projectors have become a widespread tool to share information in Human-Robot Interaction with large groups of people in a comfortable way. Finding a suitable vertical surface becomes a problem when the projector changes positions when a mobile robot is looking for suitable surfaces to project. Two problems must be addressed to achieve a correct undistorted image: (i) finding the biggest suitable surface free from obstacles and (ii) adapting the output image to correct the distortion due to the angle between the robot and a nonorthogonal surface. We propose a RANSAC-based method that detects a vertical plane inside a point cloud. Then, inside this plane, we apply a rectangle-fitting algorithm over the region in which the projector can work. Finally, the algorithm checks the surface looking for imperfections and occlusions and transforms the original image using a homography matrix to display it over the area detected. The proposed solution can detect projection areas in real-time using a single Kinect camera, which makes it suitable for applications where a robot interacts with other people in unknown environments. Our Projection Surfaces Detector and the Image Correction module allow a mobile robot to find the right surface and display images without deformation, improving its ability to interact with people.

Sign in / Sign up

Export Citation Format

Share Document