Graphics to H.264 video encoding for 3D scene representation and interaction on mobile devices using region of interest

A Graphics Adaptation Framework and Video Streaming Technique for 3D Scene Representation and Interaction on Mobile Devices

Lecture Notes in Computer Science - Universal Acess in Human Computer Interaction. Coping with Diversity ◽

10.1007/978-3-540-73279-2_54 ◽

2007 ◽

pp. 481-490

Author(s):

Congdu Nguyen ◽

Minh Tuan Le ◽

Dae-Il Yoon ◽

Hae-Kwang Kim

Keyword(s):

Mobile Devices ◽

Video Streaming ◽

Scene Representation ◽

3D Scene

Download Full-text

Passthrough+

Proceedings of the ACM on Computer Graphics and Interactive Techniques ◽

10.1145/3384540 ◽

2020 ◽

Vol 3 (1) ◽

pp. 1-17

Author(s):

Gaurav Chaurasia ◽

Arthur Nieuwoudt ◽

Alexandru-Eugen Ichim ◽

Richard Szeliski ◽

Alexander Sorkine-Hornung

Keyword(s):

Real Time ◽

Video Encoding ◽

View Synthesis ◽

Performance Constraints ◽

Trade Offs ◽

Stereoscopic View ◽

Target User ◽

Spatio Temporal ◽

And Performance ◽

3D Scene

We present an end-to-end system for real-time environment capture, 3D reconstruction, and stereoscopic view synthesis on a mobile VR headset. Our solution allows the user to use the cameras on their VR headset as their eyes to see and interact with the real world while still wearing their headset, a feature often referred to as Passthrough. The central challenge when building such a system is the choice and implementation of algorithms under the strict compute, power, and performance constraints imposed by the target user experience and mobile platform. A key contribution of this paper is a complete description of a corresponding system that performs temporally stable passthrough rendering at 72 Hz with only 200 mW power consumption on a mobile Snapdragon 835 platform. Our algorithmic contributions for enabling this performance include the computation of a coarse 3D scene proxy on the embedded video encoding hardware, followed by a depth densification and filtering step, and finally stereoscopic texturing and spatio-temporal up-sampling. We provide a detailed discussion and evaluation of the challenges we encountered, as well as algorithm and performance trade-offs in terms of compute and resulting passthrough quality.;AB@The described system is available to users as the Passthrough+ feature on Oculus Quest. We believe that by publishing the underlying system and methods, we provide valuable insights to the community on how to design and implement real-time environment sensing and rendering on heavily resource constrained hardware.

Download Full-text

Scene Wireframes Sketching for Drones

Proceedings ◽

10.3390/proceedings2181193 ◽

2018 ◽

Vol 2 (18) ◽

pp. 1193

Author(s):

Roi Santos ◽

Xose Pardo ◽

Xose Fdez-Vidal

Keyword(s):

State Of The Art ◽

Point Clouds ◽

Outdoor Environment ◽

Point Matching ◽

3D Point Clouds ◽

Complementary Approach ◽

Straight Line ◽

Scene Representation ◽

Feature Point Matching ◽

3D Scene

The increasing use of autonomous UAVs inside buildings and around human-made structures demands new accurate and comprehensive representation of their operation environments. Most of the 3D scene abstraction methods use invariant feature point matching, nevertheless some sparse 3D point clouds do not concisely represent the structure of the environment. Likewise, line clouds constructed by short and redundant segments with inaccurate directions limit the understanding of scenes as those that include environments with poor texture, or whose texture resembles a repetitive pattern. The presented approach is based on observation and representation models using the straight line segments, whose resemble the limits of an urban indoor or outdoor environment. The goal of the work is to get a full method based on the matching of lines that provides a complementary approach to state-of-the-art methods when facing 3D scene representation of poor texture environments for future autonomous UAV.

Download Full-text