scholarly journals Neural Light Transport for Relighting and View Synthesis

2021 ◽  
Vol 40 (1) ◽  
pp. 1-17
Author(s):  
Xiuming Zhang ◽  
Sean Fanello ◽  
Yun-Ta Tsai ◽  
Tiancheng Sun ◽  
Tianfan Xue ◽  
...  
Keyword(s):  
Neural Representation ◽  
Light Transport ◽  
View Synthesis ◽  
Unified Framework ◽  
Subsurface Scattering ◽  
Lighting Condition ◽  
Qualitative And Quantitative ◽  
Lighting Conditions ◽  
Complete Knowledge ◽  
Physically Based

The light transport (LT) of a scene describes how it appears under different lighting conditions from different viewing directions, and complete knowledge of a scene’s LT enables the synthesis of novel views under arbitrary lighting. In this article, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach for learning a neural representation of the LT that is embedded in a texture atlas of known but possibly rough geometry. We model all non-diffuse and global LT as residuals added to a physically based diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination (such as diffuse interreflection), while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse observations. Qualitative and quantitative experiments demonstrate that our Neural Light Transport (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without requiring separate treatments for both problems that prior work requires. The code and data are available at http://nlt.csail.mit.edu.

scholarly journals Lift: An Educational Interactive Stochastic Ray Tracing Framework with AI-Accelerated Denoiser

2021 ◽  
Author(s):  
Gonçalo Soares ◽  
João Madeiras Pereira
Keyword(s):  
Computer Graphics ◽  
Real Time ◽  
Ray Tracing ◽  
Path Length ◽  
Image Resolution ◽  
Light Transport ◽  
Lighting Conditions ◽  
Educational Framework ◽  
Physically Based ◽  
Physically Based Rendering

Real-time physically based rendering has long been looked at as the holy grail in Computer Graphics. With the introduction of Nvidia RTX-enabled GPUs family, light transport simulations under real-time constraint started to look like a reality. This paper presents Lift, an educational framework written in C++ that explores the RTX hardware pipeline by using the low-level Vulkan API and its Ray Tracing extension, recently made available by Khronos Group. Furthermore, to accomplish low variance rendered images, we integrated the AI-based denoiser available from the Nvidia ́s OptiX framework. Lift’s development arose primarily in the context of the graduate 3D Programming course taught at Instituto Superior Técnico and Master Theses focused on Real-Time Ray Trac- ing and provides the foundations for laboratory assignments and projects development. The platform aims to make easier students to learn and to develop, by programming the shaders of the RT pipeline, their physically-based ren- dering approaches and to compare them with the built-in progressive unidirectional and bidirectional path tracers. The GUI allows a user to specify camera settings and navigation speed, to select the input scene as well as the rendering method, to define the number of samples per pixel and the path length as well as to denoise the generated image either every frame or just the final frame. Statistics related with the timings, image resolution and total number of accumulated samples are provided too. Such platform will teach that nowadays physically-accurate images can be rendered in real-time under different lighting conditions and how well a denoiser can reconstruct images rendered with just one sample per pixel.

scholarly journals NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting

2021 ◽  
Author(s):  
TIANCHENG CHEN
Keyword(s):  
State Of The Art ◽  
Light Transport ◽  
View Synthesis ◽  
Computer Algorithms ◽  
3D Space ◽  
Lighting Conditions ◽  
Camera View ◽  
The Face ◽  
Environmental Lighting ◽  
Synthetic Models

Human portraits exhibit various appearances when observed from different views under different lighting conditions. We can easily imagine how the face will look like in another setup, but computer algorithms still fail on this problem given limited observations. To this end, we present a system for portrait view synthesis and relighting: given multiple portraits, we use a neural network to predict the light-transport field in 3D space, and from the predicted Neural Light-transport Field (NeLF)produce a portrait from a new camera view under a new environmental lighting. Our system is trained on a large number of synthetic models, and can generalize to different synthetic and real portraits under various lighting conditions. Our method achieves simultaneous view synthesis and relighting given multi-view portraits as the input, and achieves state-of-the-art results.

scholarly journals Active Recurrence of Lighting Condition for Fine-Grained Change Detection

2018 ◽  
Author(s):  
Qian Zhang ◽  
Wei Feng ◽  
Liang Wan ◽  
Fei-Peng Tian ◽  
Ping Tan
Keyword(s):  
Change Detection ◽  
Light Source ◽  
Real World ◽  
Reference Image ◽  
Light Sources ◽  
Lighting Condition ◽  
Fine Grained ◽  
Lighting Conditions ◽  
Point Light ◽  
Change Monitoring

This paper addresses active lighting recurrence (ALR), a new problem that actively relocalizes a light source to physically reproduce the lighting condition for a same scene from single reference image. ALR is of great importance for fine-grained visual monitoring and change detection, because some phenomena or minute changes can only be clearly observed under particular lighting conditions. Hence, effective ALR should be able to online navigate a light source toward the target pose, which is challenging due to the complexity and diversity of real-world lighting \& imaging processes. We propose to use the simple parallel lighting as an analogy model and based on Lambertian law to compose an instant navigation ball for this purpose. We theoretically prove the feasibility of this ALR strategy for realistic near point light sources and its invariance to the ambiguity of normal \& lighting decomposition. Extensive quantitative experiments and challenging real-world tasks on fine-grained change monitoring of cultural heritages verify the effectiveness of our approach. We also validate its generality to non-Lambertian scenes. 

scholarly journals Effect of Indoors Artificial Lighting Conditions on Computer-Based Learning Performance

2020 ◽  
Vol 17 (7) ◽  
pp. 2537
Author(s):  
Rui Zhang ◽  
Yalong Yang ◽  
Qiansheng Fang ◽  
Yufu Liu ◽  
Xulai Zhu ◽  
...  
Keyword(s):  
Human Performance ◽  
Learning Performance ◽  
Performance Measurements ◽  
Lighting Condition ◽  
Artificial Lighting ◽  
Computer Based Learning ◽  
Lighting Conditions ◽  
High Illumination ◽  
Key Factor ◽  
Computer Based

Lighting condition is essential to human performance. With the widespread use of computer-based learning, the performance measurements become difficult, and the effects of artificial lighting conditions towards the new learning forms are not investigated extensively. The current study conducts a subject-within experiment with a 45-min-long online learning along with electroencephalogram (EEG)-based measurements, and a post-interview under five lighting setups respectively (300 lx, 3000 K; 300 lx, 4000 K; 300 lx, 6500 K; 500 lx, 4000 K; 1000 lx, 4000 K). Attention is chosen as the key factor to represent the learning performance. The results show that the attention of people aged in the 20s is not affected by the experimental lighting conditions. The results also demonstrate that people in high illumination at 1000 lx are more inclined to sustain attention despite the discomfort and dissatisfaction. Taking the EEG-based attention measurements and post-interview answers into consideration, lighting conditions at 300 lx, 4000 K are the recommended set points for university architectures among the investigated conditions, providing a practical basis when adjusting the lighting standard for its advantage in energy saving.

scholarly journals The Role of Subsurface Scattering in Glossiness Perception

2021 ◽  
Vol 18 (3) ◽  
pp. 1-26
Author(s):  
Davit Gigilashvili ◽  
Weiqi Shi ◽  
Zeyu Wang ◽  
Marius Pedersen ◽  
Jon Yngve Hardeberg ◽  
...  
Keyword(s):  
Visual Cues ◽  
Light Transmission ◽  
Light Transport ◽  
Future Research ◽  
Subsurface Scattering ◽  
Visual Appearance ◽  
Material Appearance ◽  
Related Quality ◽  
The Difference ◽  
Subsurface Light

This study investigates the potential impact of subsurface light transport on gloss perception for the purposes of broadening our understanding of visual appearance in computer graphics applications. Gloss is an important attribute for characterizing material appearance. We hypothesize that subsurface scattering of light impacts the glossiness perception. However, gloss has been traditionally studied as a surface-related quality and the findings in the state-of-the-art are usually based on fully opaque materials, although the visual cues of glossiness can be impacted by light transmission as well. To address this gap and to test our hypothesis, we conducted psychophysical experiments and found that subjects are able to tell the difference in terms of gloss between stimuli that differ in subsurface light transport but have identical surface qualities and object shape. This gives us a clear indication that subsurface light transport contributes to a glossy appearance. Furthermore, we conducted additional experiments and found that the contribution of subsurface scattering to gloss varies across different shapes and levels of surface roughness. We argue that future research on gloss should include transparent and translucent media and to extend the perceptual models currently limited to surface scattering to more general ones inclusive of subsurface light transport.

scholarly journals Robust Weed Recognition Through Color Based Image Segmentation and Convolution Neural Network Based Classification

2019 ◽  
Author(s):  
M. Nazmuzzaman Khan ◽  
Sohel Anwar
Keyword(s):  
Neural Network ◽  
Input Image ◽  
Convolution Neural Network ◽  
Neural Net ◽  
Weed Species ◽  
Illumination Variation ◽  
Weed Detection ◽  
Lighting Condition ◽  
Lighting Conditions ◽  
Quality Image

Abstract Current image classification techniques for weed detection (classic vision techniques and deep-neural net) provide encouraging results under controlled environment. But most of the algorithms are not robust enough for real-world application. Different lighting conditions and shadows directly impact vegetation color. Varying outdoor lighting conditions create different colors, noise levels, contrast and brightness. High component of illumination causes sensor (industrial camera) saturation. As a result, threshold-based classification algorithms usually fail. To overcome this shortfall, we used visible spectral-index based segmentation to segment the weeds from background. Mean, variance, kurtosis, and skewness are calculated for each input image and image quality (good or bad) is determined. Bad quality image is converted to good-quality image using contrast limited adaptive histogram equalization (CLAHE) before segmentation. A convolution neural network (CNN) based classifier is then trained to classify three different types of weed (Ragweed, Pigweed and Cocklebur) common in a corn field. The main objective of this work is to construct a robust classifier, capable of classifying between three weed species in the presence of occlusion, noise, illumination variation, and motion blurring. Proposed histogram statistics-based image enhancement process solved weed mis-segmentation under extreme lighting condition. CNN based classifier shows accurate, robust classification under low-to-mid level motion blurring and various levels of noise.

scholarly journals Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement

2021 ◽  
Author(s):  
Huiwen Luo ◽  
Koki Nagano ◽  
Han-Wei Kung ◽  
Mclean Goldwhite
Keyword(s):  
Cutting Edge ◽  
Superior Performance ◽  
3D Face Reconstruction ◽  
High Quality ◽  
Ground Truth Data ◽  
Lighting Condition ◽  
3D Face ◽  
Lighting Conditions ◽  
Reconstruction Methods ◽  
Non Linear

We introduce a highly robust GAN-based framework for digitizing a normalized 3D avatar of a person from a single unconstrained photo. While the input image can be of a smiling person or taken in extreme lighting conditions, our method can reliably produce a high-quality textured model of a person's face in neutral expression and skin textures under diffuse lighting condition. Cutting-edge 3D face reconstruction methods use non-linear morphable face models combined with GAN-based decoders to capture the likeness and details of a person but fail to produce neutral head models with unshaded albedo textures which is critical for creating relightable and animation-friendly avatars for integration in virtual environments. The key challenges for existing methods to work is the lack of training and ground truth data containing normalized 3D faces. We propose a two-stage approach to address this problem. First, we adopt a highly robust normalized 3D face generator by embedding a non-linear morphable face model into a StyleGAN2 network. This allows us to generate detailed but normalized facial assets. This inference is then followed by a perceptual refinement step that uses the generated assets as regularization to cope with the limited available training samples of normalized faces. We further introduce a Normalized Face Dataset, which consists of a combination photogrammetry scans, carefully selected photographs, and generated fake people with neutral expressions in diffuse lighting conditions. While our prepared dataset contains two orders of magnitude less subjects than cutting edge GAN-based 3D facial reconstruction methods, we show that it is possible to produce high-quality normalized face models for very challenging unconstrained input images, and demonstrate superior performance to the current state-of-the-art.

scholarly journals Recognition of vehicle make and model in low light conditions

2020 ◽  
Vol 9 (2) ◽  
Author(s):  
Aymen Fadhil Abbas ◽  
Usman Ullah Sheikh ◽  
Mohd Norzali Haji Mohd
Keyword(s):  
Image Quality ◽  
Image Enhancement ◽  
Operating Conditions ◽  
Light Conditions ◽  
Low Light ◽  
Lighting Condition ◽  
Front View ◽  
Recognition Model ◽  
Lighting Conditions ◽  
Model Recognition

This paper presents a method for vehicle make and model recognition (MMR) in low lighting conditions. While many MMR methods exist in the literature, these methods are designed to be used only in perfect operating conditions. The various camera configuration, lighting condition, and viewpoints cause variations in image quality.  In the presented method, the vehicle is first detected, image enhancement is then carried out on the detected front view of the vehicle, followed by features extraction and classification. The performance is then examined on a low-light dataset. The results show around 6% improvement in the ability of MMR with the use of image enhancement over the same recognition model without image enhancement.

scholarly journals Influence of Image Quality and Light Consistency on the Performance of Convolutional Neural Networks for Weed Mapping

2021 ◽  
Vol 13 (11) ◽  
pp. 2140
Author(s):  
Chengsong Hu ◽  
Bishwa B. Sapkota ◽  
J. Alex Thomasson ◽  
Muthukumar V. Bagavathiannan
Keyword(s):  
Neural Networks ◽  
Image Quality ◽  
Convolutional Neural Networks ◽  
Motion Blur ◽  
Lighting Condition ◽  
Weed Mapping ◽  
Lighting Conditions ◽  
Performance Reduction ◽  
Gaussian Blur ◽  
The Impact

Recent computer vision techniques based on convolutional neural networks (CNNs) are considered state-of-the-art tools in weed mapping. However, their performance has been shown to be sensitive to image quality degradation. Variation in lighting conditions adds another level of complexity to weed mapping. We focus on determining the influence of image quality and light consistency on the performance of CNNs in weed mapping by simulating the image formation pipeline. Faster Region-based CNN (R-CNN) and Mask R-CNN were used as CNN examples for object detection and instance segmentation, respectively, while semantic segmentation was represented by Deeplab-v3. The degradations simulated in this study included resolution reduction, overexposure, Gaussian blur, motion blur, and noise. The results showed that the CNN performance was most impacted by resolution, regardless of plant size. When the training and testing images had the same quality, Faster R-CNN and Mask R-CNN were moderately tolerant to low levels of overexposure, Gaussian blur, motion blur, and noise. Deeplab-v3, on the other hand, tolerated overexposure, motion blur, and noise at all tested levels. In most cases, quality inconsistency between the training and testing images reduced CNN performance. However, CNN models trained on low-quality images were more tolerant against quality inconsistency than those trained by high-quality images. Light inconsistency also reduced CNN performance. Increasing the diversity of lighting conditions in the training images may alleviate the performance reduction but does not provide the same benefit from the number increase of images with the same lighting condition. These results provide insights into the impact of image quality and light consistency on CNN performance. The quality threshold established in this study can be used to guide the selection of camera parameters in future weed mapping applications.

Sign in / Sign up

Export Citation Format

Share Document