scholarly journals 3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques

2021 ◽  
Vol 13 (18) ◽  
pp. 3647
Author(s):  
Ghizlane Karara ◽  
Rafika Hajji ◽  
Florent Poux
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Point Cloud ◽  
Point Clouds ◽  
Research Field ◽  
Virtual Camera ◽  
3D Point Clouds ◽  
Active Research ◽  
2D Images ◽  
Instance Segmentation

Semantic augmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionised image segmentation and classification, its impact on point cloud is an active research field. In this paper, we propose an instance segmentation and augmentation of 3D point clouds using deep learning architectures. We show the potential of an indirect approach using 2D images and a Mask R-CNN (Region-Based Convolution Neural Network). Our method consists of four core steps. We first project the point cloud onto panoramic 2D images using three types of projections: spherical, cylindrical, and cubic. Next, we homogenise the resulting images to correct the artefacts and the empty pixels to be comparable to images available in common training libraries. These images are then used as input to the Mask R-CNN neural network, designed for 2D instance segmentation. Finally, the obtained predictions are reprojected to the point cloud to obtain the segmentation results. We link the results to a context-aware neural network to augment the semantics. Several tests were performed on different datasets to test the adequacy of the method and its potential for generalisation. The developed algorithm uses only the attributes X, Y, Z, and a projection centre (virtual camera) position as inputs.

Real-Time 3D object detection using improved convolutional neural network based on image-driven point cloud

2021 ◽  
Vol 14 ◽  
Author(s):  
Zhiyong Gao ◽  
Jianhong Xiang
Keyword(s):  
Neural Network ◽  
Object Detection ◽  
Convolutional Neural Network ◽  
Real Time ◽  
Point Cloud ◽  
Point Clouds ◽  
3D Point Cloud ◽  
3D Object ◽  
3D Object Detection ◽  
Instance Segmentation

Background: While detecting the object directly from the 3D point cloud, the natural 3D patterns and invariance of 3D data are often obscure. Objective: In this work, we aimed at studying the 3D object detection from discrete, disordered and sparse 3D point clouds. Methods: The CNN is composed of the frustum sequence module, 3D instance segmentation module S-NET, 3D point cloud transformation module T-NET, and 3D boundary box estimation module E-NET. The search space of the object is determined by the frustum sequence module. The instance segmentation of the point cloud is performed by the 3D instance segmentation module. The 3D coordinates of the object are confirmed by the transformation module and the 3D bounding box estimation module. Results: Evaluated on KITTI benchmark dataset, our method outperforms the state of the art by remarkable margins while having real-time capability. Conclusion: We achieve real-time 3D object detection by proposing an improved convolutional neural network (CNN) based on image-driven point clouds.

scholarly journals Food Volume Estimation Based on Deep Learning View Synthesis from a Single Depth Map

Nutrients ◽  
2018 ◽  
Vol 10 (12) ◽  
pp. 2005 ◽  
Author(s):  
Frank Lo ◽  
Yingnan Sun ◽  
Jianing Qiu ◽  
Benny Lo
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Point Cloud ◽  
Volume Estimation ◽  
Assessment System ◽  
View Synthesis ◽  
Depth Image ◽  
3D Point Cloud ◽  
Viewing Angle ◽  
3D Point Clouds

An objective dietary assessment system can help users to understand their dietary behavior and enable targeted interventions to address underlying health problems. To accurately quantify dietary intake, measurement of the portion size or food volume is required. For volume estimation, previous research studies mostly focused on using model-based or stereo-based approaches which rely on manual intervention or require users to capture multiple frames from different viewing angles which can be tedious. In this paper, a view synthesis approach based on deep learning is proposed to reconstruct 3D point clouds of food items and estimate the volume from a single depth image. A distinct neural network is designed to use a depth image from one viewing angle to predict another depth image captured from the corresponding opposite viewing angle. The whole 3D point cloud map is then reconstructed by fusing the initial data points with the synthesized points of the object items through the proposed point cloud completion and Iterative Closest Point (ICP) algorithms. Furthermore, a database with depth images of food object items captured from different viewing angles is constructed with image rendering and used to validate the proposed neural network. The methodology is then evaluated by comparing the volume estimated by the synthesized 3D point cloud with the ground truth volume of the object items.

scholarly journals A Green Prospective for Learned Post-processing in Sparse-view Tomographic Reconstruction

2021 ◽  
Author(s):  
Elena Morotti ◽  
Davide Evangelista ◽  
Elena Loli Piccolomini
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Tomographic Reconstruction ◽  
Research Field ◽  
Filtered Backprojection ◽  
Post Processing ◽  
Training Set ◽  
Imaging Reconstruction ◽  
Active Research

Deep Learning is developing interesting tools which are of great interest for inverse imaging applications. In this work, we consider a medical imaging reconstruction task from subsampled measurements, which is an active research field where Convolutional Neural Networks have already revealed their great potential. However, the commonly used architectures are very deep and, hence, prone to overfitting and unfeasible for clinical usages. Inspired by the ideas of the green-AI literature, we here propose a shallow neural network to perform an efficient Learned Post-Processing on images roughly reconstructed by the filtered backprojection algorithm. The results obtained on images from the training set and on unseen images, using both the non-expensive network and the widely used very deep ResUNet show that the proposed network computes images of comparable or higher quality in about one fourth of time.

scholarly journals Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage

2020 ◽  
Vol 12 (6) ◽  
pp. 1005 ◽  
Author(s):  
Roberto Pierdicca ◽  
Marina Paolanti ◽  
Francesca Matrone ◽  
Massimo Martini ◽  
Christian Morbidoni ◽  
...  
Keyword(s):  
Deep Learning ◽  
Cultural Heritage ◽  
Point Cloud ◽  
Semantic Segmentation ◽  
Point Clouds ◽  
Information Modeling ◽  
Dynamic Graph ◽  
Historical Building ◽  
Architectural Elements ◽  
3D Point Clouds

In the Digital Cultural Heritage (DCH) domain, the semantic segmentation of 3D Point Clouds with Deep Learning (DL) techniques can help to recognize historical architectural elements, at an adequate level of detail, and thus speed up the process of modeling of historical buildings for developing BIM models from survey data, referred to as HBIM (Historical Building Information Modeling). In this paper, we propose a DL framework for Point Cloud segmentation, which employs an improved DGCNN (Dynamic Graph Convolutional Neural Network) by adding meaningful features such as normal and colour. The approach has been applied to a newly collected DCH Dataset which is publicy available: ArCH (Architectural Cultural Heritage) Dataset. This dataset comprises 11 labeled points clouds, derived from the union of several single scans or from the integration of the latter with photogrammetric surveys. The involved scenes are both indoor and outdoor, with churches, chapels, cloisters, porticoes and loggias covered by a variety of vaults and beared by many different types of columns. They belong to different historical periods and different styles, in order to make the dataset the least possible uniform and homogeneous (in the repetition of the architectural elements) and the results as general as possible. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the proposed approach.

scholarly journals JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds

2020 ◽  
Vol 34 (07) ◽  
pp. 12951-12958 ◽  
Author(s):  
Lin Zhao ◽  
Wenbing Tao
Keyword(s):  
Point Cloud ◽  
Large Scale ◽  
Feature Fusion ◽  
Mean Shift ◽  
Semantic Segmentation ◽  
Point Clouds ◽  
Semantic Features ◽  
Backbone Network ◽  
3D Point Clouds ◽  
Instance Segmentation

In this paper, we propose a novel joint instance and semantic segmentation approach, which is called JSNet, in order to address the instance and semantic segmentation of 3D point clouds simultaneously. Firstly, we build an effective backbone network to extract robust features from the raw point clouds. Secondly, to obtain more discriminative features, a point cloud feature fusion module is proposed to fuse the different layer features of the backbone network. Furthermore, a joint instance semantic segmentation module is developed to transform semantic features into instance embedding space, and then the transformed features are further fused with instance features to facilitate instance segmentation. Meanwhile, this module also aggregates instance features into semantic feature space to promote semantic segmentation. Finally, the instance predictions are generated by applying a simple mean-shift clustering on instance embeddings. As a result, we evaluate the proposed JSNet on a large-scale 3D indoor point cloud dataset S3DIS and a part dataset ShapeNet, and compare it with existing approaches. Experimental results demonstrate our approach outperforms the state-of-the-art method in 3D instance segmentation with a significant improvement in 3D semantic prediction and our method is also beneficial for part segmentation. The source code for this work is available at https://github.com/dlinzhao/JSNet.

scholarly journals Hole Repairing Algorithm for 3D Point Cloud Model of Symmetrical Objects Grasped by the Manipulator

Sensors ◽  
2021 ◽  
Vol 21 (22) ◽  
pp. 7558
Author(s):  
Linyan Cui ◽  
Guolong Zhang ◽  
Jinshen Wang
Keyword(s):  
Point Cloud ◽  
Cloud Model ◽  
Engineering Application ◽  
Point Clouds ◽  
Real Point ◽  
3D Point Cloud ◽  
Virtual Camera ◽  
3D Point Clouds ◽  
Cloud Models ◽  
Point Cloud Model

For the engineering application of manipulator grasping objects, mechanical arm occlusion and limited imaging angle produce various holes in the reconstructed 3D point clouds of objects. Acquiring a complete point cloud model of the grasped object plays a very important role in the subsequent task planning of the manipulator. This paper proposes a method with which to automatically detect and repair the holes in the 3D point cloud model of symmetrical objects grasped by the manipulator. With the established virtual camera coordinate system and boundary detection, repair and classification of holes, the closed boundaries for the nested holes were detected and classified into two kinds, which correspond to the mechanical claw holes caused by mechanical arm occlusion and the missing surface produced by limited imaging angle. These two kinds of holes were repaired based on surface reconstruction and object symmetry. Experiments on simulated and real point cloud models demonstrate that our approach outperforms the other state-of-the-art 3D point cloud hole repair algorithms.

scholarly journals Structure-Aware Convolution for 3D Point Cloud Classification and Segmentation

2020 ◽  
Vol 12 (4) ◽  
pp. 634 ◽  
Author(s):  
Lei Wang ◽  
Yuxuan Liu ◽  
Shenman Zhang ◽  
Jixing Yan ◽  
Pengjie Tao
Keyword(s):  
Deep Learning ◽  
Template Matching ◽  
Point Cloud ◽  
Structure Learning ◽  
Feature Learning ◽  
Point Clouds ◽  
Learning Networks ◽  
Geometric Structures ◽  
Learning Capability ◽  
3D Point Clouds

Semantic feature learning on 3D point clouds is quite challenging because of their irregular and unordered data structure. In this paper, we propose a novel structure-aware convolution (SAC) to generalize deep learning on regular grids to irregular 3D point clouds. Similar to the template-matching process of convolution on 2D images, the key of our SAC is to match the point clouds’ neighborhoods with a series of 3D kernels, where each kernel can be regarded as a “geometric template” formed by a set of learnable 3D points. Thus, the interested geometric structures of the input point clouds can be activated by the corresponding kernels. To verify the effectiveness of the proposed SAC, we embedded it into three recently developed point cloud deep learning networks (PointNet, PointNet++, and KCNet) as a lightweight module, and evaluated its performance on both classification and segmentation tasks. Experimental results show that, benefiting from the geometric structure learning capability of our SAC, all these back-end networks achieved better classification and segmentation performance (e.g., +2.77% mean accuracy for classification and +4.99% mean intersection over union (IoU) for segmentation) with few additional parameters. Furthermore, results also demonstrate that the proposed SAC is helpful in improving the robustness of networks with the constraints of geometric structures.

scholarly journals Review: Deep Learning on 3D Point Clouds

2020 ◽  
Vol 12 (11) ◽  
pp. 1729 ◽  
Author(s):  
Saifullahi Aminu Bello ◽  
Shangshu Yu ◽  
Cheng Wang ◽  
Jibril Muhmmad Adam ◽  
Jonathan Li
Keyword(s):  
Deep Learning ◽  
Point Cloud ◽  
General Structure ◽  
Point Clouds ◽  
Autonomous Driving ◽  
Point Cloud Data ◽  
3D Vision ◽  
Cloud Data ◽  
3D Point Clouds ◽  
Learning Techniques

A point cloud is a set of points defined in a 3D metric space. Point clouds have become one of the most significant data formats for 3D representation and are gaining increased popularity as a result of the increased availability of acquisition devices, as well as seeing increased application in areas such as robotics, autonomous driving, and augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision and is becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, the point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes the use of deep learning for its direct processing very challenging. This paper contains a review of the recent state-of-the-art deep learning techniques, mainly focusing on raw point cloud data. The initial work on deep learning directly with raw point cloud data did not model local regions; therefore, subsequent approaches model local regions through sampling and grouping. More recently, several approaches have been proposed that not only model the local regions but also explore the correlation between points in the local regions. From the survey, we conclude that approaches that model local regions and take into account the correlation between points in the local regions perform better. Contrary to existing reviews, this paper provides a general structure for learning with raw point clouds, and various methods were compared based on the general structure. This work also introduces the popular 3D point cloud benchmark datasets and discusses the application of deep learning in popular 3D vision tasks, including classification, segmentation, and detection.

scholarly journals Comparison of 3D interest point detectors and descriptors for point cloud fusion

2014 ◽  
Vol II-3 ◽  
pp. 57-64 ◽  
Author(s):  
R. Hänsch ◽  
T. Weber ◽  
O. Hellwich
Keyword(s):  
Point Cloud ◽  
Point Clouds ◽  
Specific Method ◽  
Scanner Data ◽  
3D Scanner ◽  
Strong Reduction ◽  
Interest Point Detectors ◽  
3D Point Clouds ◽  
3D Data ◽  
2D Images

The extraction and description of keypoints as salient image parts has a long tradition within processing and analysis of 2D images. Nowadays, 3D data gains more and more importance. This paper discusses the benefits and limitations of keypoints for the task of fusing multiple 3D point clouds. For this goal, several combinations of 3D keypoint detectors and descriptors are tested. The experiments are based on 3D scenes with varying properties, including 3D scanner data as well as Kinect point clouds. The obtained results indicate that the specific method to extract and describe keypoints in 3D data has to be carefully chosen. In many cases the accuracy suffers from a too strong reduction of the available points to keypoints.

Sign in / Sign up

Export Citation Format

Share Document