scholarly journals Features to Text: A Comprehensive Survey of Deep Learning on Semantic Segmentation and Image Captioning

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-19
Author(s):  
Ariyo Oluwasammi ◽  
Muhammad Umar Aftab ◽  
Zhiguang Qin ◽  
Son Tung Ngo ◽  
Thang Van Doan ◽  
...  
Keyword(s):  
Deep Learning ◽  
Evaluation Criteria ◽  
Semantic Segmentation ◽  
Extraction Methods ◽  
Object Identification ◽  
Image Captioning ◽  
Convolutional Network ◽  
Segmentation Analysis ◽  
Comprehensive Survey ◽  
High Level

With the emergence of deep learning, computer vision has witnessed extensive advancement and has seen immense applications in multiple domains. Specifically, image captioning has become an attractive focal direction for most machine learning experts, which includes the prerequisite of object identification, location, and semantic understanding. In this paper, semantic segmentation and image captioning are comprehensively investigated based on traditional and state-of-the-art methodologies. In this survey, we deliberate on the use of deep learning techniques on the segmentation analysis of both 2D and 3D images using a fully convolutional network and other high-level hierarchical feature extraction methods. First, each domain’s preliminaries and concept are described, and then semantic segmentation is discussed alongside its relevant features, available datasets, and evaluation criteria. Also, the semantic information capturing of objects and their attributes is presented in relation to their annotation generation. Finally, analysis of the existing methods, their contributions, and relevance are highlighted, informing the importance of these methods and illuminating a possible research continuation for the application of semantic image segmentation and image captioning approaches.

scholarly journals A Survey on Deep Learning Based Approaches for Scene Understanding in Autonomous Driving

Electronics ◽  
2021 ◽  
Vol 10 (4) ◽  
pp. 471
Author(s):  
Zhiyang Guo ◽  
Yingping Huang ◽  
Xing Hu ◽  
Hongjian Wei ◽  
Baigan Zhao
Keyword(s):  
Deep Learning ◽  
Evaluation Criteria ◽  
Scene Understanding ◽  
Semantic Segmentation ◽  
Autonomous Driving ◽  
Performance Comparison ◽  
Advantages And Disadvantages ◽  
Future Challenges ◽  
Benchmark Datasets ◽  
Comprehensive Survey

As a prerequisite for autonomous driving, scene understanding has attracted extensive research. With the rise of the convolutional neural network (CNN)-based deep learning technique, research on scene understanding has achieved significant progress. This paper aims to provide a comprehensive survey of deep learning-based approaches for scene understanding in autonomous driving. We categorize these works into four work streams, including object detection, full scene semantic segmentation, instance segmentation, and lane line segmentation. We discuss and analyze these works according to their characteristics, advantages and disadvantages, and basic frameworks. We also summarize the benchmark datasets and evaluation criteria used in the research community and make a performance comparison of some of the latest works. Lastly, we summarize the review work and provide a discussion on the future challenges of the research domain.

scholarly journals Self-Attention in Reconstruction Bias U-Net for Semantic Segmentation of Building Rooftops in Optical Remote Sensing Images

2021 ◽  
Vol 13 (13) ◽  
pp. 2524
Author(s):  
Ziyi Chen ◽  
Dilong Li ◽  
Wentao Fan ◽  
Haiyan Guan ◽  
Cheng Wang ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Deep Learning ◽  
Semantic Segmentation ◽  
Extraction Methods ◽  
The Self ◽  
Optical Remote Sensing ◽  
Building Extraction ◽  
Learning Models ◽  
Remote Sensing Images ◽  
Segmentation Methods

Deep learning models have brought great breakthroughs in building extraction from high-resolution optical remote-sensing images. Among recent research, the self-attention module has called up a storm in many fields, including building extraction. However, most current deep learning models loading with the self-attention module still lose sight of the reconstruction bias’s effectiveness. Through tipping the balance between the abilities of encoding and decoding, i.e., making the decoding network be much more complex than the encoding network, the semantic segmentation ability will be reinforced. To remedy the research weakness in combing self-attention and reconstruction-bias modules for building extraction, this paper presents a U-Net architecture that combines self-attention and reconstruction-bias modules. In the encoding part, a self-attention module is added to learn the attention weights of the inputs. Through the self-attention module, the network will pay more attention to positions where there may be salient regions. In the decoding part, multiple large convolutional up-sampling operations are used for increasing the reconstruction ability. We test our model on two open available datasets: the WHU and Massachusetts Building datasets. We achieve IoU scores of 89.39% and 73.49% for the WHU and Massachusetts Building datasets, respectively. Compared with several recently famous semantic segmentation methods and representative building extraction methods, our method’s results are satisfactory.

Implementation of Deep Learning Neural Network for Retinal Images

2020 ◽  
pp. 77-95
Author(s):  
R. Murugan
Keyword(s):  
Deep Learning ◽  
Semantic Segmentation ◽  
Neural System ◽  
Neural Systems ◽  
Retinal Images ◽  
Manual Segmentation ◽  
Convolutional Network ◽  
The World ◽  
Deep Learning Neural Network ◽  
Wide Assortment

The retinal parts segmentation has been recognized as a key component in both ophthalmological and cardiovascular sickness analysis. The parts of retinal pictures, vessels, optic disc, and macula segmentations, will add to the indicative outcome. In any case, the manual segmentation of retinal parts is tedious and dreary work, and it additionally requires proficient aptitudes. This chapter proposes a supervised method to segment blood vessel utilizing deep learning methods. All the more explicitly, the proposed part has connected the completely convolutional network, which is normally used to perform semantic segmentation undertaking with exchange learning. The convolutional neural system has turned out to be an amazing asset for a few computer vision assignments. As of late, restorative picture investigation bunches over the world are rapidly entering this field and applying convolutional neural systems and other deep learning philosophies to a wide assortment of uses, and uncommon outcomes are rising constantly.

scholarly journals Transformer-Based Decoder Designs for Semantic Segmentation on Remotely Sensed Images

2021 ◽  
Vol 13 (24) ◽  
pp. 5100
Author(s):  
Teerapong Panboonyuen ◽  
Kulsawasd Jitkajornwanich ◽  
Siam Lawawirojwong ◽  
Panu Srestasathiern ◽  
Peerapon Vateekul
Keyword(s):  
Image Processing ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Language Processing ◽  
State Of The Art ◽  
Semantic Segmentation ◽  
Landsat 8 ◽  
Convolutional Network ◽  
Image Labeling ◽  
Feature Pyramid

Transformers have demonstrated remarkable accomplishments in several natural language processing (NLP) tasks as well as image processing tasks. Herein, we present a deep-learning (DL) model that is capable of improving the semantic segmentation network in two ways. First, utilizing the pre-training Swin Transformer (SwinTF) under Vision Transformer (ViT) as a backbone, the model weights downstream tasks by joining task layers upon the pretrained encoder. Secondly, decoder designs are applied to our DL network with three decoder designs, U-Net, pyramid scene parsing (PSP) network, and feature pyramid network (FPN), to perform pixel-level segmentation. The results are compared with other image labeling state of the art (SOTA) methods, such as global convolutional network (GCN) and ViT. Extensive experiments show that our Swin Transformer (SwinTF) with decoder designs reached a new state of the art on the Thailand Isan Landsat-8 corpus (89.8% F1 score), Thailand North Landsat-8 corpus (63.12% F1 score), and competitive results on ISPRS Vaihingen. Moreover, both our best-proposed methods (SwinTF-PSP and SwinTF-FPN) even outperformed SwinTF with supervised pre-training ViT on the ImageNet-1K in the Thailand, Landsat-8, and ISPRS Vaihingen corpora.

scholarly journals Learning to See the Hidden Part of the Vehicle in the Autopilot Scene

Electronics ◽  
2019 ◽  
Vol 8 (3) ◽  
pp. 331 ◽  
Author(s):  
Yifeng Xu ◽  
Huigang Wang ◽  
Xing Liu ◽  
Henry He ◽  
Qingyue Gu ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Deep Learning ◽  
Object Detection ◽  
Vision System ◽  
Image Inpainting ◽  
Semantic Segmentation ◽  
Fatal Crash ◽  
White Side ◽  
Original Size ◽  
High Level

Recent advances in deep learning have shown exciting promise in low-level artificial intelligence tasks such as image classification, speech recognition, object detection, and semantic segmentation, etc. Artificial intelligence has made an important contribution to autopilot, which is a complex high-level intelligence task. However, the real autopilot scene is quite complicated. The first accident of autopilot occurred in 2016. It resulted in a fatal crash where the white side of a vehicle appeared similar to a brightly lit sky. The root of the problem is that the autopilot vision system cannot identify the part of a vehicle when the part is similar to the background. A method called DIDA was first proposed based on the deep learning network to see the hidden part. DIDA cascades the following steps: object detection, scaling, image inpainting assuming a hidden part beside the car, object re-detection from inpainted image, zooming back to the original size, and setting an alarm region by comparing two detected regions. DIDA was tested in a similar scene and achieved exciting results. This method solves the aforementioned problem only by using optical signals. Additionally, the vehicle dataset captured in Xi’an, China can be used in subsequent research.

Weapon and Object Detection Using Mobile-Net SSD Model in Deep Neural Network

2021 ◽  
Vol 9 (8) ◽  
pp. 1573-1582
Author(s):  
S Gopi Naik
Keyword(s):  
Neural Network ◽  
Computer Vision ◽  
Deep Learning ◽  
Object Detection ◽  
Convolutional Neural Network ◽  
Optimization Algorithms ◽  
Object Identification ◽  
Integrated System ◽  
Raspberry Pi ◽  
High Level

Abstract: The plan is to establish an integrated system that can manage high-quality visual information and also detect weapons quickly and efficiently. It is obtained by integrating ARM-based computer vision and optimization algorithms with deep neural networks able to detect the presence of a threat. The whole system is connected to a Raspberry Pi module, which will capture live broadcasting and evaluate it using a deep convolutional neural network. Due to the intimate interaction between object identification and video and image analysis in real-time objects, By generating sophisticated ensembles that incorporate various low-level picture features with high-level information from object detection and scenario classifiers, their performance can quickly plateau. Deep learning models, which can learn semantic, high-level, deeper features, have been developed to overcome the issues that are present in optimization algorithms. It presents a review of deep learning based object detection frameworks that use Convolutional Neural Network layers for better understanding of object detection. The Mobile-Net SSD model behaves differently in network design, training methods, and optimization functions, among other things. The crime rate in suspicious areas has been reduced as a consequence of weapon detection. However, security is always a major concern in human life. The Raspberry Pi module, or computer vision, has been extensively used in the detection and monitoring of weapons. Due to the growing rate of human safety protection, privacy and the integration of live broadcasting systems which can detect and analyse images, suspicious areas are becoming indispensable in intelligence. This process uses a Mobile-Net SSD algorithm to achieve automatic weapons and object detection. Keywords: Computer Vision, Weapon and Object Detection, Raspberry Pi Camera, RTSP, SMTP, Mobile-Net SSD, CNN, Artificial Intelligence.

scholarly journals IMPROVING DEEP LEARNING BASED SEMANTIC SEGMENTATION WITH MULTI VIEW OUTLIER CORRECTION

2020 ◽  
Vol XLIII-B2-2020 ◽  
pp. 711-716
Author(s):  
T. Peters ◽  
C. Brenner ◽  
M. Song
Keyword(s):  
Deep Learning ◽  
Transfer Learning ◽  
Input Data ◽  
Semantic Segmentation ◽  
Object Point ◽  
Mobile Mapping ◽  
Convolutional Network ◽  
3D Object ◽  
The Mean ◽  
2D Images

Abstract. The goal of this paper is to use transfer learning for semi supervised semantic segmentation in 2D images: given a pretrained deep convolutional network (DCNN), our aim is to adapt it to a new camera-sensor system by enforcing predictions to be consistent for the same object in space. This is enabled by projecting 3D object points into multi-view 2D images. Since every 3D object point is usually mapped to a number of 2D images, each of which undergoes a pixelwise classification using the pretrained DCNN, we obtain a number of predictions (labels) for the same object point. This makes it possible to detect and correct outlier predictions. Ultimately, we retrain the DCNN on the corrected dataset in order to adapt the network to the new input data. We demonstrate the effectiveness of our approach on a mobile mapping dataset containing over 10’000 images and more than 1 billion 3D points. Moreover, we manually annotated a subset of the mobile mapping images and show that we were able to rise the mean intersection over union (mIoU) by approximately 10% with Deeplabv3+, using our approach.

scholarly journals Deep Residual Autoencoder with Multiscaling for Semantic Segmentation of Land-Use Images

2019 ◽  
Vol 11 (18) ◽  
pp. 2142 ◽  
Author(s):  
Lianfa Li
Keyword(s):  
Land Use ◽  
Deep Learning ◽  
Semantic Segmentation ◽  
Remotely Sensed ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Residual Learning ◽  
Fully Convolutional Networks ◽  
Remotely Sensed Images ◽  
Real World Datasets

Semantic segmentation is a fundamental means of extracting information from remotely sensed images at the pixel level. Deep learning has enabled considerable improvements in efficiency and accuracy of semantic segmentation of general images. Typical models range from benchmarks such as fully convolutional networks, U-Net, Micro-Net, and dilated residual networks to the more recently developed DeepLab 3+. However, many of these models were originally developed for segmentation of general or medical images and videos, and are not directly relevant to remotely sensed images. The studies of deep learning for semantic segmentation of remotely sensed images are limited. This paper presents a novel flexible autoencoder-based architecture of deep learning that makes extensive use of residual learning and multiscaling for robust semantic segmentation of remotely sensed land-use images. In this architecture, a deep residual autoencoder is generalized to a fully convolutional network in which residual connections are implemented within and between all encoding and decoding layers. Compared with the concatenated shortcuts in U-Net, these residual connections reduce the number of trainable parameters and improve the learning efficiency by enabling extensive backpropagation of errors. In addition, resizing or atrous spatial pyramid pooling (ASPP) can be leveraged to capture multiscale information from the input images to enhance the robustness to scale variations. The residual learning and multiscaling strategies improve the trained model’s generalizability, as demonstrated in the semantic segmentation of land-use types in two real-world datasets of remotely sensed images. Compared with U-Net, the proposed method improves the Jaccard index (JI) or the mean intersection over union (MIoU) by 4-11% in the training phase and by 3-9% in the validation and testing phases. With its flexible deep learning architecture, the proposed approach can be easily applied for and transferred to semantic segmentation of land-use variables and other surface variables of remotely sensed images.

scholarly journals Accelerating Neural Network Inference on FPGA-Based Platforms—A Survey

Electronics ◽  
2021 ◽  
Vol 10 (9) ◽  
pp. 1025
Author(s):  
Ran Wu ◽  
Xinmin Guo ◽  
Jian Du ◽  
Junbao Li
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Network Inference ◽  
Semantic Segmentation ◽  
Object Identification ◽  
Data Reuse ◽  
Practical Applications ◽  
Resource Limited ◽  
Video Recognition ◽  
Future Work

The breakthrough of deep learning has started a technological revolution in various areas such as object identification, image/video recognition and semantic segmentation. Neural network, which is one of representative applications of deep learning, has been widely used and developed many efficient models. However, the edge implementation of neural network inference is restricted because of conflicts between the high computation and storage complexity and resource-limited hardware platforms in applications scenarios. In this paper, we research neural networks which are involved in the acceleration on FPGA-based platforms. The architecture of networks and characteristics of FPGA are analyzed, compared and summarized, as well as their influence on acceleration tasks. Based on the analysis, we generalize the acceleration strategies into five aspects—computing complexity, computing parallelism, data reuse, pruning and quantization. Then previous works on neural network acceleration are introduced following these topics. We summarize how to design a technical route for practical applications based on these strategies. Challenges in the path are discussed to provide guidance for future work.

Sign in / Sign up

Export Citation Format

Share Document