scholarly journals Deep Feature-Level Sensor Fusion Using Skip Connections for Real-Time Object Detection in Autonomous Driving

Electronics ◽  
2021 ◽  
Vol 10 (4) ◽  
pp. 424
Author(s):  
Vijay John ◽  
Seiichi Mita
Keyword(s):  
Feature Extraction ◽  
Object Detection ◽  
Sensor Fusion ◽  
Autonomous Driving ◽  
Driver Assistance Systems ◽  
Wave Radar ◽  
Level Sensor ◽  
Environmental Variations ◽  
The Individual ◽  
Public Datasets

Object detection is an important perception task in autonomous driving and advanced driver assistance systems. The visible camera is widely used for perception, but its performance is limited by illumination and environmental variations. For robust vision-based perception, we propose a deep learning framework for effective sensor fusion of the visible camera with complementary sensors. A feature-level sensor fusion technique, using skip connection, is proposed for the sensor fusion of the visible camera with the millimeter-wave radar and the thermal camera. The two networks are called the RV-Net and the TV-Net, respectively. These networks have two input branches and one output branch. The input branches contain separate branches for the individual sensor feature extraction, which are then fused in the output perception branch using skip connections. The RVNet and the TVNet simultaneously perform sensor-specific feature extraction, feature-level fusion and object detection within an end-to-end framework. The proposed networks are validated with baseline algorithms on public datasets. The results obtained show that the feature-level sensor fusion is better than baseline early and late fusion frameworks.

scholarly journals DEVELOPMENT OF VEHICLE TRACKING USING SENSOR FUSION

2021 ◽  
Vol 9 (2) ◽  
pp. 731-739
Author(s):  
M Hyndhavi, Et. al.
Keyword(s):  
Sensor Fusion ◽  
Weather Conditions ◽  
Autonomous Driving ◽  
Vehicle Tracking ◽  
Security Requirements ◽  
Real Time Control ◽  
Driver Assistance Systems ◽  
Radar Sensors ◽  
Time Control ◽  
Sensor Information

The development of vehicle tracking using sensor fusion is presented in this paper. Advanced driver assistance systems (ADAS) are becoming more popular in recent years. These systems use sensor information for real-time control. To improve the standard and robustness, especially in the presence of environmental noises like varying lighting, weather conditions, and fusion of sensors has been the center of attention in recent studies. Faced with complex traffic conditions, the single sensor has been unable to meet the security requirements of ADAS and autonomous driving. The common environment perception sensors consist of radar, camera, and lidar which have both pros and cons. The sensor fusion is a necessary technology for autonomous driving which provides a better vision and understanding of vehicles surrounding. We mainly focus on highway scenarios that enable an autonomous car to comfortably follow other cars at various speeds while keeping a secure distance and mix the advantages of both sensors with a sensor fusion approach. The radar and vision sensor information are fused to produce robust and accurate measurements. And the experimental results indicate that the comparison of using only radar sensors and sensor fusion of both camera and radar sensors is presented in this paper. The algorithm is described along with simulation results by using MATLAB.

Sensor Fusion for Advanced Driver Assistance Systems

2017 ◽  
Author(s):  
Mario Amoruso ◽  
Stefano Caiola ◽  
Giuseppe Doronzo ◽  
Marino Difino
Keyword(s):  
Sensor Fusion ◽  
Autonomous Driving ◽  
White Paper ◽  
System Level ◽  
Driver Assistance ◽  
Driver Assistance Systems ◽  
Advanced Driver Assistance Systems ◽  
Fusion Concept ◽  
Assistance Systems ◽  
Initial Research

As vehicles move toward autonomous capability, there is a rising need for hardware-in-the loop (HIL) testing to validate and verify the functionality of advanced driver assistance systems (ADAS), which are anticipated to play a central role in autonomous driving. This white paper gives an overview of the ADAS HIL with sensor fusion concept, shares main takeaways from initial research efforts, and highlights key system-level elements used to implement the application.

scholarly journals An Enhanced Feature Pyramid Object Detection Network for Autonomous Driving

2019 ◽  
Vol 9 (20) ◽  
pp. 4363 ◽  
Author(s):  
Yutian Wu ◽  
Shuming Tang ◽  
Shuwei Zhang ◽  
Harutoshi Ogai
Keyword(s):  
Feature Extraction ◽  
Object Detection ◽  
Autonomous Driving ◽  
Detection Accuracy ◽  
Parallel Detection ◽  
Occluded Objects ◽  
Feature Weight ◽  
Time Requirements ◽  
Feature Pyramid ◽  
High Level

Feature Pyramid Network (FPN) builds a high-level semantic feature pyramid and detects objects of different scales in corresponding pyramid levels. Usually, features within the same pyramid levels have the same weight for subsequent object detection, which ignores the feature requirements of different scale objects. As we know, for most detection networks, it is hard to detect small objects and occluded objects because there is little information to exploit. To solve the above problems, we propose an Enhanced Feature Pyramid Object Detection Network (EFPN), which innovatively constructs an enhanced feature extraction subnet and adaptive parallel detection subnet. Enhanced feature extraction subnet introduces Feature Weight Module (FWM) to enhance pyramid features by weighting the fusion feature map. Adaptive parallel detection subnet introduces Adaptive Context Expansion (ACE) and Parallel Detection Branch (PDB). ACE aims to generate the features of adaptively enlarged object context region and original region. PDB predicts classification and regression results separately with the two features. Experiments showed that EFPN outperforms FPN in detection accuracy on Pascal VOC and KITTI datasets. Furthermore, the performance of EFPN meets the real-time requirements of autonomous driving systems.

scholarly journals LiDAR and Camera Fusion Approach for Object Distance Estimation in Self-Driving Vehicles

Symmetry ◽  
2020 ◽  
Vol 12 (2) ◽  
pp. 324 ◽  
Author(s):  
G Ajay Kumar ◽  
Jin Hee Lee ◽  
Jongrak Hwang ◽  
Jaehyeong Park ◽  
Sung Hoon Youn ◽  
...  
Keyword(s):  
Sensor Fusion ◽  
Autonomous Vehicles ◽  
Distance Estimation ◽  
Autonomous Driving ◽  
Geometric Transformation ◽  
Low Level ◽  
Object Distance ◽  
Level Sensor ◽  
And Performance ◽  
Fusion Approach

The fusion of light detection and ranging (LiDAR) and camera data in real-time is known to be a crucial process in many applications, such as in autonomous driving, industrial automation, and robotics. Especially in the case of autonomous vehicles, the efficient fusion of data from these two types of sensors is important to enabling the depth of objects as well as the detection of objects at short and long distances. As both the sensors are capable of capturing the different attributes of the environment simultaneously, the integration of those attributes with an efficient fusion approach greatly benefits the reliable and consistent perception of the environment. This paper presents a method to estimate the distance (depth) between a self-driving car and other vehicles, objects, and signboards on its path using the accurate fusion approach. Based on the geometrical transformation and projection, low-level sensor fusion was performed between a camera and LiDAR using a 3D marker. Further, the fusion information is utilized to estimate the distance of objects detected by the RefineDet detector. Finally, the accuracy and performance of the sensor fusion and distance estimation approach were evaluated in terms of quantitative and qualitative analysis by considering real road and simulation environment scenarios. Thus the proposed low-level sensor fusion, based on the computational geometric transformation and projection for object distance estimation proves to be a promising solution for enabling reliable and consistent environment perception ability for autonomous vehicles.

scholarly journals Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 2140
Author(s):  
De Jong Yeong ◽  
Gustavo Velasco-Hernandez ◽  
John Barry ◽  
Joseph Walsh
Keyword(s):  
Object Detection ◽  
Sensor Fusion ◽  
Autonomous Vehicles ◽  
Autonomous Driving ◽  
Obstacle Detection ◽  
Future Research ◽  
Sensor Calibration ◽  
Automated Driving ◽  
Detection Techniques ◽  
Technical Performance

With the significant advancement of sensor and communication technology and the reliable application of obstacle detection techniques and algorithms, automated driving is becoming a pivotal technology that can revolutionize the future of transportation and mobility. Sensors are fundamental to the perception of vehicle surroundings in an automated driving system, and the use and performance of multiple integrated sensors can directly determine the safety and feasibility of automated driving vehicles. Sensor calibration is the foundation block of any autonomous system and its constituent sensors and must be performed correctly before sensor fusion and obstacle detection processes may be implemented. This paper evaluates the capabilities and the technical performance of sensors which are commonly employed in autonomous vehicles, primarily focusing on a large selection of vision cameras, LiDAR sensors, and radar sensors and the various conditions in which such sensors may operate in practice. We present an overview of the three primary categories of sensor calibration and review existing open-source calibration packages for multi-sensor calibration and their compatibility with numerous commercial sensors. We also summarize the three main approaches to sensor fusion and review current state-of-the-art multi-sensor fusion techniques and algorithms for object detection in autonomous driving applications. The current paper, therefore, provides an end-to-end review of the hardware and software methods required for sensor fusion object detection. We conclude by highlighting some of the challenges in the sensor fusion field and propose possible future research directions for automated driving systems.

scholarly journals A Two-Stage Data Association Approach for 3D Multi-Object Tracking

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 2894
Author(s):  
Minh-Quan Dao ◽  
Vincent Frémont
Keyword(s):  
Object Detection ◽  
Object Tracking ◽  
Moving Objects ◽  
Data Association ◽  
Autonomous Driving ◽  
Tracking Accuracy ◽  
Two Stage ◽  
Bipartite Matching ◽  
3D Object ◽  
3D Object Detection

Multi-Object Tracking (MOT) is an integral part of any autonomous driving pipelines because it produces trajectories of other moving objects in the scene and predicts their future motion. Thanks to the recent advances in 3D object detection enabled by deep learning, track-by-detection has become the dominant paradigm in 3D MOT. In this paradigm, a MOT system is essentially made of an object detector and a data association algorithm which establishes track-to-detection correspondence. While 3D object detection has been actively researched, association algorithms for 3D MOT has settled at bipartite matching formulated as a Linear Assignment Problem (LAP) and solved by the Hungarian algorithm. In this paper, we adapt a two-stage data association method which was successfully applied to image-based tracking to the 3D setting, thus providing an alternative for data association for 3D MOT. Our method outperforms the baseline using one-stage bipartite matching for data association by achieving 0.587 Average Multi-Object Tracking Accuracy (AMOTA) in NuScenes validation set and 0.365 AMOTA (at level 2) in Waymo test set.

Sign in / Sign up

Export Citation Format

Share Document