Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations

2019 ◽  
Author(s):  
Hao Wu ◽  
Jiayuan Mao ◽  
Yufeng Zhang ◽  
Yuning Jiang ◽  
Lei Li ◽  
...  
Keyword(s):  
Structured Meaning ◽  
Vision And Language

scholarly journals Joint Multimodal Embedding and Backtracking Search in Vision-and-Language Navigation

Sensors ◽  
2021 ◽  
Vol 21 (3) ◽  
pp. 1012
Author(s):  
Jisu Hwang ◽  
Incheol Kim
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Input Data ◽  
Language Instruction ◽  
Scoring Method ◽  
Processing Technologies ◽  
Backtracking Search ◽  
Panoramic Images ◽  
Benchmark Datasets ◽  
Vision And Language

Due to the development of computer vision and natural language processing technologies in recent years, there has been a growing interest in multimodal intelligent tasks that require the ability to concurrently understand various forms of input data such as images and text. Vision-and-language navigation (VLN) require the alignment and grounding of multimodal input data to enable real-time perception of the task status on panoramic images and natural language instruction. This study proposes a novel deep neural network model (JMEBS), with joint multimodal embedding and backtracking search for VLN tasks. The proposed JMEBS model uses a transformer-based joint multimodal embedding module. JMEBS uses both multimodal context and temporal context. It also employs backtracking-enabled greedy local search (BGLS), a novel algorithm with a backtracking feature designed to improve the task success rate and optimize the navigation path, based on the local and global scores related to candidate actions. A novel global scoring method is also used for performance improvement by comparing the partial trajectories searched thus far with a plurality of natural language instructions. The performance of the proposed model on various operations was then experimentally demonstrated and compared with other models using the Matterport3D Simulator and room-to-room (R2R) benchmark datasets.

scholarly journals Neural processing of vision and language in kindergarten is associated with prereading skills and predicts future literacy

2021 ◽  
Author(s):  
Johanna Liebig ◽  
Eva Froehlich ◽  
Teresa Sylvester ◽  
Mario Braun ◽  
Hauke R. Heekeren ◽  
...  
Keyword(s):  
Neural Processing ◽  
Prereading Skills ◽  
Vision And Language

scholarly journals Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

2020 ◽  
Author(s):  
Alexander Ku ◽  
Peter Anderson ◽  
Roma Patel ◽  
Eugene Ie ◽  
Jason Baldridge
Keyword(s):  
Vision And Language

scholarly journals Active scene recognition with vision and language

2011 ◽  
Author(s):  
Xiaodong Yu ◽  
Cornelia Fermuller ◽  
Ching Lik Teo ◽  
Yezhou Yang ◽  
Yiannis Aloimonos
Keyword(s):  
Scene Recognition ◽  
Vision And Language
Sign in / Sign up

Export Citation Format

Share Document