scholarly journals Efficient Content-Based Sparse Attention with Routing Transformers

2021 ◽  
Vol 9 ◽  
pp. 53-68
Author(s):  
Aurko Roy ◽  
Mohammad Saffar ◽  
Ashish Vaswani ◽  
David Grangier
Keyword(s):  
State Of The Art ◽  
Language Modeling ◽  
Sequence Length ◽  
Image Generation ◽  
Data Set ◽  
Sliding Windows ◽  
Sequence Modeling ◽  
Wide Range ◽  
Small Set ◽  
Transformer Model

Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic computation and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: It combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to O( n1.5d) from O( n2d) for sequence length n and hidden dimension d. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity), as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22 layer Routing Transformer model trained on sequences of length 8192. We open-source the code for Routing Transformer in Tensorflow.1

Ensemble-based deep meta learning for medical image segmentation

2021 ◽  
pp. 1-7
Author(s):  
Usman Ahmed ◽  
Jerry Chun-Wei Lin ◽  
Gautam Srivastava
Keyword(s):  
Image Segmentation ◽  
Deep Learning ◽  
Learning Algorithm ◽  
State Of The Art ◽  
Medical Image Segmentation ◽  
The State ◽  
Data Set ◽  
Deep Learning Algorithm ◽  
Meta Learning ◽  
Small Set

Deep learning methods have led to a state of the art medical applications, such as image classification and segmentation. The data-driven deep learning application can help stakeholders to collaborate. However, limited labelled data set limits the deep learning algorithm to generalize for one domain into another. To handle the problem, meta-learning helps to learn from a small set of data. We proposed a meta learning-based image segmentation model that combines the learning of the state-of-the-art model and then used it to achieve domain adoption and high accuracy. Also, we proposed a prepossessing algorithm to increase the usability of the segments part and remove noise from the new test image. The proposed model can achieve 0.94 precision and 0.92 recall. The ability to increase 3.3% among the state-of-the-art algorithms.

scholarly journals Universal Word Segmentation: Implementation and Interpretation

2018 ◽  
Vol 6 ◽  
pp. 421-435 ◽  
Author(s):  
Yan Shao ◽  
Christian Hardmeier ◽  
Joakim Nivre
Keyword(s):  
State Of The Art ◽  
Word Segmentation ◽  
Experimental Results ◽  
Word Boundary ◽  
Writing Systems ◽  
Low Level ◽  
Wide Range ◽  
Segmentation Accuracy ◽  
Small Set

Word segmentation is a low-level NLP task that is non-trivial for a considerable number of languages. In this paper, we present a sequence tagging framework and apply it to word segmentation for a wide range of languages with different writing systems and typological characteristics. Additionally, we investigate the correlations between various typological factors and word segmentation accuracy. The experimental results indicate that segmentation accuracy is positively related to word boundary markers and negatively to the number of unique non-segmental terms. Based on the analysis, we design a small set of language-specific settings and extensively evaluate the segmentation system on the Universal Dependencies datasets. Our model obtains state-of-the-art accuracies on all the UD languages. It performs substantially better on languages that are non-trivial to segment, such as Chinese, Japanese, Arabic and Hebrew, when compared to previous work.

scholarly journals Bokeh Effect Rendering with Vision Transformers

2022 ◽  
Author(s):  
Hariharan Nagasubramaniam ◽  
Rabih Younes
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Large Diameter ◽  
Depth Map ◽  
Global Information ◽  
Initial Training ◽  
Data Set ◽  
Current State ◽  
Mobile Cameras ◽  
Transformer Model

Bokeh effect is growing to be an important feature in photography, essentially to choose an object of interest to be in focus with the rest of the background being blurred. While naturally rendering this effect requires a DSLR with large diameter of aperture, with the current advancements in Deep Learning, this effect can also be produced in mobile cameras. Most of the existing methods use Convolutional Neural Networks while some relying on the depth map to render this effect. In this paper, we propose an end-to-end Vision Transformer model for Bokeh rendering of images from monocular camera. This architecture uses vision transformers as backbone, thus learning from the entire image rather than just the parts from the filters in a CNN. This property of retaining global information coupled with initial training of the model for image restoration before training to render the blur effect for the background, allows our method to produce clearer images and outperform the current state-of-the-art models on the EBB! Data set. The code to our proposed method can be found at: https://github.com/Soester10/ Bokeh-Rendering-with-Vision-Transformers.

Densification: Semantic document analysis using Wikipedia

2013 ◽  
Vol 20 (4) ◽  
pp. 469-500 ◽  
Author(s):  
IUSTIN DORNESCU ◽  
CONSTANTIN ORĂSAN
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Document Analysis ◽  
Semantic Retrieval ◽  
Entity Linking ◽  
Data Set ◽  
Named Entity ◽  
Wide Range ◽  
The Impact ◽  
Level Of Agreement

AbstractThis paper proposes a new method for semantic document analysis: densification, which identifies and ranks Wikipedia pages relevant to a given document. Although there are similarities with established tasks such as wikification and entity linking, the method does not aim for strict disambiguation of named entity mentions. Instead, densification uses existing links to rank additional articles that are relevant to the document, a form of explicit semantic indexing that enables higher-level semantic retrieval procedures that can be beneficial for a wide range of NLP applications. Because a gold standard for densification evaluation does not exist, a study is carried out to investigate the level of agreement achievable by humans, which questions the feasibility of creating an annotated data set. As a result, a semi-supervised approach is employed to develop a two-stage densification system: filtering unlikely candidate links and then ranking the remaining links. In a first evaluation experiment, Wikipedia articles are used to automatically estimate the performance in terms of recall. Results show that the proposed densification approach outperforms several wikification systems. A second experiment measures the impact of integrating the links predicted by the densification system into a semantic question answering (QA) system that relies on Wikipedia links to answer complex questions. Densification enables the QA system to find twice as many additional answers than when using a state-of-the-art wikification system.

Accurate Detection of Built-Up Areas from High-Resolution Remote Sensing Imagery Using a Fully Convolutional Network

2019 ◽  
Vol 85 (10) ◽  
pp. 737-752
Author(s):  
Yihua Tan ◽  
Shengzhou Xiong ◽  
Zhi Li ◽  
Jinwen Tian ◽  
Yansheng Li
Keyword(s):  
Remote Sensing ◽  
State Of The Art ◽  
Automatic Extraction ◽  
Feature Maps ◽  
Convolutional Network ◽  
Data Set ◽  
Fully Convolutional Network ◽  
Sensing Applications ◽  
Wide Range ◽  
Remote Sensing Applications

The analysis of built-up areas has always been a popular research topic for remote sensing applications. However, automatic extraction of built-up areas from a wide range of regions remains challenging. In this article, a fully convolutional network (FCN)–based strategy is proposed to address built-up area extraction. The proposed algorithm can be divided into two main steps. First, divide the remote sensing image into blocks and extract their deep features by a lightweight multi-branch convolutional neural network (LMB-CNN). Second, rearrange the deep features into feature maps that are fed into a well-designed FCN for image segmentation. Our FCN is integrated with multi-branch blocks and outputs multi-channel segmentation masks that are utilized to balance the false alarm and missing alarm. Experiments demonstrate that the overall classification accuracy of the proposed algorithm can achieve 98.75% in the test data set and that it has a faster processing compared with the existing state-of-the-art algorithms.

scholarly journals Character-Level Language Modeling with Deeper Self-Attention

2019 ◽  
Vol 33 ◽  
pp. 3159-3166 ◽  
Author(s):  
Rami Al-Rfou ◽  
Dokook Choe ◽  
Noah Constant ◽  
Mandy Guo ◽  
Llion Jones
Keyword(s):  
State Of The Art ◽  
Language Modeling ◽  
Large Margin ◽  
Backpropagation Through Time ◽  
Network Layers ◽  
Strong Performance ◽  
Transformer Model

LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model (Vaswani et al. 2017) with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.

scholarly journals Bokeh Effect Rendering with Vision Transformers

2022 ◽  
Author(s):  
Hariharan Nagasubramaniam ◽  
Rabih Younes
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Large Diameter ◽  
Depth Map ◽  
Global Information ◽  
Initial Training ◽  
Data Set ◽  
Current State ◽  
Mobile Cameras ◽  
Transformer Model

Bokeh effect is growing to be an important feature in photography, essentially to choose an object of interest to be in focus with the rest of the background being blurred. While naturally rendering this effect requires a DSLR with large diameter of aperture, with the current advancements in Deep Learning, this effect can also be produced in mobile cameras. Most of the existing methods use Convolutional Neural Networks while some relying on the depth map to render this effect. In this paper, we propose an end-to-end Vision Transformer model for Bokeh rendering of images from monocular camera. This architecture uses vision transformers as backbone, thus learning from the entire image rather than just the parts from the filters in a CNN. This property of retaining global information coupled with initial training of the model for image restoration before training to render the blur effect for the background, allows our method to produce clearer images and outperform the current state-of-the-art models on the EBB! Data set. The code to our proposed method can be found at: https://github.com/Soester10/ Bokeh-Rendering-with-Vision-Transformers.

Structure Guided Molecular Docking Assisted Alignment Dependent 3DQSAR Study on Steroidal Aromatase Inhibitors (SAIs) as Anti-breast Cancer Agents

2019 ◽  
Vol 16 (7) ◽  
pp. 808-817 ◽  
Author(s):  
Laxmi Banjare ◽  
Sant Kumar Verma ◽  
Akhlesh Kumar Jain ◽  
Suresh Thareja
Keyword(s):  
Breast Cancer ◽  
Molecular Docking ◽  
Aromatase Inhibitors ◽  
Direct Synthesis ◽  
3D Qsar ◽  
Chemotherapeutic Agents ◽  
Modeling Tools ◽  
Data Set ◽  
Wide Range ◽  
Steroidal Aromatase Inhibitors

Background: In spite of the availability of various treatment approaches including surgery, radiotherapy, and hormonal therapy, the steroidal aromatase inhibitors (SAIs) play a significant role as chemotherapeutic agents for the treatment of estrogen-dependent breast cancer with the benefit of reduced risk of recurrence. However, due to greater toxicity and side effects associated with currently available anti-breast cancer agents, there is emergent requirement to develop target-specific AIs with safer anti-breast cancer profile. Methods: It is challenging task to design target-specific and less toxic SAIs, though the molecular modeling tools viz. molecular docking simulations and QSAR have been continuing for more than two decades for the fast and efficient designing of novel, selective, potent and safe molecules against various biological targets to fight the number of dreaded diseases/disorders. In order to design novel and selective SAIs, structure guided molecular docking assisted alignment dependent 3D-QSAR studies was performed on a data set comprises of 22 molecules bearing steroidal scaffold with wide range of aromatase inhibitory activity. Results: 3D-QSAR model developed using molecular weighted (MW) extent alignment approach showed good statistical quality and predictive ability when compared to model developed using moments of inertia (MI) alignment approach. Conclusion: The explored binding interactions and generated pharmacophoric features (steric and electrostatic) of steroidal molecules could be exploited for further design, direct synthesis and development of new potential safer SAIs, that can be effective to reduce the mortality and morbidity associated with breast cancer.

CLINICAL AND FORENSIC ASPECTS OF PHARMACOBEZOARS

2020 ◽  
Vol 12 ◽  
Author(s):  
Francisco Basílio ◽  
Ricardo Jorge Dinis-Oliveira
Keyword(s):  
State Of The Art ◽  
Signs And Symptoms ◽  
Immediate Release ◽  
Diagnosis And Treatment ◽  
Blood Concentrations ◽  
Forensic Practice ◽  
Wide Range ◽  
Therapeutic Doses ◽  
Complete History ◽  
Lethal Blood

Background: Pharmacobezoars are specific types of bezoars formed when medicines, such as tablets, suspensions, and/or drug delivery systems, aggregate and may cause death by occluding airways with tenacious material or by eluting drugs resulting in toxic or lethal blood concentrations. Objective: This work aims to fully review the state-of-the-art regarding pathophysiology, diagnosis, treatment and other relevant clinical and forensic features of pharmacobezoars. Results: patients of a wide range of ages and in both sexes present with signs and symptoms of intoxications or more commonly gastrointestinal obstructions. The exact mechanisms of pharmacobezoar formation are unknown but is likely multifactorial. The diagnosis and treatment depend on the gastrointestinal segment affected and should be personalized to the medication and the underlying factor. A good and complete history, physical examination, image tests, upper endoscopy and surgery through laparotomy of the lower tract are useful for diagnosis and treatment. Conclusion: Pharmacobezoars are rarely seen in clinical and forensic practice. They are related to controlled or immediate-release formulations, liquid or non-digestible substances, in normal or altered digestive motility/anatomy tract, and in overdoses or therapeutic doses, and should be suspected in the presence of risk factors or patients taking drugs which may form pharmacobezoars.

Integrative Data Analysis from a Unifying Research Synthesis Perspective

2018 ◽  
Author(s):  
Eun-Young Mun ◽  
Anne E. Ray
Keyword(s):  
Data Analysis ◽  
Large Scale ◽  
Research Synthesis ◽  
Alcohol Intervention ◽  
Data Set ◽  
Integrative Data Analysis ◽  
Level Data ◽  
Model Complex ◽  
Wide Range ◽  
Individual Participant

Integrative data analysis (IDA) is a promising new approach in psychological research and has been well received in the field of alcohol research. This chapter provides a larger unifying research synthesis framework for IDA. Major advantages of IDA of individual participant-level data include better and more flexible ways to examine subgroups, model complex relationships, deal with methodological and clinical heterogeneity, and examine infrequently occurring behaviors. However, between-study heterogeneity in measures, designs, and samples and systematic study-level missing data are significant barriers to IDA and, more broadly, to large-scale research synthesis. Based on the authors’ experience working on the Project INTEGRATE data set, which combined individual participant-level data from 24 independent college brief alcohol intervention studies, it is also recognized that IDA investigations require a wide range of expertise and considerable resources and that some minimum standards for reporting IDA studies may be needed to improve transparency and quality of evidence.

Sign in / Sign up

Export Citation Format

Share Document