scholarly journals Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

2021 ◽  
Vol 9 ◽  
pp. 1061-1080
Author(s):  
Prakhar Ganesh ◽  
Yao Chen ◽  
Xin Lou ◽  
Mohammad Ali Khan ◽  
Yin Yang ◽  
...  
Keyword(s):  
Language Processing ◽  
Large Scale ◽  
State Of The Art ◽  
Future Research ◽  
Research Directions ◽  
Research Attention ◽  
Model Compression ◽  
Future Research Directions ◽  
Or Applications

Abstract Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.

scholarly journals Designing a solver competition: the QBFEVAL'10 case study

10.29007/cv3b ◽  
2018 ◽  
Author(s):  
Claudia Peschiera ◽  
Luca Pulina ◽  
Armando Tacchella
Keyword(s):  
State Of The Art ◽  
Future Research ◽  
Research Directions ◽  
Quantified Boolean Formulas ◽  
Current State ◽  
Boolean Formulas ◽  
Future Research Directions

In this paper we report about QBFEVAL'10, the seventh in a series of events established with the aim of assessing the advancements in reasoning about quantified Boolean formulas (QBFs). The paper discusses the results obtained and the evaluation setup, from the criteria used to select QBF instances down to the hardware infrastructure. We also discuss the current state-of-the-art in light of past challenges and we envision future research directions that are motivated by the results of QBFEVAL'10.

scholarly journals SEMANTIC NETWORKS FOR ENGINEERING DESIGN: A SURVEY

2021 ◽  
Vol 1 ◽  
pp. 2621-2630
Author(s):  
Ji Han ◽  
Serhad Sarica ◽  
Feng Shi ◽  
Jianxi Luo
Keyword(s):  
Engineering Design ◽  
Language Processing ◽  
Large Scale ◽  
Design Research ◽  
Semantic Network ◽  
Semantic Networks ◽  
Future Research ◽  
Research Directions ◽  
Engineering Data ◽  
Future Research Directions

AbstractThere have been growing uses of semantic networks in the past decade, such as leveraging large-scale pre-trained graph knowledge databases for various natural language processing (NLP) tasks in engineering design research. Therefore, the paper provides a survey of the research that has employed semantic networks in the engineering design research community. The survey reveals that engineering design researchers have primarily relied on WordNet, ConceptNet, and other common-sense semantic network databases trained on non-engineering data sources to develop methods or tools for engineering design. Meanwhile, there are emerging efforts to mine large scale technical publication and patent databases to construct engineering-contextualized semantic network databases, e.g., B-Link and TechNet, to support NLP in engineering design. On this basis, we recommend future research directions for the construction and applications of engineering-related semantic networks in engineering design research and practice.

scholarly journals Deep Learning for Video Captioning: A Review

2019 ◽  
Author(s):  
Shaoxiang Chen ◽  
Ting Yao ◽  
Yu-Gang Jiang
Keyword(s):  
Deep Learning ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Future Research ◽  
Research Directions ◽  
Video Captioning ◽  
Future Research Directions ◽  
Review State ◽  
Vision And Language

Deep learning has achieved great successes in solving specific artificial intelligence problems recently. Substantial progresses are made on Computer Vision (CV) and Natural Language Processing (NLP). As a connection between the two worlds of vision and language, video captioning is the task of producing a natural-language utterance (usually a sentence) that describes the visual content of a video. The task is naturally decomposed into two sub-tasks. One is to encode a video via a thorough understanding and learn visual representation. The other is caption generation, which decodes the learned representation into a sequential sentence, word by word. In this survey, we first formulate the problem of video captioning, then review state-of-the-art methods categorized by their emphasis on vision or language, and followed by a summary of standard datasets and representative approaches. Finally, we highlight the challenges which are not yet fully understood in this task and present future research directions.

scholarly journals A Survey on Computational Propaganda Detection

2020 ◽  
Author(s):  
Giovanni Da San Martino ◽  
Stefano Cresci ◽  
Alberto Barrón-Cedeño ◽  
Seunghak Yu ◽  
Roberto Di Pietro ◽  
...  
Keyword(s):  
Social Networks ◽  
Social Network ◽  
Network Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Future Research ◽  
The Internet ◽  
Persuasive Messages ◽  
Research Directions ◽  
Future Research Directions

Propaganda campaigns aim at influencing people's mindset with the purpose of advancing a specific agenda. They exploit the anonymity of the Internet, the micro-profiling ability of social networks, and the ease of automatically creating and managing coordinated networks of accounts, to reach millions of social network users with persuasive messages, specifically targeted to topics each individual user is sensitive to, and ultimately influencing the outcome on a targeted issue. In this survey, we review the state of the art on computational propaganda detection from the perspective of Natural Language Processing and Network Analysis, arguing about the need for combined efforts between these communities. We further discuss current challenges and future research directions.

scholarly journals On the Security and Privacy Challenges of Virtual Assistants

Sensors ◽  
2021 ◽  
Vol 21 (7) ◽  
pp. 2312
Author(s):  
Tom Bolton ◽  
Tooska Dargahi ◽  
Sana Belguith ◽  
Mabrook S. Al-Rakhami ◽  
Ali Hassan Sodhro
Keyword(s):  
Language Processing ◽  
State Of The Art ◽  
Security And Privacy ◽  
Future Research ◽  
Research Directions ◽  
Privacy Concerns ◽  
Current Trends ◽  
Current State ◽  
Future Research Directions ◽  
Privacy Issues

Since the purchase of Siri by Apple, and its release with the iPhone 4S in 2011, virtual assistants (VAs) have grown in number and popularity. The sophisticated natural language processing and speech recognition employed by VAs enables users to interact with them conversationally, almost as they would with another human. To service user voice requests, VAs transmit large amounts of data to their vendors; these data are processed and stored in the Cloud. The potential data security and privacy issues involved in this process provided the motivation to examine the current state of the art in VA research. In this study, we identify peer-reviewed literature that focuses on security and privacy concerns surrounding these assistants, including current trends in addressing how voice assistants are vulnerable to malicious attacks and worries that the VA is recording without the user’s knowledge or consent. The findings show that not only are these worries manifold, but there is a gap in the current state of the art, and no current literature reviews on the topic exist. This review sheds light on future research directions, such as providing solutions to perform voice authentication without an external device, and the compliance of VAs with privacy regulations.

scholarly journals Social commerce—state-of-the-art and future research directions

2016 ◽  
Vol 26 (3) ◽  
pp. 269-290 ◽  
Author(s):  
Catherine Baethge ◽  
Julia Klier ◽  
Mathias Klier
Keyword(s):  
State Of The Art ◽  
Future Research ◽  
Social Commerce ◽  
Research Directions ◽  
Future Research Directions
Sign in / Sign up

Export Citation Format

Share Document