scholarly journals Convergence analysis of projected fixed-point iteration on a low-rank matrix manifold

2018 ◽  
Vol 25 (5) ◽  
pp. e2140 ◽  
Author(s):  
D. A. Kolesnikov ◽  
I. V. Oseledets
Keyword(s):  
Fixed Point ◽  
Convergence Analysis ◽  
Low Rank ◽  
Fixed Point Iteration ◽  
Rank Matrix ◽  
Low Rank Matrix

scholarly journals Online Embedding Compression for Text Classification Using Low Rank Matrix Factorization

2019 ◽  
Vol 33 ◽  
pp. 6196-6203 ◽  
Author(s):  
Anish Acharya ◽  
Rahul Goel ◽  
Angeliki Metallinou ◽  
Inderjit Dhillon
Keyword(s):  
Fixed Point ◽  
Adaptive Learning ◽  
Matrix Factorization ◽  
Learning Rate ◽  
Word Embedding ◽  
Low Rank ◽  
Alternative Methods ◽  
Rank Matrix ◽  
Sentence Classification ◽  
Low Rank Matrix

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training, to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed-point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.

Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks

2018 ◽  
Author(s):  
Daniel Povey ◽  
Gaofeng Cheng ◽  
Yiming Wang ◽  
Ke Li ◽  
Hainan Xu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Matrix Factorization ◽  
Deep Neural Networks ◽  
Low Rank ◽  
Rank Matrix ◽  
Low Rank Matrix

Adaptive Local Low-rank Matrix Approximation for Recommendation

2019 ◽  
Vol 37 (4) ◽  
pp. 1-34 ◽  
Author(s):  
Huafeng Liu ◽  
Liping Jing ◽  
Yuhua Qian ◽  
Jian Yu
Keyword(s):  
Low Rank ◽  
Matrix Approximation ◽  
Rank Matrix ◽  
Low Rank Matrix ◽  
Low Rank Matrix Approximation

scholarly journals Single-cell data clustering based on sparse optimization and low-rank matrix factorization

2021 ◽  
Author(s):  
Yinlei Hu ◽  
Bin Li ◽  
Falai Chen ◽  
Kun Qu
Keyword(s):  
Single Cell ◽  
Rna Sequencing ◽  
Matrix Factorization ◽  
Data Clustering ◽  
Cell Types ◽  
Low Rank ◽  
Sequencing Data ◽  
Rank Matrix ◽  
Single Cell Rna Sequencing ◽  
Low Rank Matrix

Abstract Unsupervised clustering is a fundamental step of single-cell RNA sequencing data analysis. This issue has inspired several clustering methods to classify cells in single-cell RNA sequencing data. However, accurate prediction of the cell clusters remains a substantial challenge. In this study, we propose a new algorithm for single-cell RNA sequencing data clustering based on Sparse Optimization and low-rank matrix factorization (scSO). We applied our scSO algorithm to analyze multiple benchmark datasets and showed that the cluster number predicted by scSO was close to the number of reference cell types and that most cells were correctly classified. Our scSO algorithm is available at https://github.com/QuKunLab/scSO. Overall, this study demonstrates a potent cell clustering approach that can help researchers distinguish cell types in single-cell RNA sequencing data.

Sign in / Sign up

Export Citation Format

Share Document