Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression
arXiv (Cornell University)
16 Jun 2021
Abstract
Compressing Deep Neural Network (DNN) models to alleviate the storage and
computation requirements is essential for practical applications, especially
for resource limited devices. Although capable of reducing a reasonable amount
of model parameters, previous unstructured or structured weight pruning methods
can hardly truly accelerate inference, either due to the poor hardware
compatibility of the unstructured sparsity or due to the low sparse rate of the
structurally pruned network. Aiming at reducing both storage and computation,
as well as preserving the original task performance, we propose a generalized
weight unification framework at a hardware compatible micro-structured level to
achieve high amount of compression and acceleration. Weight coefficients of a
selected micro-structured block are unified to reduce the storage and
computation of the block without changing the neuron connections, which turns
to a micro-structured pruning special case when all unified coefficients are
set to zero, where neuron connections (hence storage and computation) are
completely removed. In addition, we developed an effective training framework
based on the alternating direction method of multipliers (ADMM), which converts
our complex constrained optimization into separately solvable subproblems.
Through iteratively optimizing the subproblems, the desired micro-structure can
be ensured with high compression ratio and low performance degradation. We
extensively evaluated our method using a variety of benchmark models and
datasets for different applications. Experimental results demonstrate
state-of-the-art performance.
Metrics
3 Record Views
Details
- Title
- Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression
- Creators
- Sheng LinWei JiangWei WangKaidi XuYanzhi WangShan LiuSongnan Li
- Publication Details
- arXiv (Cornell University)
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science (Computing)
- Other Identifier
- 991021871478204721