Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
arXiv.org
18 Mar 2024
Abstract
Compressing high-capability Large Language Models (LLMs) has emerged as a
favored strategy for resource-efficient inferences. While state-of-the-art
(SoTA) compression methods boast impressive advancements in preserving benign
task performance, the potential risks of compression in terms of safety and
trustworthiness have been largely neglected. This study conducts the first,
thorough evaluation of three (3) leading LLMs using five (5) SoTA compression
techniques across eight (8) trustworthiness dimensions. Our experiments
highlight the intricate interplay between compression and trustworthiness,
revealing some interesting patterns. We find that quantization is currently a
more effective approach than pruning in achieving efficiency and
trustworthiness simultaneously. For instance, a 4-bit quantized model retains
the trustworthiness of its original counterpart, but model pruning
significantly degrades trustworthiness, even at 50% sparsity. Moreover,
employing quantization within a moderate bit range could unexpectedly improve
certain trustworthiness dimensions such as ethics and fairness. Conversely,
extreme quantization to very low bit levels (3 bits) tends to significantly
reduce trustworthiness. This increased risk cannot be uncovered by looking at
benign performance alone, in turn, mandating comprehensive trustworthiness
evaluation in practice. These findings culminate in practical recommendations
for simultaneously achieving high utility, efficiency, and trustworthiness in
LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
Metrics
8 Record Views
Details
- Title
- Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
- Creators
- Junyuan HongJinhao DuanChenhui ZhangZhangheng LiChulin XieKelsey LiebermanJames DiffenderferBrian BartoldsonAjay JaiswalKaidi XuBhavya KailkhuraDan HendrycksDawn SongZhangyang WangBo Li
- Publication Details
- arXiv.org
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science (Computing)
- Other Identifier
- 991021871482504721