Conference proceeding
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
International Conference on Machine Learning, ICML 2024, v 235, pp 18611-18633
2024
Abstract
Compressing high-capability Large Language Models (LLMs) has emerged as a favored strategy for resource-efficient inferences. While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected. This study conducts the first, thorough evaluation of three (3) leading LLMs using five (5) SoTA compression techniques across eight (8) trustworthiness dimensions. Our experiments highlight the intricate interplay between compression and trustworthiness, revealing some interesting patterns. We find that quantization is currently a more effective approach than pruning in achieving efficiency and trustworthiness simultaneously. For instance, a 4- bit quantized model retains the trustworthiness of its original counterpart, but model pruning significantly degrades trustworthiness, even at 50% sparsity. Moreover, employing quantization within a moderate bit range could unexpectedly improve certain trustworthiness dimensions such as ethics and fairness. Conversely, extreme quantization to very low bit levels (3 bits) tends to reduce trustworthiness significantly. This increased risk cannot be uncovered by looking at benign performance alone, in turn, mandating comprehensive trustworthiness evaluation in practice. These findings culminate in practical recommendations for simultaneously achieving high utility, efficiency, and trustworthiness in LLMs.
Metrics
1 Record Views
Details
- Title
- Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
- Creators
- Junyuan Hong - The University of Texas at AustinJinhao Duan - Drexel UniversityChenhui Zhang - Massachusetts Institute of TechnologyZhangheng Li - The University of Texas at AustinChulin Xie - University of Illinois Urbana-ChampaignKelsey Lieberman - Duke UniversityJames Diffenderfer - Lawrence Livermore National LaboratoryBrian Bartoldson - Lawrence Livermore National LaboratoryAjay Jaiswal - The University of Texas at AustinKaidi Xu - Drexel UniversityBhavya Kailkhura - Lawrence Livermore National LaboratoryDan Hendrycks - Center for AI Safety (United States, San Francisco)Dawn Song - University of California, BerkeleyZhangyang Wang - The University of Texas at AustinBo Li - University of Chicago
- Publication Details
- International Conference on Machine Learning, ICML 2024, v 235, pp 18611-18633
- Conference
- ICML'24: Proceedings of the 41st International Conference on Machine Learning, 41st (Vienna, Austria , 21 Jul 2024–27 Jul 2024)
- Publisher
- ACM
- Number of pages
- 23
- Grant note
- IIS-2212176 / National Natural Science Foundation of China (501100001809) 1910100 / National Natural Science Foundation of China (501100001809) Lawrence Livermore National Laboratory (http://data.elsevier.com/vocabulary/SciValFunders/100006227) 2319242; 1910100; 2046726; 2229876 / National Science Foundation (http://data.elsevier.com/vocabulary/SciValFunders/100000001) 23-ER-030 (LLNL-CONF-860188) / Lawrence Livermore National Laboratory (100006227) 2319242 / National Natural Science Foundation of China (501100001809) 80NSSC20M0229 / National Aeronautics and Space Administration (100000104) IIS-2212176 / Alfred P. Sloan Fellowship Laboratory Directed Research and Development (http://data.elsevier.com/vocabulary/SciValFunders/100007000) 2046726 / National Natural Science Foundation of China (501100001809)
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Computer Science
- Scopus ID
- 2-s2.0-85203817859
- Other Identifier
- 991022202508004721