Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
RBFormer: Improve Adversarial Robustness of Transformer by Robust Bias
22 Sep 2023
Abstract
Recently, there has been a surge of interest and attention in
Transformer-based structures, such as Vision Transformer (ViT) and Vision
Multilayer Perceptron (VMLP). Compared with the previous convolution-based
structures, the Transformer-based structure under investigation showcases a
comparable or superior performance under its distinctive attention-based input
token mixer strategy. Introducing adversarial examples as a robustness
consideration has had a profound and detrimental impact on the performance of
well-established convolution-based structures. This inherent vulnerability to
adversarial attacks has also been demonstrated in Transformer-based structures.
In this paper, our emphasis lies on investigating the intrinsic robustness of
the structure rather than introducing novel defense measures against
adversarial attacks. To address the susceptibility to robustness issues, we
employ a rational structure design approach to mitigate such vulnerabilities.
Specifically, we enhance the adversarial robustness of the structure by
increasing the proportion of high-frequency structural robust biases. As a
result, we introduce a novel structure called Robust Bias Transformer-based
Structure (RBFormer) that shows robust superiority compared to several existing
baseline structures. Through a series of extensive experiments, RBFormer
outperforms the original structures by a significant margin, achieving an
impressive improvement of +16.12% and +5.04% across different evaluation
criteria on CIFAR-10 and ImageNet-1k, respectively.
Metrics
4 Record Views
Details
- Title
- RBFormer: Improve Adversarial Robustness of Transformer by Robust Bias
- Creators
- Hao ChengJinhao DuanHui LiLyutianyang ZhangJiahang CaoPing WangJize ZhangKaidi XuRenjing Xu
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science (Computing)
- Other Identifier
- 991021871466804721