Files and links (1)
Published, Version of Record (VoR)Open Access via Drexel Libraries Read and Publish Program 2024CC BY-NC-ND V4.0, Open
Journal article
Two-Stage Fine-Tuning with ChatGPT Data Augmentation for Learning Class-Imbalanced Data
Neurocomputing [e-journal], v 592, 127801
11 May 2024
Featured in Collection : Research Supported by Drexel Libraries' OA Programs
Abstract
Classification of long-tailed distributed data is a challenging problem, which suffers from serious class imbalance and hence poor performance on tail classes, which have only a few samples. Owing to this paucity of samples, learning on the tail classes is especially challenging for fine-tuning when transferring a pretrained model to a downstream task. In this work, we present a simple modification of standard fine-tuning to cope with these challenges. Specifically, we propose a two-stage fine-tuning. In Stage 1, we fine-tune the final layer of the pretrained model with class-balanced augmented data, generated using ChatGPT. As a large generative language model, ChatGPT is capable of generating novel and contextually similar responses to a given prompt, which makes it an excellent candidate for data augmentation. In Stage 2, we perform the standard fine-tuning. Our modification has several benefits: (1) it leverages pretrained representations by only fine-tuning a small portion of the model parameters while keeping the rest untouched; (2) it allows the model to learn an initial representation of the specific task; and importantly (3) it protects the learning of tail classes from being at a disadvantage during the model updating. We conduct extensive experiments on synthetic datasets of both two-class and multi-class tasks of text classification as well as a real-world application to ADME (i.e., absorption, distribution, metabolism, and excretion) semantic drug labeling. The experimental results show that the proposed two-stage fine-tuning outperforms vanilla fine-tuning and state-of-the-art methods on the above datasets.
Metrics
Details
- Title
- Two-Stage Fine-Tuning with ChatGPT Data Augmentation for Learning Class-Imbalanced Data
- Creators
- Hualou Liang (Corresponding Author) - Drexel University, School of Biomedical Engineering, Science, and Health SystemsTaha ValizadehAslani - Drexel UniversityYiwen Shi - Drexel UniversityJing Wang - Center for Drug Evaluation and ResearchPing Ren - Center for Drug Evaluation and ResearchYi Zhang - Center for Drug Evaluation and ResearchMeng Hu - United States Food and Drug AdministrationLiang Zhao - United States Food and Drug Administration
- Publication Details
- Neurocomputing [e-journal], v 592, 127801
- Publisher
- Elsevier
- Grants
- 75F40119C10106, United States Food and Drug Administration (United States, Silver Spring) - FDA
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Information Science; Electrical and Computer Engineering; School of Biomedical Engineering, Science, and Health Systems; College of Computing and Informatics
- Web of Science ID
- WOS:001241102100001
- Scopus ID
- 2-s2.0-85193011832
- Other Identifier
- 991021874215204721
UN Sustainable Development Goals (SDGs)
This publication has contributed to the advancement of the following goals:
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- Web of Science research areas
- Computer Science, Artificial Intelligence