Files and links (1)
Published, Version of Record (VoR)Open Access via Drexel Libraries Read and Publish Program 2024CC BY-NC-ND V4.0, Open
Conference proceeding
Exploring ChatGPT for Toxicity Detection in GitHub
Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, pp 6-10
24 May 2024
Featured in Collection : Research Supported by Drexel Libraries' OA Programs
Abstract
Fostering a collaborative and inclusive environment is crucial for the sustained progress of open source development. However, the prevalence of negative discourse, often manifested as toxic comments, poses significant challenges to developer well-being and productivity. To identify such negativity in project communications, especially within large projects, automated toxicity detection models are necessary. To train these models effectively, we need large software engineering-specific toxicity datasets. However, such datasets are limited in availability and often exhibit imbalance (e.g., only 6 in 1000 GitHub issues are toxic) [1], posing challenges for training effective toxicity detection models. To address this problem, we explore a zero-shot LLM (ChatGPT) that is pre-trained on massive datasets but without being fine-tuned specifically for the task of detecting toxicity in software-related text. Our preliminary evaluation indicates that ChatGPT shows promise in detecting toxicity in GitHub, and warrants further investigation. We experimented with various prompts, including those designed for justifying model outputs, thereby enhancing model interpretability and paving the way for potential integration of ChatGPT-enabled toxicity detection into developer communication channels.
Metrics
Details
- Title
- Exploring ChatGPT for Toxicity Detection in GitHub
- Creators
- Shyamal Mishra - Drexel UniversityPreetha Chatterjee (Corresponding Author) - Drexel University, Computer Science
- Publication Details
- Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, pp 6-10
- Conference
- ICSE-NIER'24: 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (Lisbon, Portugal, 14 Apr 2024–20 Apr 2024)
- Series
- ACM Conferences
- Publisher
- Association for Computing Machinery
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Computer Science
- Web of Science ID
- WOS:001234856600002
- Scopus ID
- 2-s2.0-85195145335
- Other Identifier
- 991021880014804721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Web of Science research areas
- Computer Science, Artificial Intelligence
- Computer Science, Software Engineering
- Computer Science, Theory & Methods