Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
Unveiling Glitches: A Deep Dive into Image Encoding Bugs within CLIP
30 Jun 2024
Abstract
Understanding the limitations and weaknesses of state-of-the-art models in
artificial intelligence is crucial for their improvement and responsible
application. In this research, we focus on CLIP, a model renowned for its
integration of vision and language processing. Our objective is to uncover
recurring problems and blind spots in CLIP's image comprehension. By delving
into both the commonalities and disparities between CLIP and human image
understanding, we augment our comprehension of these models' capabilities.
Through our analysis, we reveal significant discrepancies in CLIP's
interpretation of images compared to human perception, shedding light on areas
requiring improvement. Our methodologies, the Discrepancy Analysis Framework
(DAF) and the Transformative Caption Analysis for CLIP (TCAC), enable a
comprehensive evaluation of CLIP's performance. We identify 14 systemic faults,
including Action vs. Stillness confusion, Failure to identify the direction of
movement or positioning of objects in the image, Hallucination of Water-like
Features, Misattribution of Geographic Context, among others. By addressing
these limitations, we lay the groundwork for the development of more accurate
and nuanced image embedding models, contributing to advancements in artificial
intelligence.
Metrics
7 Record Views
Details
- Title
- Unveiling Glitches: A Deep Dive into Image Encoding Bugs within CLIP
- Creators
- Ayush RanjanDaniel WenKarthik Bhat
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Information Science
- Other Identifier
- 991021985096304721