Conference proceeding
Multimodal Deep Learning using Images and Text for Information Graphic Classification
ASSETS'18: PROCEEDINGS OF THE 20TH INTERNATIONAL ACM SIGACCESS CONFERENCE ON COMPUTERS AND ACCESSIBILITY, pp 143-148
01 Jan 2018
Abstract
Information graphics, e.g. line or bar graphs, are often displayed in documents and popular media to support an intended message, but for a growing number of people, they are missing the point. The World Health Organization estimates that the number of people with vision impairment could triple in the next thirty years due to population growth and aging. If a graphic is not described, explained in the text, or missing alt tags and other metadata (as is often the case in popular media), the intended message is lost or not adequately conveyed. In this work, we describe a multimodal deep learning approach that supports the communication of the intended message. The multimodal model uses both the pixel data and text data in a single neural network to classify the information graphic into an intention category that has previously been validated as useful for people who are blind or who are visually impaired. Furthermore, we collect a new dataset of information graphics and present qualitative and quantitative results that show our multimodal model exceeds the performance of any one modality alone, and even surpasses the capabilities of the average human annotator.
Metrics
Details
- Title
- Multimodal Deep Learning using Images and Text for Information Graphic Classification
- Creators
- Edward Kim - Villanova UniversityKathleen F. McCoy - University of Delaware
- Publication Details
- ASSETS'18: PROCEEDINGS OF THE 20TH INTERNATIONAL ACM SIGACCESS CONFERENCE ON COMPUTERS AND ACCESSIBILITY, pp 143-148
- Publisher
- Assoc Computing Machinery
- Number of pages
- 6
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Computer Science (Computing)
- Web of Science ID
- WOS:000455178500015
- Scopus ID
- 2-s2.0-85056848824
- Other Identifier
- 991021884689904721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- Web of Science research areas
- Computer Science, Hardware & Architecture
- Computer Science, Interdisciplinary Applications
- Computer Science, Software Engineering