Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
EigenNoise: A Contrastive Prior to Warm-Start Representations
arXiv (Cornell University)
09 May 2022
Abstract
In this work, we present a naive initialization scheme for word vectors based
on a dense, independent co-occurrence model and provide preliminary results
that suggest it is competitive and warrants further investigation.
Specifically, we demonstrate through information-theoretic minimum description
length (MDL) probing that our model, EigenNoise, can approach the performance
of empirically trained GloVe despite the lack of any pre-training data (in the
case of EigenNoise). We present these preliminary results with interest to set
the stage for further investigations into how this competitive initialization
works without pre-training data, as well as to invite the exploration of more
intelligent initialization schemes informed by the theory of harmonic
linguistic structure. Our application of this theory likewise contributes a
novel (and effective) interpretation of recent discoveries which have
elucidated the underlying distributional information that linguistic
representations capture from data and contrast distributions.
Metrics
17 Record Views
Details
- Title
- EigenNoise: A Contrastive Prior to Warm-Start Representations
- Creators
- Hunter Scott HeidenreichJake Ryland Williams
- Publication Details
- arXiv (Cornell University)
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Information Science
- Other Identifier
- 991021806682704721