Salle, A. and Villavicencio, A. orcid.org/0000-0002-3731-9168 (2018) Restricted recurrent neural tensor networks: Exploiting word frequency and compositionality. In: Gurevych, I. and Miyao, Y., (eds.) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers). 56th Annual Meeting of the Association for Computational Linguistics, 15-20 Jul 2018, Melbourne, Australia. Association for Computational Linguistics , pp. 8-13. ISBN 9781948087346
Abstract
Increasing the capacity of recurrent neural networks (RNN) usually involves augmenting the size of the hidden layer, with significant increase of computational cost. Recurrent neural tensor networks (RNTN) increase capacity using distinct hidden layer weights for each word, but with greater costs in memory usage. In this paper, we introduce restricted recurrent neural tensor networks (r-RNTN) which reserve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights for infrequent words. Perplexity evaluations show that for fixed hidden layer sizes, r-RNTNs improve language model performance over RNNs using only a small fraction of the parameters of unrestricted RNTNs. These results hold for r-RNTNs using Gated Recurrent Units and Long Short-Term Memory.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2018 Association for Computational Linguistics |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 21 Nov 2019 15:51 |
Last Modified: | 21 Nov 2019 16:33 |
Status: | Published |
Publisher: | Association for Computational Linguistics |
Refereed: | Yes |
Identification Number: | 10.18653/v1/P18-2002 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:153558 |