Li, R., Li, X., Chen, G. et al. (1 more author) (2020) Improving variational autoencoder for text modelling with timestep-wise regularisation. In: Scott, D., Bel, N. and Zong, C., (eds.) Proceedings of the 28th International Conference on Computational Linguistics. 28th International Conference on Computational Linguistics, 08-13 Dec 2020, Barcelona, Spain (Online). Association for Computational Linguistics (ACL) , pp. 2381-2397. ISBN 9781952148279
Abstract
The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2020 The Authors. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 13 Aug 2021 08:48 |
Last Modified: | 13 Aug 2021 08:48 |
Published Version: | https://aclanthology.org/2020.coling-main.216 |
Status: | Published |
Publisher: | Association for Computational Linguistics (ACL) |
Refereed: | Yes |
Identification Number: | 10.18653/v1/2020.coling-main.216 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:177035 |