Petrak, D., Moosavi, N.S. orcid.org/0000-0002-8332-307X and Gurevych, I. (2023) Arithmetic-based pretraining improving numeracy of pretrained language models. In: Palmer, A. and Camacho-collados, J., (eds.) Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023). The 12th Joint Conference on Lexical and Computational Semantics, 13-14 Jul 2023, Toronto, Canada. Association for Computational Linguistics , pp. 477-493. ISBN 9781959429760
Abstract
State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and Sci-Gen datasets.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2023 Association for Computational Linguistics. This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 20 Mar 2024 11:12 |
Last Modified: | 20 Mar 2024 11:12 |
Status: | Published |
Publisher: | Association for Computational Linguistics |
Identification Number: | 10.18653/v1/2023.starsem-1.42 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:210589 |