Utama, P.A., Moosavi, N.S. orcid.org/0000-0002-8332-307X, Sanh, V. et al. (1 more author) (2021) Avoiding inference heuristics in few-shot prompt-based finetuning. In: Moens, M.-F., Huang, X., Specia, L. and Yih, S. W.-T., (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), 07-11 Nov 2021, Punta Cana, Dominican Republic (Online). Association for Computational Linguistics Note: , pp. 9063-9074. ISBN 9781955917094
Abstract
Recent prompt-based approaches allow pretrained language models to achieve strong performances on few-shot finetuning by reformulating downstream tasks as a language modeling problem. In this work, we demonstrate that, despite its advantages on low data regimes, finetuned prompt-based models for sentence pair classification tasks still suffer from a common pitfall of adopting inference heuristics based on lexical overlap, e.g., models incorrectly assuming a sentence pair is of the same meaning because they consist of the same set of words. Interestingly, we find that this particular inference heuristic is significantly less present in the zero-shot evaluation of the prompt-based model, indicating how finetuning can be destructive to useful knowledge learned during the pretraining. We then show that adding a regularization that preserves pretraining weights is effective in mitigating this destructive tendency of few-shot finetuning. Our evaluation on three datasets demonstrates promising improvements on the three corresponding challenge datasets used to diagnose the inference heuristics.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2021 Association for Computational Linguistics. Available under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 07 Sep 2022 15:32 |
Last Modified: | 07 Sep 2022 15:32 |
Status: | Published |
Publisher: | Association for Computational Linguistics Note: |
Refereed: | Yes |
Identification Number: | 10.18653/v1/2021.emnlp-main.713 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:190597 |