This is the latest version of this eprint.
Utama, P.A., Bambrick, J., Moosavi, N.S. orcid.org/0000-0002-8332-307X et al. (1 more author) (2022) Falsesum : generating document-level NLI examples for recognizing factual inconsistency in summarization. In: Carpuat, M., de Marneffe, M.-C. and Meza Ruiz, I.V., (eds.) Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL 2022 - Annual Conference of the North American Chapter of the Association for Computational Linguistics, 10-15 Jul 2022, Seattle, WA, USA. ACL - Association for Computational Linguistics , pp. 2763-2776.
Abstract
Neural abstractive summarization models are prone to generate summaries that are factually inconsistent with their source documents. Previous work has introduced the task of recognizing such factual inconsistency as a downstream application of natural language inference (NLI). However, state-of-the-art NLI models perform poorly in this context due to their inability to generalize to the target task. In this work, we show that NLI models can be effective for this task when the training data is augmented with high-quality task-oriented examples. We introduce Falsesum, a data generation pipeline leveraging a controllable text generation model to perturb human-annotated summaries, introducing varying types of factual inconsistencies. Unlike previously introduced document-level NLI datasets, our generated dataset contains examples that are diverse and inconsistent yet plausible. We show that models trained on a Falsesum-augmented NLI dataset improve the state-of-the-art performance across four benchmarks for detecting factual inconsistency in summarization.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2022 Association for Computational Linguistics. Available under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 20 May 2022 09:17 |
Last Modified: | 07 Jun 2023 15:41 |
Status: | Published |
Publisher: | ACL - Association for Computational Linguistics |
Refereed: | Yes |
Identification Number: | 10.18653/v1/2022.naacl-main.199 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:186939 |
Available Versions of this Item
-
Falsesum: generating document-level NLI examples for recognizing factual inconsistency in summarization. (deposited 07 Jun 2023 15:40)
- Falsesum : generating document-level NLI examples for recognizing factual inconsistency in summarization. (deposited 20 May 2022 09:17) [Currently Displayed]