Wang, J. orcid.org/0000-0003-0048-3893 and Gaizauskas, R. (2016) Don't mention the shoe! A learning to rank approach to content selection for image description generation. In: Proceedings of the 9th International Natural Language Generation conference. International Natural Language Generation Conference (INLG 2016), 05-08 Sep 2016, Edinburgh, Scotland. Association for Computational Linguistics (ACL) , pp. 193-202.
Abstract
We tackle the sub-task of content selection as part of the broader challenge of automatically generating image descriptions. More specifically, we explore how decisions can be made to select what object instances should be mentioned in an image description, given an image and labelled bounding boxes. We propose casting the content selection problem as a learning to rank problem, where object instances that are most likely to be mentioned by humans when describing an image are ranked higher than those that are less likely to be mentioned. Several features are explored: those derived from bounding box localisations, from concept labels, and from image regions. Object instances are then selected based on the ranked list, where we investigate several methods for choosing a stopping criterion as the ‘cut-off’ point for objects in the ranked list. Our best-performing method achieves state-of-the-art performance on the ImageCLEF2015 sentence generation challenge.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2020 ACL. This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number Engineering and Physical Science Research Council EP/K019082/1 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 16 Aug 2016 15:44 |
Last Modified: | 19 Jun 2020 14:03 |
Published Version: | https://www.aclweb.org/anthology/W16-6631 |
Status: | Published |
Publisher: | Association for Computational Linguistics (ACL) |
Refereed: | Yes |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:103421 |