Al Harbi, N. and Gotoh, Y. (2017) Natural language descriptions for human activities in video streams. In: Alonso, J.M., Bugarin, A. and Reiter, E., (eds.) Proceedings of the 10th International Conference on Natural Language Generation. 10th International Conference on Natural Language Generation (INLG2017), 04-07 Sep 2017, Santiago de Compostela, Spain. Association for Computational Linguistics (ACL) , pp. 85-94. ISBN 9781945626524
Abstract
There has been continuous growth in the volume and ubiquity of video material. It has become essential to define video semantics in order to aid the searchability and retrieval of this data. We present a framework that produces textual descriptions of video, based on the visual semantic content. Detected action classes rendered as verbs, participant objects converted to noun phrases, visual properties of detected objects rendered as adjectives and spatial relations between objects rendered as prepositions. Further, in cases of zero-shot action recognition, a language model is used to infer a missing verb, aided by the detection of objects and scene settings. These extracted features are converted into textual descriptions using a template-based approach. The proposed video descriptions framework evaluated on the NLDHA dataset using ROUGE scores and human judgment evaluation.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © 2017 Association for Computational Linguistics. Article available under the terms of the Creative Commons Attribution Licence (http://creativecommons.org/licenses/by/4.0). |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 27 Jul 2017 09:56 |
Last Modified: | 23 Jun 2020 13:12 |
Published Version: | https://www.aclweb.org/anthology/W17-3512 |
Status: | Published |
Publisher: | Association for Computational Linguistics (ACL) |
Refereed: | Yes |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:119548 |