Khan, M., AlHarbi, N. and Gotoh, Y. (2015) A framework for creating natural language descriptions of video streams. Information Sciences, 303. 61 - 82. ISSN 1872-6291
Abstract
This contribution addresses generation of natural language descriptions for important visual content present in video streams. The work starts with implementation of conventional image processing techniques to extract high-level visual features such as humans and their activities. These features are converted into natural language descriptions using a template-based approach built on a context free grammar, incorporating spatial and temporal information. The task is challenging particularly because feature extraction processes are erroneous at various levels. In this paper we explore approaches to accommodating potentially missing information, thus creating a coherent description. Sample automatic annotations are created for video clips presenting humans’ close-ups and actions, and qualitative analysis of the approach is made from various aspects. Additionally a task-based scheme is introduced that provides quantitative evaluation for relevance of generated descriptions. Further, to show the framework’s potential for extension, a scalability study is conducted using video categories that are not targeted during the development.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2015 Elsevier. This is an author produced version of a paper subsequently published in Information Sciences. Uploaded in accordance with the publisher's self-archiving policy. |
Keywords: | Video retrieval; Video annotation; Natural language generation |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 30 Jun 2015 15:00 |
Last Modified: | 17 May 2016 12:52 |
Published Version: | http://dx.doi.org/10.1016/j.ins.2014.12.034 |
Status: | Published |
Publisher: | Elsevier |
Identification Number: | 10.1016/j.ins.2014.12.034 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:87414 |