Acar, E., Hopfgartner, F. orcid.org/0000-0003-0380-6088 and Albayrak, S. (2017) A comprehensive study on mid-level representation and ensemble learning for emotional analysis of video material. Multimedia Tools and Applications, 76 (9). pp. 11809-11837. ISSN 1380-7501
Abstract
In today’s society where audio-visual content such as professionally edited and user-generated videos is ubiquitous, automatic analysis of this content is a decisive functionality. Within this context, there is an extensive ongoing research about understanding the semantics (i.e., facts) such as objects or events in videos. However, little research has been devoted to understanding the emotional content of the videos. In this paper, we address this issue and introduce a system that performs emotional content analysis of professionally edited and user-generated videos. We concentrate both on the representation and modeling aspects. Videos are represented using mid-level audio-visual features. More specifically, audio and static visual representations are automatically learned from raw data using convolutional neural networks (CNNs). In addition, dense trajectory based motion and SentiBank domain-specific features are incorporated. By means of ensemble learning and fusion mechanisms, videos are classified into one of predefined emotion categories. Results obtained on the VideoEmotion dataset and a subset of the DEAP dataset show that (1) higher level representations perform better than low-level features, (2) among audio features, mid-level learned representations perform better than mid-level handcrafted ones, (3) incorporating motion and domain-specific information leads to a notable performance gain, and (4) ensemble learning is superior to multi-class support vector machines (SVMs) for video affective content analysis.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © Springer Science+Business Media New York 2016. |
Keywords: | Video affective content analysis; Ensemble learning; Deep learning; MFCC; Color; Dense trajectories; SentiBank |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 10 Oct 2018 13:39 |
Last Modified: | 10 Oct 2018 14:02 |
Published Version: | https://doi.org/10.1007/s11042-016-3618-5 |
Status: | Published |
Publisher: | Springer Verlag |
Refereed: | Yes |
Identification Number: | 10.1007/s11042-016-3618-5 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:136946 |