Visual Memory Schemas for Localised Image Memorability Prediction

Visual memory schemas (VMS) capture the regions of scene images that cause that scene to be remembered, providing a two-dimensional memorability map which indicates the parts of a given scene that match to mental schemas held in the mind. Despite the advantage of determining which parts of an image lead to remembering said image, VMS prediction capabilities lag behind those of single-score memorability. Compared to predicting single-score ratings for the likelihood of a person remembering an image, VMS prediction is a significantly harder task, due to increased computational complexity, minimal model development compared to single score, and lack of relevant data. In this work, we aim to improve methods for two-dimensional memorability prediction. We first significantly increase the size of a database containing VMS maps obtained from participants in a scene memorisation experiment, and then we develop an architecture which leverages existing single-score image memorability datasets to predict VMS maps. Our final model, 'DFVMS' significantly outperforms existing VMS prediction models, with a performance increase of 11.8%. Additionally, we explore the semantic structures which are actually captured by visual memory schemas, determining the combination of scene elements which lead to remembering that scene.