A two-step attention-based feature combination cross-attention system for speech-based dementia detection

Dementia poses a significant global challenge, with profound personal, societal, and economic impacts. Although it is incurable, early detection is crucial for ensuring appropriate care and support. Dementia can impair a person's speech and language abilities, and studies have demonstrated promising results in using spoken language for automatic dementia detection. Recently, deep learning-based self-supervised learning models, such as wav2vec2.0 (w2v) and BERT, have shown success in extracting acoustic and linguistic information. However, most studies have relied on single datasets and relatively straightforward methods for extracting and combining acoustic and linguistic modalities. This paper presents an in-depth exploration of the application of SSL models in this context by proposing the Two-Step Attention-based Feature Combination Cross-attention system (TSAC-ATT) for speech-based dementia detection. The contributions of this paper are as follows: i) we explore and analyse acoustic and linguistic feature extraction pipelines using SSL models, including the proposed TSAC framework to create high-performing acoustic features from w2v's contextual layers; ii) we demonstrate that these features, when fused using cross-attention, outperform various feature combination approaches; iii) all experimental work is conducted on two publicly available datasets (DementiaBank and ADReSS), as well as the IVA dataset collected by the Royal Hallamshire Hospital, which includes recordings of the standard Cookie Theft task. We present state-of-the-art results, highlighting that acoustic-only features based on the w2v model can achieve very high performance across multiple datasets. Furthermore, we show that the upstream performance of the automatic speech recognition module does not always predict downstream classification performance.