Multimodal assessment of upper limb rehabilitation in stroke patients based on cross-attention mechanism

Stroke, characterized by high incidence, disability, and mortality rates, often causes brain damage and upper limb impairment, making objective, efficient, and personalized rehabilitation assessment crucial. This study developed a multimodal system for assessing upper limb motor function in stroke patients. First, a markerless motion capture system using the Azure Kinect DK was developed and validated against a gold standard NOKOV Motion Capture, achieving high accuracy (joint angle MAE < 6°). Second, a multi-modal feature set integrating kinematics and electromyography (EMG) data was constructed based on a customized Short-FMA test protocol. Third, an improved Res-Transformer model was proposed, demonstrating superior efficiency (40% faster training) and higher accuracy (avg. 0.807–0.811 for single modalities) compared to Transformer, LSTM, and TCN. Finally, a multimodal fusion model (ReT-CA), integrating Res-Transformer with a cross-attention mechanism, achieved a high classification accuracy (average 0.892). This model formed the core of a digital assessment system, featuring clinical score prediction, visualization, and report generation. The system offers a low-cost, accurate solution for objective stroke rehabilitation assessment, with the potential to be widely adopted in communities, hospitals, and other use scenarios.