Multimodal learning for predicting mortality in patients with pulmonary arterial hypertension

Pulmonary Arterial Hypertension (PAH) is a lifethreatening disorder. The prediction of mortality in PAH patients can play a crucial role in the clinical management of this disease. The prediction of mortality from one modality is a difficult task that may only provide limited performance. Therefore, we propose a multimodal learning approach in this work to predict one-year mortality in PAH patients. We have utilised three modalities, which include extracted numerical imaging features, echo report categorical features, and echo report text features from Electronic Health Records (EHRs) of patients. We have proposed a feature integration module to combine features from multiple modalities. The text features have been extracted from the echo reports using the Bidirectional Encoder Representations from Transformers (BERT). An attention mechanism and a weighted summation method are also adopted during the process of feature integration. We have performed different experiments to evaluate the performance of the proposed framework for mortality prediction. The experimental results indicate that we can achieve the best AUC score of 0.89 for predicting one-year mortality by combining all three modalities. The source code of this paper is available at https://github.com/Mdnaimulislam/MultimodalTab.