A generalisation study in deep learning-based segmentation of lower-limb muscles across different populations

Accurate and consistent segmentation of lower-limb muscles across different populations (e.g., children, young adults, and older individuals) remains challenging due to substantial anatomical differences. This study evaluated the performance of deep learning models for the automatic segmentation of lower-limb muscles in typically developed children (TDC). We present a novel investigation into their generalization ability across different cohorts (healthy young people (HY) and post-menopausal women (PMW)). Our focus was on the Attention-Feature-Fusion-Unet (AFFU) model, which incorporates a feature fusion module into U-Net. First, manual segmentation of T1-weighted images from TDC cohort was conducted by different operators and a reproducibility analysis was evaluated. Then a comparison study was carried out with UNet, UNet +  + , and Attention UNet. The model AFFU achieved the best Dice Similarity Coefficient 0.86 and Relative Volume Error 0.09 on children cohort. It also significantly reduced the Hausdorff Distance and Average Symmetric Surface Distance by approximately 34% and 20%, respectively, compared to the baseline U-Net (p < 0.01). It can be observed that larger, regularly shaped muscles achieved higher segmentation accuracy, while smaller and irregular muscles posed difficulties. The experiment showed that a single type of cohort model training is not enough to improve the model generalisation ability. The best results in terms of generalisation were achieved with a training set of mixed multi-class cohorts and a complex model using attention mechanisms.