Deep learning-based H&E-derived risk scores in colorectal cancer: associations with tumour morphology, biology, and predicted drug response

Over recent years, several deep learning (DL) models have been presented to predict colorectal cancer (CRC) patient survival directly from haematoxylin and eosin (H&E)-stained routine whole-slide images (WSIs). Unlike traditional studies that rely on manually defined histopathological features, weakly supervised DL allows training directly on clinical endpoints without prior specification of the model's focus. This offers a unique opportunity to study the tissue morphology underlying these predictions, improving our understanding of disease biology. Here, we present a comprehensive analysis of the clinicopathological features, tumour morphology and biology, as well as gene expression-based predicted drug response of over 4,000 CRC patients derived from four different international cohorts with available H&E-inferred DL-based risk scores (low- versus high-risk as well as absolute risk scores). The results from our study suggest that conventional clinicopathological risk factors, such as grade of differentiation, presence of lymph node metastasis, tumour budding, and percentage of tumour necrosis, are positively associated with DL-based risk scores. Moreover, CRCs with direct tumour–adipocyte interactions are enriched in the DL-based high-risk group. Through detailed morphologic review, we provide comprehensive evidence that direct tumour–adipocyte interaction, a high degree of tumour budding, and poorly differentiated morphology are linked to high DL-based risk scores. Transcriptomic and genetic subgroups show only limited association with H&E-derived DL-based risk scores. Moreover, we present data suggesting that DL-based low- versus high-risk CRCs may be characterised by differential drug sensitivity. Our study highlights that DL-based risk scores derived from H&E WSIs not only align with established clinicopathological features but also highlight morphological features, such as tumour–adipocyte interaction, that are not routinely captured by established clinicopathological scoring systems. Moreover, DL-based risk groups may be associated with a differential treatment response, underlining their potential to guide patient stratification in routine clinical practice.