Cross-scale Multi-instance Learning for Pathological Image Diagnosis
Ruining Deng Can Cui Lucas W. Remedios Shunxing Bao R. Michael Womick Sophie Chiron Jia Li Joseph T. Roland Ken S. Lau Qi Liu
Analyzing high resolution whole slide images (WSIs) with regard to information across multiple scales poses a significant challenge in digital pathology. Multi-instance learning (MIL) is a common solution for working with high resolution images by classifying ...
https://www.ncbi.nlm.nih.gov/coreutils/nwds/img/favicons/favicon.ico
PubMed Central (PMC)
Med Image Anal. Author manuscript; available in PMC 2024 May 1.

Published in final edited form as:

PMCID: PMC11016375

NIHMSID: NIHMS1973298

PMID: 38428271

Cross-scale Multi-instance Learning for Pathological Image Diagnosis

,a ,a ,a ,a ,b ,c ,c ,c ,a ,c ,c,d ,c ,c,d ,a,c and a,*

Ruining Deng

aVanderbilt University, Nashville, TN 37215, USA

Find articles by Ruining Deng

Can Cui

aVanderbilt University, Nashville, TN 37215, USA

Find articles by Can Cui

Lucas W. Remedios

aVanderbilt University, Nashville, TN 37215, USA

Find articles by Lucas W. Remedios

Shunxing Bao

aVanderbilt University, Nashville, TN 37215, USA

Find articles by Shunxing Bao

R. Michael Womick

bThe University of North Carolina at Chapel Hill, Chapel Hill, NC 27514, USA

Find articles by R. Michael Womick

Sophie Chiron

cVanderbilt University Medical Center, Nashville, TN 37232, USA

Find articles by Sophie Chiron

Jia Li

cVanderbilt University Medical Center, Nashville, TN 37232, USA

Find articles by Jia Li

Joseph T. Roland

cVanderbilt University Medical Center, Nashville, TN 37232, USA

Find articles by Joseph T. Roland

Ken S. Lau

aVanderbilt University, Nashville, TN 37215, USA

Find articles by Ken S. Lau

Qi Liu

cVanderbilt University Medical Center, Nashville, TN 37232, USA

Find articles by Qi Liu

Keith T. Wilson

cVanderbilt University Medical Center, Nashville, TN 37232, USA

dVeterans Affairs Tennessee Valley Healthcare System, Nashville, TN 37212, USA

Find articles by Keith T. Wilson

Yaohong Wang

cVanderbilt University Medical Center, Nashville, TN 37232, USA

Find articles by Yaohong Wang

Lori A. Coburn

cVanderbilt University Medical Center, Nashville, TN 37232, USA

dVeterans Affairs Tennessee Valley Healthcare System, Nashville, TN 37212, USA

Find articles by Lori A. Coburn

Bennett A. Landman

aVanderbilt University, Nashville, TN 37215, USA

cVanderbilt University Medical Center, Nashville, TN 37232, USA

Find articles by Bennett A. Landman

Yuankai Huo

aVanderbilt University, Nashville, TN 37215, USA

Find articles by Yuankai Huo

aVanderbilt University, Nashville, TN 37215, USA

bThe University of North Carolina at Chapel Hill, Chapel Hill, NC 27514, USA

cVanderbilt University Medical Center, Nashville, TN 37232, USA

dVeterans Affairs Tennessee Valley Healthcare System, Nashville, TN 37212, USA

*Corresponding author: ude.tlibrednav@ouh.iaknauy

Abstract

Analyzing high resolution whole slide images (WSIs) with regard to information across multiple scales poses a significant challenge in digital pathology. Multi-instance learning (MIL) is a common solution for working with high resolution images by classifying bags of objects (i.e. sets of smaller image patches). However, such processing is typically performed at a single scale (e.g., 20× magnification) of WSIs, disregarding the vital inter-scale information that is key to diagnoses by human pathologists. In this study, we propose a novel cross-scale MIL algorithm to explicitly aggregate inter-scale relationships into a single MIL network for pathological image diagnosis. The contribution of this paper is three-fold: (1) A novel cross-scale MIL (CS-MIL) algorithm that integrates the multi-scale information and the inter-scale relationships is proposed; (2) A toy dataset with scale-specific morphological features is created and released to examine and visualize differential cross-scale attention; (3) Superior performance on both in-house and public datasets is demonstrated by our simple cross-scale MIL strategy. The official implementation is publicly available at https://github.com/hrlblab/CS-MIL.

Keywords: Multi-instance Learning, Multi-scale, Attention Mechanism, Pathology

1. Introduction

Pathology is a gold standard to diagnose inflammatory bowel disease (e.g., Crohn’s disease) (Gubatan et al., 2021; Yeshi et al., 2020). In the current clinical practice, pathologists examine morphological patterns at multiple scales through microscopes (Bejnordi et al., 2017), which is a laborious process. With the rapid advancements in whole slide imaging and deep learning techniques, the potential for computer-assisted clinical diagnosis and exploration in digital pathology (Kraszewski et al., 2021; Con et al., 2021; Kiyokawa et al., 2022; Syed and Stidham, 2020) is rapidly increasing, making it a promising field of study. However, annotating images pixel- or patch-wise is computationally expensive for a standard supervised deep learning system (Hou et al., 2016; Mousavi et al., 2015; Maksoud et al., 2020; Dimitriou et al., 2019). In order to obtain accurate diagnoses from images with weak annotations (e.g., patient-wise diagnosis), multi-instance Learning (MIL) has emerged as a popular weakly supervised learning paradigm for tasks in digital pathology (Wang et al., 2019; Skrede et al., 2020; Chen et al., 2021; Lu et al., 2021b,a). For example, DeepAttnMISL (Yao et al., 2020) used MIL to cluster image patches into different “bags” to model and aggregate diverse local features for patient-level diagnosis.

Despite the success, prior efforts–especially “natural image driven” MIL algorithms–largely overlook the multi-scale (i.e. pyramidal) nature of WSIs, which can consist of scales from, thereby allowing pathologists to examine both local and global morphological features (Bejnordi et al., 2015; Gao et al., 2016; Tokunaga et al., 2019). Recent efforts have been made to mimic human pathological assessments by using multi-scale images in a WSI (Hashimoto et al., 2020a; Li et al., 2021). These methods typically extract features independently at each scale and then perform a “late fusion” step. In this study, we examine the feasibility of introducing interaction between different scales at an earlier stage as an attention-based “early fusion” paradigm.

Different from the current “multi-scale” MIL strategy, we propose a novel “cross-scale” attention mechanism. The key innovation is to introduce an attention-guided MIL scheme to explicitly model inter-scale interactions during feature extraction ( ). The proposed method not only utilizes the morphological features at different scales (with different fields of view), but also learns their inter-scale interactions as an “early fusion” learning paradigm. Through empirical validation, our cross-scale MIL approach achieves higher Area under the Curve (AUC) scores and Average Precision (AP) scores compared with other multi-scale MIL benchmarks. The study is built upon our earlier work (Deng et al., 2022), with a more comprehensive and detailed methodological illustration, a newly released toy dataset, and new validation via a public dataset.

The contribution of this study is three-fold: (1) A novel cross-scale MIL (CS-MIL) algorithm is proposed to explicitly model the inter-scale relationships during feature extraction; (2) A toy dataset with scale-specific morphological features to examine and visualize differential cross-scale attention; (3) Superior performance on both in-house and public datasets is demonstrated by our simple cross-scale MIL strategy. The code has been made publicly available at https://github.com/hrlblab/CS-MIL.

This work extends our conference paper (Deng et al., 2022) with the following new efforts and contributions: (1) new experiments are conducted, with three more datasets (one TCGA dataset and two toy datasets) and more recent benchmark methods, for a more rigorous evaluation; (2) a more comprehensive and detailed introduction and illustration of the proposed method are presented in this paper; (3) we release of an end-to-end Docker container for pathological image analysis, facilitating the effective reproduction of the results.

2. Related Works

2.1. Multi-instance Learning in Digital Pathology

In the realm of clinical digital pathology, disease-related tissue regions may be confined to a relatively small fraction of the entire tissue sample, giving rise to a substantial number of disease-free patches. Pathologists meticulously examine tissues at various magnifications utilizing microscopes to detect the disease-related regions and subsequently scrutinize morphological patterns. Nevertheless, the patch-level annotation of disease-related regions by skilled pathologists is a laborious task that poses challenges in scaling to gigapixel large-scale images. To address this challenge, several recent studies (Hou et al., 2016; Campanella et al., 2019; Hashimoto et al., 2020b; Wang et al., 2019; Skrede et al., 2020; Lu et al., 2021b,a) have demonstrated the promise of weakly supervised technology, multi-instance Learning (MIL) – a widely used weakly supervised learning paradigm–on patch-level analysis, wherein a patch-based classifier (e.g., patient-wise diagnosis) is trained solely on slide-level labels.

Within the context of MIL, every Whole Slide Image (WSI) is treated as a bag that comprises numerous instances of patches. A WSI bag is marked as disease-relevant if any of its patches (i.e., instances) exhibit disease-related characteristics (e.g., lesions, tumors, abnormal tissues). The classifier refines, extracts, and aggregates patch-level features or scores to anticipate slide-level labels (Li et al., 2021). Recent MIL based approaches have greatly benefited from using deep neural networks for feature extraction and aggregation (Ilse et al., 2018; Wang et al., 2016; Oquab et al., 2015). For example, Yao et al., (Yao et al., 2020) utilized a bag-level approach where image patches were clustered into distinct “bags” to model and aggregate diverse local features for patient-level diagnosis. In a similar vein, Hou et al., (Hou et al., 2016) proposed a decision fusion model that aggregated patch-level predictions generated by patch-level CNNs. Hashimoto et al., (Hashimoto et al., 2020b) proposed a novel CNN-based technique for cancer subtype classification by effectively merging multiple-instance, domain adversarial, and multi-scale learning frameworks at the patch level.

2.2. Multi-scale in Digital Pathology

Digital pathology works with pyramidally structured gigapixel images. Different resolutions present different levels of heterogeneous structural patterns on tissue samples. Therefore, pathologists are required to carefully examine biopsies at multiple scales through digital pathology to capture morphological patterns for disease diagnosis (Gordon et al., 2020). This process is labor-intensive and causes a loss of spatial correlation with sequential zoom-in/zoom-out operations. Using AI models to analyze images at multiple scales not only improves model performance by using scale-aware knowledge but also makes use of inter-scale relationships with spatial consistency learned by the model.

Previous studies have considered morphological features at multiple scales. Hashimoto et al., (Hashimoto et al., 2020a) proposed an innovative CNN-based method for cancer subtype classification, effectively integrating multiple-instance, domain adversarial, and multi-scale learning frameworks to combine knowledge from different scales. Li et al., (Li et al., 2021) employed a feature concatenation strategy, where high-level features of each region from different scales were merged to incorporate cross-scale morphological patterns obtained from a CNN feature extractor. Barbano et al., (Barbano et al., 2021a) proposed a multi-resolution approach for dysplasia grading.

Vision Transformers (ViTs) have emerged as a promising approach for feature learning from large-scale images, owing to their ability to leverage locational attention. Chen et al. (Chen et al., 2022) recently proposed a novel ViT architecture that exploits the inherent hierarchical structure of WSIs using two levels of self-supervised learning to learn high-resolution image representations. However, none of those methods holistically learn knowledge from multiple scales, that is, regarding inter-scale relationships. To address this limitation, we propose an attention-based “early fusion” paradigm that offers a promising approach for modeling inter-scale relationships at an early stage.

3. Methods

The overall pipeline of the proposed CS-MIL is presented in . We propose a novel attention-based “early fusion” paradigm that aims to capture inter-scale relationships in a holistic manner. First, patches with similar center coordinates but from different scales are jointly tiled from the WSIs. Then, patch-wise phenotype features are extracted using a self-supervised model. Local feature-based clustering is applied to each WSI, which distributes the phenotype patterns into each MIL bag. Next, cross-scale attention-guided MIL is performed to aggregate features across multi-scale and multi-clustered settings. Finally, a cross-scale attention map is generated for human visual examination.

3.1. Feature embedding and phenotype clustering

In the MIL community, the majority of histopathological image analysis methods are divided into two stages (Schirris et al., 2021; Dehaene et al., 2020): (1) the self-supervised feature embedding stage, and (2) the weakly supervised feature-based learning stage. Our approach follows a similar design by utilizing our dataset to train a contrastive-learning model, SimSiam (Chen and He, 2021), as a phenotype encoder Es to extract high-level phenotype features Fs from patches Is, as shown in equation 1. SimSiam has demonstrated superior feature extraction performance compared to other backbones by maximizing the intra-sample similarity between different image augmentations without any labels.

Fs=EsIs,s∈(1,S)

(1)

where S is the number of the scales on WSIs. Three pretrained encoders Es were trained by patches from different scales, respectively. This self-supervised learning stage is crucial for effective feature extraction before the subsequent weakly supervised feature-based learning stage. All of the patches were then embedded into low-dimensional feature vectors for the classification in the second stage.

Inspired by (Yao et al., 2020), k-means clustering is used to cluster patches on the patient level based on their self-supervised embeddings at 20× magnification from the first stage. It is noted that high-level features are more comprehensive than low-resolution thumbnail images in representing phenotypes (Zhu et al., 2017). The patches were gathered equally from different clusters in each bag and then the bag with better generalization for the MIL model is organized by distinctive phenotype patterns sparsely distributed on WSIs. On the other hand, patches with similar high-level features is aggregated for classification without spatial limitation.

3.2. Cross-scale attention mechanism

Our approach builds upon previous work in MIL-related literature by incorporating a cross-scale attention mechanism that captures patterns across scale in whole-slide images (WSIs). Specifically, we utilize an CNN-based encoder to refine patch embeddings from corresponding phenotype clusters. The instance-wise features are then aggregated to achieve patient-wise classification, resulting in superior performance on survival prediction with WSIs. While attention mechanisms have been proposed in previous work to enhance the models’ use of patterns across spatial locations in WSIs (Ilse et al., 2018; Lu et al., 2021b), they do not take advantage of patterns across scale in WSIs. Other approaches have aggregated multi-scale features into deep learning models from WSIs (Hashimoto et al., 2020a; Li et al., 2021), but have demonstrated limitations in their ability to leverage the interplay between multiple resolutions within the same location.

To address this issue, we propose a novel cross-scale attention mechanism to represent awareness at different scales in the backbone. Firstly, the embedding cross-scale features fs from phenotype encoders Es are further processed among different scales by a multi-scale encoder (EMS with a siamese Multiple Instance Fully Convolutional Networks (MI-FCN) from DeepAttnMISL (Yao et al., 2020) in 2:

fs=EMSFs,s∈(1,S)

(2)

where S is the number of the scales on WSIs. All multi-scale encoders EMS are weight-shared among the different scales.

Next, a cross-scale attention mechanism is applied to consider the importance of each scale in the cross-scale attention within the same location. Cross-scale features fs are simultaneously fed into a cross-scale multi-instance learning network (CS-MIL), which contains two fully convolutional layers with kernel size of 1×1, and an ReLU activation function. The output of CS-MIL is the set of cross-scale attention scores as by considering cross-scale features as a whole. This is achieved using Equation 3:

as=expWTReLU(VfsT)∑s=1SexpWTReLU(VfsT)

(3)

where W∈RL×1 and V∈RL×M are trainable parameters in the CS-MIL, L is the size of the EMS output fs,M is the output channel of the first layer of CS-MIL, tanh(.) is the tangent element-wise non-linear activation function, and S is the number of the scales on WSIs.

The cross-scale attention scores as are then multiplied with cross-scale features, resulting in a fused cross-scale representation (demonstrated in Equation 4):

Fcs=∑s=1Sasfs

(4)

Finally, the attention-based instance-level pooling operator (C) from (Yao et al., 2020) is deployed to achieve patient-wise classification with cross-scale embedding in 5, with a bag size of n.

Ypred=CFcs1,Fcs2…,Fcsn

(5)

3.3. Cross-scale attention visualisation

The cross-scale attention mechanism produces attention scores as for each region Is based on cross-scale features fs from the CS-MIL. These attention scores reflect the relative importance of phenotype features at different scales for diagnosis when fusing the cross-scale representation (Fcs) for final classification (C). By filling these scores back to the corresponding location on WSIs, we obtain an attention map As that combines scale and location information. This map provides insights for disease-guided exploration in various contexts, highlighting the versatility and practicality of the cross-scale mechanism.

4. Experiments

4.1. Data

In-house CD dataset:

50 H&E-stained Ascending Colon (AC) Diseased biopsies from (Bao et al., 2021) were collected from 20 CD patients along with 30 healthy controls for training. The stained tissues were scanned at 20× magnification. For the pathological diagnosis, the 20 slides from CD patients were scored as normal, quiescent, mild, moderate, or severe, while the remaining tissue slides from healthy controls were scored as normal. 116 AC biopsies were stained and scanned for testing with the same procedure as the above training set. The biopsies were acquired from 72 CD patients who have no overlap with the patients in the training data.

TCGA-GBMLGG dataset:

To demonstrate the generalizebility of our proposed architecture, we conduct experiments on a glioma dataset (GBMLGG) obtained from The Cancer Genome Atlas (TCGA). The dataset contains 613 patient samples, of which 330 patients have Isocitrate dehydrogenase (IDH) mutations, while the remaining patients are normal.

4.2. Experimental setting

In-house CD dataset:

All WSIs from the two datasets were cropped into regions with the size of 4096 × 4096 pixels to fairly compare the performance between MIL methods and the ViT method in 20×. For 20× patches, each pixel is equal to 0.5 Microns. Then, 256×256 pixels patches were tiled at three scales (20×, 10× and 5×) for those regions. Three individual models following the official SimSiam model with a ResNet-50 backbone were trained at three different scales with all of the patches (504,444 256 × 256 foreground patches).The training was conducted in 200 epochs with a batch size of 128 with the official settings of the SimSiam. 2048-channel embedding vectors were obtained for all patches. Phenotype clustering was performed within the single-scale features at three resolutions using k-means clustering with a class number of 8, and cross-scale features were generated that included all resolutions for each patient. For feature extraction with HIPT (Chen et al., 2022), the official pre-training implementation was used with 1650 4096 × 4096 regions..

The training dataset was randomly organized into 10 data splits using a “leave-one-out” strategy, while the testing dataset was divided into 10 splits with balanced numbers accordingly. MIL models were used to collect each bag for every patient, with an equal selection of different phenotype clustering classes marked with a slide-wise label (Yslide) from clinicians. The hyper-parameters for training were consistent with those of DeepAttnMISL (Yao et al., 2020). The Negative Log-Likelihood Loss (Yao et al., 2019) was employed to compare the slide-wise prediction Ypred for the bag with the weakly label in 6.

L(θ)=-logpYslide∣Ypred;θ

(6)

where θ represents the model parameters. All the models were updated every four epochs to smoothly converge the loss and trained in 100 epochs in total.

The optimal model on each data split was selected based on the validation loss, while the mean performance across 10 data splits was used to evaluate the testing results. During the testing stage, 100 image bags were randomly generated, each with a size of 8 to cover most of the patches on each Whole Slide Image (WSI), and the mean value of bag scores was calculated as the final prediction at the slide level. The performance of each model was estimated using Receiver Operating Characteristic (ROC) curves with Area under the Curve (AUC) scores, Precision-Recall (PR) curves with Average Precision (AP) scores, and classification accuracy. All models were trained on an NVIDIA RTX5000 GPU.

TCGA-GBMLGG dataset:

Due to computational constraints, a 10% area was randomly selected from each WSI, resulting in a dataset of 5132 4096 × 4096 regions. During the pre-training stage, only 15% of these regions were used (582,666 256 × 256 foreground patches at three scales from 755 4096 × 4096 regions) for training the SimSiam model with a ResNet-50 backbone. The official pre-training parameters and hyper-parameters for HIPT were used in the testing stage, since HIPT already included the TCGA-GBMLGG dataset (with 54158 regions) in its pre-training stage. The training, validation, and testing samples were separated at a patient level with a 6:1:3 ratio. In the testing stage, 500 image bags of size 32 were randomly generated for each WSI, and the mean of the bag scores was calculated as the final prediction at the patient level. All models were trained using NVIDIA RTXA6000 GPU.

5. Results

5.1. Empirical Validation

We implemented three identical single-scale DeepAttnMISL (Yao et al., 2020) models for patches at corresponding scales. We simultaneously trained the (4) Gated Attention (GA) model (Ilse et al., 2018) and (5) DeepAttnMISL model with multi-scale patches, without differentiating scale information. Patches from multiple scales are treated as instances when processing phenotype clustering and patch selection for MIL bags. Furthermore, we adopted multiple multi-scale methods, including (6) a multi-scale feature aggregation (MS-DA-MIL) that jointly adds embedding features from the same location at different scales into each MIL bag (Hashimoto et al., 2020a); (7) a feature concatenation (DS-MIL) at different scales (Li et al., 2021); (8) A Double-Tier Feature Distillation when aggregating features from multiple scales and multiple locations (Zhang et al., 2022); (9) a Hierarchical Image Pyramid Transformer (HIPT) with self-supervised learning (Chen et al., 2022); (10) a Hierarchical Attention Guided MIL (Xiong et al., 2023) (HAG-MIL) as well as the proposed method (11) CS-MIL.

We followed above multi-scale aggregation to input phenotype features into the DeepAttnMISL backbone to evaluate the baseline multi-scale MIL models as well as our proposed method. All of the models were trained and validated within the same hyper-parameter setting and data splits.

5.1.1. Classification performance

and indicate the performance of the classification while directly applying the models on the testing dataset in the CD classification task, without retraining. also shows IDH status classification on TCGA-GBMLGG dataset. In general, the proposed CS-MIL achieved better scores in most evaluation metrics, demonstrating the benefits of the cross-scale attention that explores the inter-scale relationship at different scales in MIL.

Table 1:

MethodSettingCDTCGA-GBMLGGPatch ScaleClusteringAUCAPp-valueAUCAPp-valueDeepAttnMISL(20×) (Yao et al., 2020)Single20×0.77750.6126 * 0.74450.7750 * DeepAttnMISL(10×) (Yao et al., 2020)Single10×0.79000.7113 * 0.73450.7568 * DeepAttnMISL(5×) (Yao et al., 2020)Single5×0.82880.7314 * 0.75090.7957 * Gated Attention (Ilse et al., 2018)MultipleMultiple0.83620.7544 * 0.74400.7606 * DeepAttnMISL (Yao et al., 2020)MultipleMultiple0.84120.7701 * 0.72000.7520 * MS-DA-MIL (Hashimoto et al., 2020a)Multiple20×0.85650.8090 * 0.75620.8001 * DS-MIL (Li et al., 2021)Multiple20×0.83860.7933 * 0.76050.7998 * DTFD-MIL (Zhang et al., 2022)Multiple20×0.70460.5380 * 0.72150.7594 * HIPT (Chen et al., 2022)Multiple20×0.78630.7459 * 0.71020.7430 * HAG-MIL (Xiong et al., 2023)Multiple20×0.83740.7459 * 0.76100.8049 * CS-MIL(Ours)Multiple20× 0.8774 0.8486 Ref. 0.7737 0.8187 Ref.

5.1.2. Cross-scale attention visualisation

shows the cross-scale attention maps generated by the proposed CS-MIL on a CD WSI. The proposed CS-MIL presents distinctive importance-of-regions on WSIs at different scales, merging multi-scale and multi-region visualization. As a result, the 20× attention map highlights the chronic inflammatory infiltrates, while the 10× attention map focuses on the crypt structures. Those regions of interest interpret the discriminative regions for CD diagnosis across multiple scales.

5.2. Ablation Studies

Inspired by (Yao et al., 2020) and (Ilse et al., 2018), we explored various attention mechanism designs in MIL, utilizing different activation functions, and evaluated these designs on two datasets. We divided the CS-MIL approach into two strategies, distinguished by whether they share kernel weights during the embedding features learning process across multiple scales. As demonstrated by the classification performance in , sharing the kernel weights in the CS-MIL strategy, coupled with a ReLU activation function (Agarap, 2018), yielded better performance with higher values across various metrics. Overall, the proposed CS-MIL design consistently achieved superior performance compared to other baseline methods. The difference of AUC between each baseline method with our method (“Ref.” in the table) has been statistically evaluated using DeLong test. The significant difference (p < 0.05) are indicated as “*”, while the non-significant ones are marked as “N.S.”.

Table 2:

StrategyCDTCGA-GBMLGGLayer KernelActivation FunctionAUCAPp-valueAUCAPp-valueNon-sharingTanh0.86740.8436 * 0.75620.8054 * Non-sharingReLU0.86860.8362 * 0.75640.7908 * SharingTanh0.86520.8388 * 0.73440.7812 * SharingReLU 0.8774 0.8486 Ref. 0.7737 0.8187 Ref.

In , we further assess the effectiveness of the proposed cross-scale attention mechanism by utilizing a mean-vector and concatenation design with the CS-MIL backbone under a fixed 1:1:1 attention score setting. Additionally, we compare the classification performance when the phenotype embedding is removed from the pipeline, in order to assess the contribution of the projection. As a result, the CS-MIL pipeline equipped with the cross-scale attention mechanism and phenotype clustering during bag generation demonstrated better performance on two empirical datasets.

Table 3:

SettingCDTCGA-GBMLGGFusionClusteringAUCAPp-valueAUCAPp-valueMean-vector✓0.81470.7448 * 0.74250.7743 * Concatenation✓0.84700.7628 * 0.73620.7664 * CS-Attention (Naive)0.76920.7087 * 0.73080.7548 * CS-Attention (Ours)✓ 0.8774 0.8486 Ref. 0.7737 0.8187 Ref.

5.3. Simulation

To assess the effectiveness of the cross-scale attention mechanism, we evaluated CS-MIL using two toy datasets that represent distinct morphological patterns at different scales in digital pathology. These datasets were selected to simulate different scenarios and test the functionality of our approach.

Data:

shows the patches for training in the two datasets (Micro-anomaly dataset and Macro-anomaly dataset). The micro white crosses pattern only appear on positive patches at 20× maganification in the micro-anomaly dataset, while the macro anomaly (ellipse) is easily recognized at 5× with larger visual fields in macro-anomaly dataset. All of the patches are extracted from normal tissue samples in Unitopatho dataset (Barbano et al., 2021b). Two datasets were released to measure the generalization of the cross-scale designs for digital pathology community. The details of two toy datasets are shown in .

Table 4:

Dataset IdTraining PatchesValidation PatchesTesting RegionsTesting bags153282772105964540227901548414186

Approach:

The CS-MIL utilizes a ResNet-18 backbone to extract features from patches. Our implementation, including hyper-parameters, followed that of DeepAttnMISL (Yao et al., 2020). During testing, each patch was randomly captured 10 times into different image bags with a size of 8 to obtain multiple attention scores. The final attention score was calculated by taking the mean value of these scores.

Results:

presents the bag-level classification performance on the two toy datasets. The proposed method differentiates distinctive patterns at different scales in a stable manner. displays the cross-scale attention maps at the instance level and multiple scales. For the Micro-anomaly dataset, the instance attention successfully highlights positive regions with higher attention scores in corresponding regions at 20×. For the Macro-anomaly dataset, the instance attention locates ellipses instead of circles with higher attention scores at 5×. The box plots in the right panel illustrate the attention score distribution at different scales, attesting to the cross-scale attention mechanism’s reliability in providing scores across various scales and demonstrating its capability in localizing scale-specific knowledge. The cross-scale attentions can be further utilized to offer AI-based guidance for knowledge exploration in pathological images in future work, distinguishing it from other methods.

Table 5:

MethodMicro-anomalyMacro-anomalyMeanAUCAPAUCAPDeepAttnMISL(20×) (Yao et al., 2020)1.00001.00000.73740.90520.9107DeepAttnMISL(10×) (Yao et al., 2020)0.48980.53581.00001.00000.7564DeepAttnMISL(5×) (Yao et al., 2020)0.54080.58440.99920.99980.7811Gated Attention (Ilse et al., 2018)0.70820.80820.66250.87060.7624DeepAttnMISL (Yao et al., 2020)0.68790.79060.65470.81430.7369MS-DA-MIL (Hashimoto et al., 2020a)1.00001.00000.74680.90850.9138DS-MIL (Li et al., 2021)0.48640.53700.99190.99770.7533DTFD-MIL (Zhang et al., 2022)0.47710.53260.75630.92230.7533HAG-MIL (Xiong et al., 2023)0.64830.72380.69790.83130.7343CS-MIL(Mean-vector)0.44290.50840.98930.10410.5112CS-MIL(Concatenation)0.46380.53161.00001.00000.7538CS-MIL(Ours)0.96270.98200.99820.9994 0.9856

6. Discussion

and demonstrate that the multi-scale models performed better than the single-scale models, suggesting the usefulness of external information from multi-scale data on WSIs. The proposed CS-MIL model outperformed the other models in most evaluation metrics, highlighting the effectiveness of cross-scale attention, which holistically learns information from multiple scales and considers the cross-scale relationships in MIL.

demonstrates that the CS-MIL model locates positive regions using instance scores, while the cross-scale attention maps identifies the correct scale where distinctive patterns occur. In Macro-anomaly dataset, the regions with larger circles are highlighted more at 5×, providing further evidence that the model differentiate between shape patterns of ellipses and circles with larger visual fields.

To further investigate the efficacy of the cross-scale attention mechanism, we conducted experiments integrating cross-scale features through mean-vector and concatenation designs as maintaining a 1:1:1 ratio of attention scores. The performance was then evaluated on two real pathological datasets, showcasing the cross-scale attention mechanism’s capability for pathological image classification. These designs were also applied to the two toy datasets, each embodying distinctive morphological patterns observed in digital pathology. In , the performance of single-scale models indicates that only the micro white cross pattern could be captured at 20×, while the macro ellipse and circle were differentiated across three scales. When analyzing the micro-anomaly dataset, where the pattern is present only at a single scale, refinement strategies (MS-DAMIL, etc.) performed well in capturing target features. Conversely, concatenation strategies (DS-MIL, mean-vector, concatenation, etc.) were more effective in aggregating patterns across scales, resulting in superior performance on the macro-anomaly dataset. Distillation-based (hierarchical-based) methods (DTFD-MIL, HAG-MIL, etc.) achieve better performance when features are inheritable and region-consistent across hierarchical scales in the Macro-anomaly dataset, while failing to recognize scale-specific patterns lacking consistency across different scales in the Micro-anomaly dataset. The results demonstrate that the proposed cross-scale attention mechanism is both efficient and flexible, enabling improved pattern localization for scale-specific knowledge and enhanced pattern integration for hierarchical knowledge across different scales. These findings underscore the versatility of our proposed cross-scale attention mechanism in addressing different morphological patterns in digital pathology.

The parameter numbers and GPU memory usages of the state-of-the-art models in different multi-scale schemas are reported in . The proposed method achieved better classification performance by only adding lower computational complexity.

Table 6:

MethodParametersGPU MemDeepAttnMISL(5×) (Yao et al., 2020)135,3621513 MiBMS-DA-MIL (Hashimoto et al., 2020a)135,3621601 MiBHAG-MIL (Xiong et al., 2023)406,0861397 MiBCS-MIL(Ours)137,4751493 MiB

There are certain limitations and scope for improvement in our study. In the present model, the pretraining process is executed separately for three models at different scales, which requires significant computational resources and does not to capture inter-scale knowledge during self-supervised learning. An Omni model trained with images from multiple scales and imbued with scale-aware knowledge in the feature embedding is promising.

Moreover, the largest visual field of the current pipeline is 1024 × 1024 pixels, which is still a relatively small area in WSIs. However, the recent advancements in ViTs present an opportunity to enhance the pipeline by incorporating larger spatial relationships and more regional information in larger visual fields, allowing it to receive all information at the slide level directly.

Although the primary purpose of feature clustering for bag generalization is to aggregate patches that are sparsely distributed on WSIs without spatial constraints, it remains intriguing to conduct similarity analysis on features across different scales. This analysis aims to comprehensively understand the relationships between features across scales and the knowledge scenarios in real datasets, in comparison to the proposed toy dataset design.

7. Conclusion

In this study, we introduce a novel cross-scale MIL approach that effectively integrates multi-scale features with inter-scale knowledge. Additionally, the proposed method utilizes cross-scale attention scores to generate importance maps, enhancing the interpretability and comprehensibility of the CS-MIL model. Our experimental and simulated results reveal that the proposed approach outperforms existing multi-scale MIL benchmarks. The visualization of cross-scale attention produces scale-specific importance maps that potentially assists clinicians in interpreting image-based disease phenotypes. This contribution highlights the potential of cross-scale MIL in digital pathology and encourages further research in this area.

8. Acknowledgements

This work is supported by The Leona M. and Harry B. Helmsley Charitable Trust grant G-1903-03793, NSF CAREER 1452485, and Veterans Affairs Merit Review grants I01BX004366 and I01CX002171, and R01DK103831. This work is also supported in part by NIH R01DK135597 (Huo).

References

Agarap AF, 2018. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375. [Google Scholar]

Bao S, Chiron S, Tang Y, Heiser CN, Southard-Smith AN, Lee HH, Ramirez MA, Huo Y, Washington MK, Scoville EA, et al., 2021. A cross-platform informatics system for the gut cell atlas: integrating clinical, anatomical and histological data, in: Medical Imaging 2021: Imaging Informatics for Healthcare, Research, and Applications, SPIE. pp. 8–15. [PMC free article] [PubMed] [Google Scholar]

Barbano CA, Perlo D, Tartaglione E, Fiandrotti A, Bertero L, Cassoni P, Grangetto M, 2021a. Unitopatho, a labeled histopathological dataset for colorectal polyps classification and adenoma dysplasia grading, in: 2021 IEEE International Conference on Image Processing (ICIP), IEEE. pp. 76–80. [Google Scholar]

Barbano CA, Perlo D, Tartaglione E, Fiandrotti A, Bertero L, Cassoni P, Grangetto M, 2021b. Unitopatho, a labeled histopathological dataset for colorectal polyps classification and adenoma dysplasia grading, in: 2021 IEEE International Conference on Image Processing (ICIP), pp. 76–80. doi: 10.1109/ICIP42928.2021.9506198. [CrossRef] [Google Scholar]

Bejnordi BE, Litjens G, Hermsen M, Karssemeijer N, van der Laak JA, 2015. A multi-scale superpixel classification approach to the detection of regions of interest in whole slide histopathology images, in: Medical Imaging 2015: Digital Pathology, SPIE. pp. 99–104. [Google Scholar]

Bejnordi BE, Veta M, Van Diest PJ, Van Ginneken B, Karssemeijer N, Litjens G, Van Der Laak JA, Hermsen M, Manson QF, Balkenhol M, et al., 2017. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama 318, 2199–2210. [PMC free article] [PubMed] [Google Scholar]

Campanella G, Hanna MG, Geneslaw L, Miraflor A, Werneck Krauss Silva V, Busam KJ, Brogi E, Reuter VE, Klimstra DS, Fuchs TJ, 2019. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine 25, 1301–1309. [PMC free article] [PubMed] [Google Scholar]

Chen J, Cheung H, Milot L, Martel AL, 2021. Aminn: Autoencoder-based multiple instance neural network improves outcome prediction in multifocal liver metastases, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 752–761. [Google Scholar]

Chen RJ, Chen C, Li Y, Chen TY, Trister AD, Krishnan RG, Mahmood F, 2022. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16144–16155. [Google Scholar]

Chen X, He K, 2021. Exploring simple siamese representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758. [Google Scholar]

Con D, van Langenberg DR, Vasudevan A, 2021. Deep learning vs conventional learning algorithms for clinical prediction in crohn’s disease: A proof-of-concept study. World Journal of Gastroenterology 27, 6476. [PMC free article] [PubMed] [Google Scholar]

Dehaene O, Camara A, Moindrot O, de Lavergne A, Courtiol P, 2020. Self-supervision closes the gap between weak and strong supervision in histology. arXiv preprint arXiv:2012.03583. [Google Scholar]

Deng R, Cui C, Remedios LW, Bao S, Womick RM, Chiron S, Li J, Roland JT, Lau KS, Liu Q, et al., 2022. Cross-scale attention guided multi-instance learning for crohn’s disease diagnosis with pathological images, in: Multiscale multimodal medical imaging: Third International Workshop, MMMI 2022, held in conjunction with MICCAI 2022, Singapore, September 22, 2022, proceedings, Springer. pp. 24–33. [PMC free article] [PubMed] [Google Scholar]

Dimitriou N, Arandjelović O, Caie PD, 2019. Deep learning for whole slide image analysis: an overview. Frontiers in medicine 6, 264. [PMC free article] [PubMed] [Google Scholar]

Gao Y, Liu W, Arjun S, Zhu L, Ratner V, Kurc T, Saltz J, Tannenbaum A, 2016. Multi-scale learning based segmentation of glands in digital colonrectal pathology images, in: Medical Imaging 2016: Digital Pathology, SPIE. pp. 175–180. [PMC free article] [PubMed] [Google Scholar]

Gordon IO, Bettenworth D, Bokemeyer A, Srivastava A, Rosty C, de Hertogh G, Robert ME, Valasek MA, Mao R, Kurada S, et al., 2020. Histopathology scoring systems of stenosis associated with small bowel crohn’s disease: a systematic review. Gastroenterology 158, 137–150. [PMC free article] [PubMed] [Google Scholar]

Gubatan J, Levitte S, Patel A, Balabanis T, Wei MT, Sinha SR, 2021. Artificial intelligence applications in inflammatory bowel disease: emerging technologies and future directions. World journal of gastroenterology 27, 1920. [PMC free article] [PubMed] [Google Scholar]

Hashimoto N, Fukushima D, Koga R, Takagi Y, Ko K, Kohno K, Nakaguro M, Nakamura S, Hontani H, Takeuchi I, 2020a. Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [Google Scholar]

Hashimoto N, Fukushima D, Koga R, Takagi Y, Ko K, Kohno K, Nakaguro M, Nakamura S, Hontani H, Takeuchi I, 2020b. Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3852–3861. [Google Scholar]

Hou L, Samaras D, Kurc TM, Gao Y, Davis JE, Saltz JH, 2016. Patch-based convolutional neural network for whole slide tissue image classification, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2424–2433. [PMC free article] [PubMed] [Google Scholar]

Ilse M, Tomczak J, Welling M, 2018. Attention-based deep multiple instance learning, in: International conference on machine learning, PMLR. pp. 2127–2136. [Google Scholar]

Kiyokawa H, Abe M, Matsui T, Kurashige M, Ohshima K, Tahara S, Nojima S, Ogino T, Sekido Y, Mizushima T, et al., 2022. Deep learning analysis of histologic images from intestinal specimen reveals adipocyte shrinkage and mast cell infiltration to predict postoperative crohn disease. The American Journal of Pathology. [PubMed] [Google Scholar]

Kraszewski S, Szczurek W, Szymczak J, Reguła M, Neubauer K, 2021. Machine learning prediction model for inflammatory bowel disease based on laboratory markers. working model in a discovery cohort study. Journal of Clinical Medicine 10, 4745. [PMC free article] [PubMed] [Google Scholar]

Li B, Li Y, Eliceiri KW, 2021. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14318–14328. [PMC free article] [PubMed] [Google Scholar]

Lu MY, Chen TY, Williamson DF, Zhao M, Shady M, Lipkova J, Mahmood F, 2021a. Ai-based pathology predicts origins for cancers of unknown primary. Nature 594, 106–110. [PubMed] [Google Scholar]

Lu MY, Williamson DF, Chen TY, Chen RJ, Barbieri M, Mahmood F, 2021b. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering 5, 555–570. [PMC free article] [PubMed] [Google Scholar]

Maksoud S, Zhao K, Hobson P, Jennings A, Lovell BC, 2020. Sos: Selective objective switch for rapid immunofluorescence whole slide image classification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3862–3871. [Google Scholar]

Mousavi HS, Monga V, Rao G, Rao AU, 2015. Automated discrimination of lower and higher grade gliomas based on histopathological image analysis. Journal of pathology informatics 6, 15. [PMC free article] [PubMed] [Google Scholar]

Oquab M, Bottou L, Laptev I, Sivic J, 2015. Is object localization for free?-weakly-supervised learning with convolutional neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 685–694. [Google Scholar]

Schirris Y, Gavves E, Nederlof I, Horlings HM, Teuwen J, 2021. Deepsmile: Self-supervised heterogeneity-aware multiple instance learning for dna damage response defect classification directly from h&e whole-slide images. arXiv preprint arXiv:2107.09405. [Google Scholar]

Skrede OJ, De Raedt S, Kleppe A, Hveem TS, Liestøl K, Maddison J, Askautrud HA, Pradhan M, Nesheim JA, Albregtsen F, et al., 2020. Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. The Lancet 395, 350–360. [PubMed] [Google Scholar]

Syed S, Stidham RW, 2020. Potential for standardization and automation for pathology and endoscopy in inflammatory bowel disease. Inflammatory Bowel Diseases 26, 1490–1497. [PMC free article] [PubMed] [Google Scholar]

Tokunaga H, Teramoto Y, Yoshizawa A, Bise R, 2019. Adaptive weighting multi-field-of-view cnn for semantic segmentation in pathology, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12597–12606. [Google Scholar]

Wang D, Khosla A, Gargeya R, Irshad H, Beck AH, 2016. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718. [Google Scholar]

Wang S, Zhu Y, Yu L, Chen H, Lin H, Wan X, Fan X, Heng PA, 2019. Rmdl: Recalibrated multi-instance deep learning for whole slide gastric image classification. Medical image analysis 58, 101549. [PubMed] [Google Scholar]

Xiong C, Chen H, Sung J, King I, 2023. Diagnose like a pathologist: Transformer-enabled hierarchical attention-guided multiple instance learning for whole slide image classification. arXiv preprint arXiv:2301.08125 [Google Scholar]

Yao H, Zhu D.l., Jiang B, Yu P, 2019. Negative log likelihood ratio loss for deep neural network classification, in: Proceedings of the Future Technologies Conference, Springer. pp. 276–282. [Google Scholar]

Yao J, Zhu X, Jonnagaddala J, Hawkins N, Huang J, 2020. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis 65, 101789. [PubMed] [Google Scholar]

Yeshi K, Ruscher R, Hunter L, Daly NL, Loukas A, Wangchuk P, 2020. Revisiting inflammatory bowel disease: pathology, treatments, challenges and emerging therapeutics including drug leads from natural products. Journal of Clinical Medicine 9, 1273. [PMC free article] [PubMed] [Google Scholar]

Zhang H, Meng Y, Zhao Y, Qiao Y, Yang X, Coupland SE, Zheng Y, 2022. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18802–18812. [Google Scholar]

Zhu X, Yao J, Zhu F, Huang J, 2017. Wsisa: Making survival prediction from whole slide histopathological images, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7234–7242. [Google Scholar]