Graph Construction on Whole Image Slides Based on Attention and Learnable Threshold

doi:10.19678/j.issn.1000-3428.0068982

Abstract

Abstract:

Different from conventional graph sampling methods for reducing graph size, such as thresholding method and merging method of edge and node, AdaptConv, an inventive Transformer-based graph neural network module is proposed. This module can remove redundant edges through dynamic learning while aggregating information in the graph structure, thereby forming a new graph, called a reconstructed graph. The reconstructed graph retains effective information of the original graph structure and offers new visual and analytical perspectives for computational pathology. To evaluate the effectiveness of AdaptConv and the reconstructed graph, AdaptConv is integrated into the Clustering-constrained Attention Multiple (CLAM) instance learning framework, and the accuracy of the models of two molecular subtypes on two clinical predictions in statuses of Hormone Receptor (HR) and Human Epidermal Growth Factor Receptor 2 (HER2) in breast cancer is demonstrated. Compared to the original CLAM model, the improved model exhibits a 4.7% increase for the HR subtype and a 0.8% increase for the HER2 subtype in the Area Under the Curve (AUC) metric. Furthermore, the proposed model generates more reasonable and reliable attention maps. The attention maps demonstrate patterns similar to the gold standard immunohistochemical staining slices. The reconstructed graph produced by this model also shows a correlation with the tissue regions and the attention map, which has potential research value.

Key words: whole slide image, multi-instance learning, Graph Neural Network (GNN), graph construction, visualization

摘要：

不同于常规阈值法或边与节点的合并等图采样方法侧重于减小图的规模, 该工作提出了一种具备创新性的基于Transformer架构的图神经网络模块AdaptConv。该模块能够在图结构中进行信息聚合的同时通过动态学习去除冗余的边, 从而构建出新的图, 称之为重构图。重构图保留了原本图结构的有效信息, 也为计算病理提供了新的可视化角度和分析角度。为了评估AdaptConv及重构图的有效性, 该工作将AdaptConv模块集成在聚类约束注意力多种实例学习(CLAM)框架中, 并在乳腺癌的激素受体(HR)和人表皮生长因子受体2(HER2)两种分子分型的计算病理诊断预测任务上验证了模型的准确性。与原生CLAM模型相比, 改进模型的曲线下面积(AUC)指标在HR分型上取得了4.7%的提升, 在HER2分型上取得了0.8%的提升。此外, AdaptConv优化模型生成的注意力分布图更合理可靠, 呈现出与诊断标准免疫组织化学染色一致的分布模式。最后, 该模型生成的重构图与特定组织区域和注意力图都表现出了关联, 具备进一步研究的价值。

关键词: 全景切片图像, 多实例学习, 图神经网络, 图构建, 可视化

CHEN Depin, ZHAO Shen, JIAO Yiping, WANG Xiangxue, LÜ Hong, XU Jun. Graph Construction on Whole Image Slides Based on Attention and Learnable Threshold[J]. Computer Engineering, 2025, 51(3): 229-240.

陈德品, 赵珅, 焦一平, 王向学, 吕泓, 徐军. 基于注意力及可学习阈值的全景切片图构建[J]. 计算机工程, 2025, 51(3): 229-240.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068982

https://www.ecice06.com/EN/Y2025/V51/I3/229

Figures/Tables 8

Fig.1 Traditional graph sampling mode and our graph sampling mode

Fig.2 Overall structure of AdaptConv

Fig.3 Flow chart of CLAM based on contextual features

Fig.4 Visualization of H&E-stained WSI, IHC-stained WSI and attention map

Fig.5 Visualization of connect subgraph in reconstructed graph overlapping HER2+ attention map and different tissues

Fig.6 Visualization of abnormal WSI regions and its surrounding HER2- branch attention map

References 51

1	MCALLISTER S S, WEINBERG R A. Tumor-host interactions: a far-reaching relationship. Journal of Clinical Oncology, 2010, 28(26): 4022- 4028. doi: 10.1200/JCO.2010.28.4257
2	SONG A H, JAUME G, WILLIAMSON D F, et al. Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering, 2023, 1, 930- 949. doi: 10.1038/s44222-023-00096-8
3	DUGGENTO A, CONTI A, MAURIELLO A, et al. Deep computational pathology in breast cancer. Seminars in Cancer Biology, 2021, 72(1): 226- 237.
4	SRINIDHI C L, CIGA O, MARTEL A L. Deep neural network models for computational histopathology: a survey. Medical Image Analysis, 2021, 67, 101813. doi: 10.1016/j.media.2020.101813
5	DURAN-LOPEZ L, DOMINGUEZ-MORALES J P, CONDE-MARTIN A F, et al. PROMETEO: a CNN-based computer-aided diagnosis system for WSI prostate cancer detection. IEEE Access, 2020, 8, 128613- 128628. doi: 10.1109/ACCESS.2020.3008868
6	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
7	CIRESAN D C, GIUSTI A, GAMBARDELLA L, et al. Deep neural networks segment neuronal membranes in electron microscopy images[C]// Proceedings of the 25th International Conference on Neural Information Processing Systems. New York, USA: Curran Associates Inc., 2012, 2: 2843-2851.
8	张喜科, 马志庆, 赵文华, 等. 基于卷积神经网络的乳腺癌组织病理学图像分类研究综述. 计算机科学, 2022, 49(S2): 362- 370.
	Zhang X K, MA Z Q, ZHAO W H, et al. Review on classification of breast cancer histopathological images based on convolutional neural network. Computer Science, 2022, 49(S2): 362- 370.
9	NGUYEN C, ASAD Z, DENG R, et al. Evaluating transformer-based semantic segmentation networks for pathological image segmentation[C]//Proceedings of the Medical Imaging 2022: Image Processing. Bellingham, USA: SPIE, 2022: 942-947.
10	LI H, YANG F, ZHAO Y, et al. DT-MIL: deformable transformer for multi-instance learning on histopathological image[C]//Proceedings of the Medical Image Computing and Computer Assisted Intervention. Cham, Switzerland: Springer, 2021: 206-216.
11	SHAO Z C, BIAN H, CHEN Y, et al. Transmil: transformer based correlated multiple instance learning for whole slide image classification[C]// Proceedings of the 35th International Conference on Neural Information Processing Systems. New York, USA: Curran Associates Inc., 2021: 2136-2147.
12	LEVY J, HAUDENSCHILD C, BARWICK C, et al. Topological feature extraction and visualization of whole slide images using graph neural networks[C]//Proceedings of the Pacific Symposium on Biocomputing 2021. Singapore, Singapore: World Scientific, 2020: 285-296.
13	PATI P, JAUME G, FONCUBIERTA-RODRIGUEZ A, et al. Hierarchical graph representations in digital pathology. Medical Image Analysis, 2022, 75, 102264. doi: 10.1016/j.media.2021.102264
14	陈思硕, 王晓东, 刘西洋. 基于图神经网络的乳腺癌病理图像分析方法综述. 计算机科学, 2024, 2(1): 172- 185.
	CHEN S S, WANG X D, LIU X Y. A survey on pathological image analysis methods based on graph neural networks. Computer Science, 2024, 2(1): 172- 185.
15	EPPSTEIN D, PATERSON M S, YAO F F. On nearest-neighbor graphs. Discrete & Computational Geometry, 1997, 17(1): 263- 282.
16	CORREDOR G, WANG X, ZHOU Y, et al. Spatial architecture and arrangement of tumor-infiltrating lymphocytes for predicting likelihood of recurrence in early-stage non-small cell lung cancer. Clinical Cancer Research, 2019, 25(5): 1526- 1534. doi: 10.1158/1078-0432.CCR-18-2013
17	CORREDOR G, TORO P, KOYUNCU C, et al. An imaging biomarker of tumor-infiltrating lymphocytes to risk-stratify patients with HPV-associated oropharyngeal cancer. Journal of the National Cancer Institute, 2022, 114(4): 609- 617. doi: 10.1093/jnci/djab215
18	WANG X, BARRERA C, BERA K, et al. Spatial interplay patterns of cancer nuclei and tumor-infiltrating lymphocytes (TILs) predict clinical benefit for immune checkpoint inhibitors. Science Advances, 2022, 8(22): eabn3966. doi: 10.1126/sciadv.abn3966
19	HOU W, HUANG H, PENG Q, et al. Spatial-hierarchical graph neural network with dynamic structure learning for histological image classification[C]//Proceedings of the Medical Image Computing and Computer Assisted Intervention. Cham, Switzerland: Springer, 2022: 181-191.
20	CHENG J, MO X, WANG X, et al. Identification of topological features in renal tumor microenvironment associated with patient survival. Bioinformatics, 2018, 34(6): 1024- 1030. doi: 10.1093/bioinformatics/btx723
21	ZENG H, ZHANG M, XIA Y, et al. Decoupling the depth and scope of graph neural networks[C]// Proceedings of the 35th International Conference on Neural Information Processing Systems. New York, USA: Curran Associates Inc., 2024, 1504: 19665-19679.
22	ZENG H, ZHOU H, SRIVASTAVA A, et al. GraphSAINT: graph sampling based inductive learning method[C]//Proceedings of the International Conference on Learning Representations. Appleton, USA: OpenReview. net, 2020: 1-19.
23	ZHOU Y, GRAHAM S, ALEMI KOOHBANANI N, et al. CGC-Net: cell graph convolutional network for grading of colorectal cancer histology images[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshops. New York, USA: IEEE Press, 2019: 388-398.
24	LU W, GRAHAM S, BILAL M, et al. Capturing cellular topology in multi-gigapixel pathology images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York, USA: IEEE Press, 2020: 260-261.
25	LEE Y, PARK J H, OH S, et al. Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nature Biomedical Engineering, 2022, 6(12): 1323- 1325. doi: 10.1038/s41551-022-00924-z
26	BAEK J, KANG M, HWANG S J. Accurate learning of graph representations with graph multiset pooling[C]//Proceedings of the International Conference on Learning Representations. Appleton, USA: OpenReview. net, 2021: 1-22.
27	满芮, 杨萍, 季程雨, 等. 乳腺癌组织病理学图像分类方法研究综述. 计算机科学, 2020, 47(S2): 145- 150.
	Man R, Yang P, JI C Y, et al. Survey of classification methods of breast cancer histopathological images. Computer Science, 2020, 47(S2): 145- 150.
28	LU M Y, WILLIAMSON D F, CHEN T Y, et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 2021, 5(6): 555- 570. doi: 10.1038/s41551-020-00682-w
29	AHMEDT-ARISTIZABAL D, ARMIN M A, DENMAN S, et al. A survey on graph-based deep learning for computational histopathology. Computerized Medical Imaging and Graphics, 2022, 95, 102027. doi: 10.1016/j.compmedimag.2021.102027
30	WANG J, CHEN R J, LU M Y, et al. Weakly supervised prostate tma classification via graph convolutional networks[C]//Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). New York, USA: IEEE Press, 2020: 239-243.
31	HALL-BEYER M. GLCM texture: a tutorial v. 3.0 March 2017[EB/OL]. (2017-04-04)[2023-11-06]. https://prism.ucalgary.ca/items/8833a1fc-5efb-4b9b-93a6-ac4ff268091c/full.
32	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, USA: IEEE Press, 2016: 770-778.
33	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, USA: IEEE Press, 2017: 4700-4708.
34	RIASATIAN A, BABAIE M, MALEKI D, et al. Fine-tuning and training of densenet for histopathology image representation using tcga diagnostic slides. Medical Image Analysis, 2021, 70, 102032. doi: 10.1016/j.media.2021.102032
35	石静文, 李嘉. 乳腺癌病理图像特征提取方法研究综述. 机电工程技术, 2022, 51(4): 16- 19.
	SHI J W, Li J. Survey on feature extraction methods of breast cancer histopathology. Mechanical & Electrical Engineering Technology, 2022, 51(4): 16- 19.
36	KLEIN D. Centrality measure in graphs. Journal of Mathematical Chemistry, 2010, 47(4): 1209- 1223. doi: 10.1007/s10910-009-9635-0
37	JIMENEZ-SANCHEZ D, ARIZ M, CHANG H, et al. NaroNet: discovery of tumor microenvironment elements from highly multiplexed images. Medical Image Analysis, 2022, 78, 102384. doi: 10.1016/j.media.2022.102384
38	ZHOU J, CUI G, HU S, et al. Graph neural networks: a review of methods and applications. AI Open, 2020, 1, 57- 81. doi: 10.1016/j.aiopen.2021.01.001
39	HAMILTON W L, YING R, LESKOVEC J. Inductive representation learning on large graphs[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: Curran Associates Inc., 2017: 1025-1035.
40	NIU Z, ZHONG G, YU H. A review on the attention mechanism of deep learning. Neurocomputing, 2021, 452(1): 48- 62.
41	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: Curran Associates Inc., 2017: 6000-6010.
42	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[C]//Proceedings of the International Conference on Learning Representations. Appleton, USA: openreview. net, 2021: 1-22.
43	SHI Y, HUANG Z, FENG S, et al. Masked label prediction: unified message passing model for semi-supervised classification[C]//Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. S. l. : International Joint Conferences on Artificial Intelligence, 2021: 1548-1554.
44	CANGEA C, VELIČKOVIĆ P, JOVANOVIĆ N, et al. Towards sparse hierarchical graph classifiers[EB/OL]. (2018-09-03)[2023-11-06]. https://arxiv.org/abs/1811.01287.
45	ZHOU X, WU O. Drop "noise" edge: an approximation of the Bayesian GNNs[C]//Proceedings of the Asian Conference on Pattern Recognition. Cham, Switzerland: Springer, 2021: 59-72.
46	CIGA O, XU T, MARTEL A L. Self supervised contrastive learning for digital histopathology. Machine Learning with Applications, 2022, 7, 100198. doi: 10.1016/j.mlwa.2021.100198
47	CHEN R J, LU M Y, SHABAN M, et al. Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks[C]//Proceedings of the Medical Image Computing and Computer Assisted Intervention. Cham, Switzerland: Springer, 2021: 339-349.
48	HAMMOND M E H, HAYES D F, DOWSETT M, et al. American Society of Clinical Oncology/College of American Pathologists guideline recommendations for immunohistochemical testing of estrogen and progesterone receptors in breast cancer. Archives of Pathology & Laboratory Medicine, 2010, 28(16): 2784- 2795.
49	WOLFF A C, HAMMOND M E H, ALLISON K H, et al. Human epidermal growth factor receptor 2 testing in breast cancer: American Society of Clinical Oncology/College of American Pathologists clinical practice guideline focused update. Archives of Pathology & Laboratory Medicine, 2018, 142(11): 1364- 1382.
50	GRAHAM S, VU Q D, RAZA S E A, et al. Hover-Net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 2019, 58, 101563. doi: 10.1016/j.media.2019.101563
51	ILSE M, TOMCZAK J M, WELLING M. Attention-based deep multiple instance learning[EB/OL]. (2018-02-13)[2023-11-06]. https://arxiv.org/abs/1802.04712.

[1]	LI Siyuan, ZHONG Xingyu, LI Kaiyin, XU Qingzhen. Strategy Teaching Research Based on Multilayer Graph Relationship and Reinforcement Learning [J]. Computer Engineering, 2025, 51(3): 122-130.
[2]	SU Qing, CHEN Jiancheng, GU Guosheng, LIU Dongning, HUANG Jianfeng. Visualization Method for Data Structure Evolution in Program Debugging [J]. Computer Engineering, 2024, 50(9): 197-207.
[3]	Zhengkang ZHANG, Dan YANG, Tiezheng NIE, Yue KOU. Self-Supervised Learning Based on Graph Structural Clustering for Disease Diagnosis Method [J]. Computer Engineering, 2024, 50(7): 360-371.
[4]	Zhongwei LI, Kaixuan GONG, Yong LI, Gege LIU. Feature Extraction and Seed Point Selection Algorithms of Ocean Flow Field on Unstructured Grids [J]. Computer Engineering, 2024, 50(1): 320-328.
[5]	ZHU Xiaoqiang, CHEN Qi. 3D Neuron Modeling Based on Controllable Convolution Surface [J]. Computer Engineering, 2023, 49(3): 231-237.
[6]	LI Ke, LI Shaomei, JI Lixin, LIU Shuo. Method of Face Forgery Detection Based on Self-Attention Capsule Network [J]. Computer Engineering, 2022, 48(2): 194-200,206.
[7]	TANG Wenlin, XIE Kai, WEN Chang, HE Jianbiao. Fast 3D Visualization of Massive Seismic Data in Deep Clustering Index [J]. Computer Engineering, 2022, 48(11): 275-283.
[8]	WANG Xin, FU Qiang, WANG Lin, XU Dawei, WANG Haofen. Survey on Visualization Query Technology of Knowledge Graph [J]. Computer Engineering, 2020, 46(6): 1-11.
[9]	CHEN Yusi, SUN Linfu. R2GPipe: A Declarative Data Pipe from Relations to Property Graph [J]. Computer Engineering, 2019, 45(9): 8-16.
[10]	LI Weichao, ZHANG Zheng, WANG Liqun, LIU Zhenwu, LIU Hao. A Web Threat Situation Analysis Method for Mimic Structure [J]. Computer Engineering, 2019, 45(8): 1-6.
[11]	YAN Dongfei,JIANG Rengui,XIE Jiancang,WANG Yamei,LI Xiaochun. Research on Water Resources Monitoring System of Weihe River Basin Based on Digital Globe [J]. Computer Engineering, 2019, 45(4): 49-55.
[12]	JIANG Rengui, YANG Siyu, XIE Jiancang, YAN Dongfei, WANG Xiaojie. Three-dimensional Visualization Emergency Management Information System of Urban Waterlogging [J]. Computer Engineering, 2019, 45(10): 46-51.
[13]	REN Zhuojun,CHEN Guang. Application of Entropy Visualization Method in Malware Classification [J]. Computer Engineering, 2017, 43(9): 167-171.
[14]	LI Zhimin,YIN Beibei,ZHANG Ping,WANG Jibing,WANG Bin,ZHANG Jinpeng. A Real-time Fault Localization Method and Its Visualization Implementation [J]. Computer Engineering, 2017, 43(2): 111-119.
[15]	MAO Fangdong,XU Hongbin,WANG Yi,TENG Juyuan,ZHANG Mao. Planning Recognition System for Mobile Robot in Orchard Based on VB/Matlab [J]. Computer Engineering, 2017, 43(12): 309-314.

Please choose a citation manager

Content to export