Background
Advancement in multiplex immunohistochemistry (mIHC) allows for single-cell tumor microenvironment analysis, greatly enhancing our comprehension of cancer immunology.1 However, clinical application of mIHC remains challenging due to technical variations and cost. Hematoxylin and eosin (H&E) staining, a cost-effective method commonly used in clinics, combined with emerging digital pathology,2 offers promising prospects. Increasing evidence suggests that deep learning (DL) can discern molecular variations from histomorphological features in H&E images.3 4 This has sparked research into applying DL to digital H&E images, a concept referred to as H&E 2.0.5 Model training demands accurate cell labels, but traditional manual annotation methods are labor-intensive, time-consuming, and show significant observer variability.3 6 In this study, we investigate and propose the best methods for image registration and nuclear segmentation to create accurate cell labels for training a predictive model particularly for CD3+ T-cell prediction. Our primary aim is to create an automated workflow that enables robust and precise virtual staining on widely used H&E images.
Methods
The cohort includes 19 triple-negative breast cancer samples in a TMA format. The TMA slide underwent mIHC staining, was subsequently washed, stained with H&E, and imaged. The mIHC staining panel consists of DAPI/CD20/CD3/Ki67/CD163/FOXP3 stains. For each of the samples, the H&E and mIHC stained images were first cropped into individual core areas. Using SIFT from OpenCV (v4.7.0.72), the mIHC images were warped to align with their corresponding H&E images. Nuclear segmentation on H&E images was performed using the pretrained ‘2D_versatile_he’ model from StarDist (v0.8.3) where various probability thresholds were assessed (figure 1). All analysis was performed using Python programming.
Results
Our findings indicate that the use of SIFT registration significantly enhances the precision of mIHC signal mapping to H&E images, both at the tissue and cellular levels. This is observed through the alignment of the nuclear mask derived from H&E images with the DAPI nuclear signal in mIHC (figure 2). Using the StarDist model on H&E images at a 0.7 probability threshold gave results closest to manual annotation, unlike the over-segmentation seen with DeepCell’s Mesmer algorithm (figure 3).
Conclusions
We have presented an optimized approach for generating ground truth with minimal user involvement. This approach can be effortlessly scaled to multiple biomarkers and an increased quantity of training images. In future work, we plan to broaden our focus to include other types of tumors and evaluate the impact on the performance of virtual staining predictions.
Acknowledgements
This work was supported by the Bioinformatics Institute, Singapore Immunology Network, and Insitute of Molecular and Cell Biology, Agency for Science, Technology and Research (A*STAR).
References
Sahardi VK, Armengol G. Molecular Biomarkers in Cancer. Biomolecules. 2022;12(8):1021.
Lee K, Lockhart JH, Xie M, Chaudhary R, Slebos RJC, Flores ER, Chung CH, Tan AC. Deep Learning of Histopathology Images at the Single Cell Level. Front Artif Intell. 2021;4:754641.
Liu Y, Li X, Zheng A, Zhu X, Liu S, Hu M, Luo Q, Liao H, Liu M, He Y, Chen Y. Predict Ki-67 Positive Cells in H&E-Stained Images Using Deep Learning Independently From IHC-Stained Images. Front Mol Biosci. 2020;7:183.
Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyo D, Moreira AL, Razavian N, Tsirigos A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med. 2018;24(10):1559–1567.
Comiter C, Vaishnav ED, Ciampricotti M, Li B, Yang Y, Rodig SJ, Turner M, Pfaff KL, Jané-Valbuena J, Slyper M, Waldman J, Vigneau S, Wu J, Blosser TR, Segerstolpe Å, Abravanel D, Wagle N, Zhuang X, Rudin CM, Klughammer J, Rozenblatt-Rosen O, Kobayash-Kirschvink KJ, Shu J, Regev A. Inference of single cell profiles from histology stains with the Single-Cell omics from Histology Analysis Framework (SCHAF). bioRxiv. 2023:2023.03.21.533680.
Couture HD. Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review. J Pers Med. 2022;12(12):2022.
Abstract 1282 Figure 1
Overview of study design on optimizing cell labeling for virtual staining of CD3+ cells from H&E images
Abstract 1282 Figure 2
Comparison of pre-SIFT registration (top row) and post-SIFT registration images (bottom row). (A) H&E stained (background layer) and mlHC (front layer, green) image pairs were blended together to show degree of alignment. (B) Inset of blended H&E and mlHC image as shown in (A). (C) StarDist nuclear segmentation mask derived from H&E image overlay on H&E image. (D) StarDist nuclear segmentation mask derived from H&E image overlay on counterpart mlHC DAPI signals. The registered image pair shows higher precision in alignment as compared to the non-registered image pair
Abstract 1282 Figure 3
Assessment of StarDist ‘2D_versatile_he’ model nuclear segmentation performance. (A) StarDist nuclear segmentation algorithm performance at varying probability threshold levels (prob) with a non- maximum suppression threshold of 0.35. StarDist performance on H&E images is compared against Mesmer model from DeepCell library on mlHC images. StarDist gave significantly closer results to the ground truth than Mesmer. Ground truth cell counts are manually determined by a histology expert using Indica Labs HALO® image analysis platform. (B) Comparison of StarDist and Mesmer derived nuclear mask overlays on mlHC DAPI signals from H&E and mlHC-stained images respectively. StarDist displays accurate nuclear segmentation whereas Mesmer yields over-segmentation of the nuclei