ResNet34-Encoded U-Net with Transfer Learning for Breast Cancer Cell Segmentation Under Data Scarcity

Abstract

Histopathological examination of hematoxylin and eosin (H&E)-stained tissue remains the clinical gold standard for breast cancer diagnosis; however, manual cell segmentation is labor-intensive, subjective, and prone to substantial inter-observer variability. Deep learning-based segmentation models have demonstrated considerable promise in computational pathology, yet their performance under conditions of severe data scarcity remains insufficiently characterized. This study proposes and evaluates a ResNet34-encoded U-Net trained via transfer learning as a solution to this challenge, benchmarking it against a vanilla U-Net trained from random initialization as a controlled baseline. The experimental dataset comprises 42 H&E-stained breast cancer whole-slide image patches acquired at Qingdao Central Hospital (2019–2022), partitioned into 34 training and 8 test images. To mitigate the severe class imbalance inherent in the dataset — in which cancerous cell regions constitute only 16.2% of image pixels on average — a weighted Binary Cross-Entropy and Dice composite loss function was employed alongside a differential learning rate strategy to preserve pretrained encoder representations during fine-tuning. Data augmentation was applied to the training set to improve generalization under extreme data constraint. Evaluation was conducted using the Dice Similarity Coefficient (DSC) and Intersection over Union (IoU) as primary metrics. The ResNet34-UNet achieved a DSC of 0.6071 and IoU of 0.4392, outperforming the vanilla U-Net (DSC: 0.5554, IoU: 0.4004) by 9.3% and 9.7% respectively, while converging 37.5% faster. These findings demonstrate that ImageNet-pretrained encoder features transfer effectively to H&E pathology domains even under extreme data constraints, providing a reproducible and computationally efficient baseline pipeline for automated cell segmentation in resource-limited clinical settings.