Quantifying Target Leakage in SEER Breast Cancer Mortality Prediction

Sestri Novia  Rizki; Vani Maharani  Nasution; Eko Amri  Jaya; Verra Budhi  Lestari; Elfina Maulid

doi:10.29207/resti.v10i5.8105

Authors

Sestri Novia Rizki Universitas Media Nusantara Citra
Vani Maharani Nasution Universitas Jakarta Internasional
Eko Amri Jaya Universitas Media Nusantara Citra
Verra Budhi Lestari Universitas Media Nusantara Citra
Elfina Maulid Universitas Media Nusantara Citra

DOI:

https://doi.org/10.29207/resti.v10i5.8105

Keywords:

breast cancer mortality prediction, class imbalance, machine learning audit, SEER dataset, SHAP analysis

Abstract

Recent machine learning studies on the SEER Breast Cancer dataset report 95–98% accuracy (2022–2025), but their methods are rarely examined. Because the dataset is imbalanced (84.7% Alive), accuracy alone is a poor discrimination metric — a trivial always-Alive classifier scores near 85% — so this audit quantifies leakage inflation on ROC-AUC, computed identically across 12 models. We measure how much performance stems from target leakage via Survival Months, which encodes retrospective survival time unavailable at prediction. Undetected, such leakage could lead clinicians to overtrust a model's risk estimates, risking inappropriate care decisions. We train 12 models under three conditions: a) naïve, with Survival Months and a non-stratified split; b) leakage-free with stratified evaluation; and c) fixed-horizon (36- and 60-month) binary mortality, excluding censored cases. Under Condition A, the best ROC-AUC is 0.889 (accuracy 0.899). Removing leakage yields a consistent AUC ceiling of 0.74–0.77 across all 12 models — an inflation of +0.07 to +0.20 AUC points. SHAP shows Survival Months dominates feature importance by 3.0–4.3× under leakage; imbalance-handling methods (BorderlineSMOTE, class weighting) raise minority-class F1 but lower mean Brier Skill Score to −0.585 and −0.597. The contribution is the audit methodology itself — a reusable protocol for detecting leakage-driven inflation in registry-based clinical prediction.

Downloads

Download data is not yet available.

References

[1] E. T. Sedeta, B. Jobre, and B. Avezbakiyev, “Breast cancer: Global patterns of incidence, mortality, and trends.,” J. Clin. Oncol., vol. 41, no. 16_suppl, pp. 10528–10528, Jun. 2023, doi: 10.1200/JCO.2023.41.16_suppl.10528.

[2] H. Sung et al., “Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries,” CA. Cancer J. Clin., vol. 71, no. 3, pp. 209–249, May 2021, doi: 10.3322/caac.21660.

[3] H. Gu et al., “Global burden and trends of breast cancer: GLOBOCAN 2022 estimates of incidence and mortality in 185 countries,” Chin. Med. J. (Engl.), vol. 139, no. 3, pp. 404–414, Feb. 2026, doi: 10.1097/CM9.0000000000003921.

[4] C. Chu, N. White, and P. Stone, “Prognostication in palliative care,” Clin. Med., vol. 19, no. 4, pp. 306–310, Jul. 2019, doi: 10.7861/clinmedicine.19-4-306.

[5] M. Maltoni et al., “Prognostication in palliative radiotherapy—ProPaRT: Accuracy of prognostic scores,” Front. Oncol., vol. 12, p. 918414, Aug. 2022, doi: 10.3389/fonc.2022.918414.

[6] J. TENG, “SEER Breast Cancer Data.” IEEE DataPort, Jan. 18, 2019. doi: 10.21227/A9QY-PH35.

[7] J. Li et al., “Predicting breast cancer 5-year survival using machine learning: A systematic review,” PLOS ONE, vol. 16, no. 4, p. e0250370, Apr. 2021, doi: 10.1371/journal.pone.0250370.

[8] S. Kagan et al., “Survival prediction in metastatic breast cancer using artificial intelligence: a scoping review,” Curr. Opin. Support. Palliat. Care, vol. 19, no. 4, pp. 279–289, Dec. 2025, doi: 10.1097/SPC.0000000000000780.

[9] P. Manikandan, U. Durga, and C. Ponnuraja, “An integrative machine learning framework for classifying SEER breast cancer,” Sci. Rep., vol. 13, no. 1, p. 5362, Apr. 2023, doi: 10.1038/s41598-023-32029-1.

[10] R. Watrianthos, R. Rayendra, E. Asri, Y. Yuhefizar, and H. Humaira, “Threshold-Optimized and Calibrated Logistic Regression for Breast Cancer Classification,” Salud Cienc. Tecnol., vol. 5, p. 2241, Oct. 2025, doi: 10.56294/saludcyt20252241.

[11] A. M. Alharthi, M. H. Lee, and Z. Y. Algama, “Improving the Diagnosis of Breast Cancer Using Regularized Logistic Regression with Adaptive Elastic Net,” Univers. J. Public Health, vol. 9, no. 5, pp. 317–323, Oct. 2021, doi: 10.13189/ujph.2021.090514.

[12] H. X. Liu et al., “Diagnosing Breast Cancer Based on Support Vector Machines,” J. Chem. Inf. Comput. Sci., vol. 43, no. 3, pp. 900–907, May 2003, doi: 10.1021/ci0256438.

[13] M. Kavitha, G. Lavanya, J. Janani, and Balaji. J, “ENHANCED SVM CLASSIFIER FOR BREAST CANCER DIAGNOSIS,” Int. J. Eng. Technol. Manag. Res., vol. 5, no. 3, pp. 67–74, Feb. 2020, doi: 10.29121/ijetmr.v5.i3.2018.178.

[14] A. Rafiq et al., “Detection and Classification of Histopathological Breast Images Using a Fusion of CNN Frameworks,” Diagnostics, vol. 13, no. 10, p. 1700, May 2023, doi: 10.3390/diagnostics13101700.

[15] P. Ghosh and D. Chatterjee, “Comparative Analysis of Machine Learning Algorithms for Breast Cancer Classification: SVM Outperforms XGBoost, CNN, RNN, and Others,” Apr. 26, 2024, Bioinformatics. doi: 10.1101/2024.04.22.590658.

[16] M. Chetry et al., “Early detection and analysis of accurate breast cancer for improved diagnosis using deep supervised learning for enhanced patient outcomes,” PeerJ Comput. Sci., vol. 11, p. e2784, Apr. 2025, doi: 10.7717/peerj-cs.2784.

[17] Agus Perdana Windarto, Anjar Wanto, S Solikhun, and Ronal Watrianthos, “A Comprehensive Bibliometric Analysis of Deep Learning Techniques for Breast Cancer Segmentation: Trends and Topic Exploration (2019-2023),” J. RESTI Rekayasa Sist. Dan Teknol. Inf., vol. 7, no. 5, pp. 1155–1164, Oct. 2023, doi: 10.29207/resti.v7i5.5274.

[18] S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, p. 100804, Sep. 2023, doi: 10.1016/j.patter.2023.100804.

[19] M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome-based machine learning models,” Nat. Commun., vol. 15, no. 1, p. 1829, Feb. 2024, doi: 10.1038/s41467-024-46150-w.

[20] L. Sasse et al., “Overview of leakage scenarios in supervised machine learning,” J. Big Data, vol. 12, no. 1, p. 135, May 2025, doi: 10.1186/s40537-025-01193-8.

[21] M. Owusu-Adjei, J. Ben Hayfron-Acquah, T. Frimpong, and G. Abdul-Salaam, “Imbalanced class distribution and performance evaluation metrics: A systematic review of prediction accuracy for determining model performance in healthcare systems,” PLOS Digit. Health, vol. 2, no. 11, p. e0000290, Nov. 2023, doi: 10.1371/journal.pdig.0000290.

[22] T. G. Baidoo and H. Rodrigo, “Data-driven survival modeling for breast cancer prognostics: A comparative study with machine learning and traditional survival modeling methods,” PLOS One, vol. 20, no. 4, p. e0318167, Apr. 2025, doi: 10.1371/journal.pone.0318167.

[23] S. Kaindal and B. Venkataramana, “A comparative analysis of parametric survival models and machine learning methods in breast cancer prognosis,” Sci. Rep., vol. 15, no. 1, p. 31288, Aug. 2025, doi: 10.1038/s41598-025-15696-0.

[24] M. Cruz-Fernandez et al., “Predicting Breast Cancer Mortality Using SEER Data: A Comparative Analysis of L1-Logistic Regression and Neural Networks,” Technologies, vol. 14, no. 1, p. 66, Jan. 2026, doi: 10.3390/technologies14010066.

[25] A. Ghasemi, S. Hashtarkhani, D. L. Schwartz, and A. Shaban‐Nejad, “Explainable artificial intelligence in breast cancer detection and risk prediction: A systematic scoping review,” Cancer Innov., vol. 3, no. 5, p. e136, Oct. 2024, doi: 10.1002/cai2.136.

[26] Y. Liu, Y. Fu, Y. Peng, and J. Ming, “Clinical decision support tool for breast cancer recurrence prediction using SHAP value in cooperative game theory,” Heliyon, vol. 10, no. 2, p. e24876, Jan. 2024, doi: 10.1016/j.heliyon.2024.e24876.

[27] S. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” 2017, arXiv. doi: 10.48550/ARXIV.1705.07874.