Model Stability Evaluation Under Random Seed Variation Using SSI and RRS
DOI:
https://doi.org/10.29207/resti.v10i5.7731Keywords:
machine learning, model stability, performance variability, random seed, Support Vector Machine, tabular data, XGBoostAbstract
Machine learning models are commonly evaluated using a single experimental run, implicitly assuming that performance remains stable under fixed conditions. However, many machine learning algorithms involve stochastic processes that may cause performance variability across different random seed configurations. We examine how sensitive tabular classifiers are to the choice of random seed. Multi-Layer Perceptron (MLP), Random Forest, Support Vector Machine (SVM), and XGBoost were each trained and tested under 30 seeds on three public benchmark datasets. To assess robustness, this study introduces two complementary stability measures: Stability Sensitivity Index (SSI) and Relative Robustness Score (RRS). Stability was measured on three metrics: Accuracy, F1-score, and ROC-AUC. The results demonstrate that model performance can vary substantially across random seed configurations; on the smallest dataset, the accuracy of a single run differed by up to 7.9 percentage points depending on the seed. XGBoost achieved the highest mean performance on the two large datasets and, in most cases, also the lowest variability there, whereas SVM was the most stable model on the small Breast Cancer dataset. MLP was generally the most seed-sensitive model on the large datasets, and Random Forest on the small one. F1-score varied considerably more than Accuracy and ROC-AUC, particularly under class imbalance. Furthermore, the smallest dataset generally exhibits greater variability than the larger datasets. These results argue for reporting seed-level variability, not only mean scores, as a routine part of model evaluation. The proposed SSI and RRS metrics can support stability-aware model selection and reproducibility-oriented evaluation in applications requiring consistent predictive performance.
Downloads
References
[1] A. Esteva, A. Robicquet, B. Ramsundar, and others, “A guide to deep learning in healthcare,” Nat. Med., vol. 25, no. 1, pp. 24–29, 2019, doi: https://doi.org/10.1038/s41591-018-0316-z.
[2] P. Domingos, “A few useful things to know about machine learning,” Commun. ACM, vol. 55, no. 10, pp. 78–87, 2012, doi: https://doi.org/10.1145/2347736.2347755.
[3] V. Borisov, T. Leemann, K. Seßler, and others, “Deep neural networks and tabular data: A survey,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 6, pp. 7499–7519, 2024, doi: https://doi.org/10.1109/TNNLS.2022.3229161.
[4] R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,” Information Fusion, vol. 81, pp. 84–90, 2022, doi: https://doi.org/10.1016/j.inffus.2021.11.011.
[5] Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko, “Revisiting deep learning models for tabular data,” Adv. Neural Inf. Process. Syst., vol. 34, pp. 18932–18943, 2021, doi: https://doi.org/10.48550/arXiv.2106.11959.
[6] M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, “Do we need hundreds of classifiers to solve real world classification problems?,” Journal of Machine Learning Research, vol. 15, no. 90, pp. 3133–3181, 2014. [Online]. Available: https://jmlr.org/papers/v15/delgado14a.html
[7] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001, doi: https://doi.org/10.1023/A:1010933404324.
[8] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785–794. doi: https://doi.org/10.1145/2939672.2939785.
[9] C. Cortes and V. Vapnik, “Support-vector networks,” Mach. Learn., vol. 20, no. 3, pp. 273–297, 1995, doi: https://doi.org/10.1007/BF00994018.
[10] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61, pp. 85–117, 2015, doi: https://doi.org/10.1016/j.neunet.2014.09.003.
[11] D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient over F1 score and accuracy in binary classification evaluation,” BMC Genomics, vol. 21, p. 6, 2020, doi: https://doi.org/10.1186/s12864-019-6413-7.
[12] O. E. Gundersen and S. Kjensmo, “State of the art: Reproducibility in artificial intelligence,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2018. doi: https://doi.org/10.1609/aaai.v32i1.11503.
[13] S. Raste, R. Singh, J. Vaughan, and V. Nair, “Quantifying inherent randomness in machine learning algorithms,” arXiv preprint, 2022, doi: https://doi.org/10.48550/arXiv.2206.12353.
[14] D. Picard, “Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision,” arXiv preprint, 2021, doi: https://doi.org/10.48550/arXiv.2109.08203.
[15] X. Bouthillier, P. Delaunay, M. Bronzi, and others, “Accounting for variance in machine learning benchmarks,” Proceedings of Machine Learning and Systems, vol. 3, pp. 747–769, 2021, doi: https://doi.org/10.48550/arXiv.2103.03098.
[16] B. Bischl, M. Binder, M. Lang, and others, “Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 13, no. 2, p. e1484, Mar. 2023, doi: https://doi.org/10.1002/widm.1484.
[17] O. Wilson, D. Schoeman, A. Bradley, and C. Clemente, “Practical guidelines for validation of supervised machine learning models in accelerometer-based animal behaviour classification,” Journal of Animal Ecology, vol. 94, no. 7, pp. 1322–1334, Jul. 2025, doi: https://doi.org/10.1111/1365-2656.70054.
[18] S. Bates, T. Hastie, and R. Tibshirani, “Cross-Validation: What Does It Estimate and How Well Does It Do It?,” J. Am. Stat. Assoc., vol. 119, no. 546, pp. 1434–1445, 2024, doi: https://doi.org/10.1080/01621459.2023.2197686.
[19] J. Pineau, P. Vincent-Lamarre, K. Sinha, and others, “Improving reproducibility in machine learning research,” Journal of Machine Learning Research, vol. 22, no. 164, pp. 1–20, 2021, doi:
https://doi.org/10.48550/arXiv.2003.12206.
[20] M. Hutson, “Artificial intelligence faces reproducibility crisis,” Science, vol. 359, no. 6377, pp. 725–726, 2018, doi: https://doi.org/10.1126/science.359.6377.725.
[21] R. Albertoni, S. Colantonio, P. Skrzypczyński, and J. Stefanowski, “Reproducibility of machine learning: Terminology, recommendations and open issues,” arXiv preprint, 2023, doi: https://doi.org/10.48550/arXiv.2302.12691.
[22] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” Journal of Machine Learning Research, vol. 13, pp. 281–305, 2012. [Online]. Available: https://www.jmlr.org/papers/v13/bergstra12a.html.
[23] B. Becker and R. Kohavi, “Adult Data Set,” 1996. doi: https://doi.org/10.24432/C5XW20.
[24] S. Moro, P. Cortez, and P. Rita, “A data-driven approach to predict the success of bank telemarketing,” Decis. Support Syst., vol. 62, pp. 22–31, 2014, doi: https://doi.org/10.1016/j.dss.2014.03.001.
[25] W. H. Wolberg, W. N. Street, and O. L. Mangasarian, “Machine learning techniques to diagnose breast cancer from image-processed nuclear features of fine needle aspirates,” Cancer Lett., vol. 77, no. 2–3, pp. 163–171, 1994, doi: https://doi.org/10.1016/0304-3835(94)90099-X.
[26] L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on tabular data?,” Adv. Neural Inf. Process. Syst., vol. 35, pp. 507–520, 2022, doi: https://doi.org/10.48550/arXiv.2207.08815.
[27] G. Somepalli, M. Goldblum, A. Schwarzschild, and others, “SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training,” arXiv preprint, 2021, doi: https://doi.org/10.48550/arXiv.2106.01342.
[28] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986, doi: https://doi.org/10.1038/323533a0.
[29] V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in International Conference on Machine Learning, 2010. doi: https://dl.acm.org/doi/10.5555/3104322.3104425
[30] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLoS One, vol. 10, no. 3, 2015, doi: https://doi.org/10.1371/journal.pone.0118432.
[31] J. Davis and M. Goadrich, “The relationship between Precision-Recall and ROC curves,” in Proceedings of the 23rd International Conference on Machine Learning (ICML '06), 2006, pp. 233–240. doi: https://doi.org/10.1145/1143844.1143874.
[32] H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263–1284, 2009, doi: https://doi.org/10.1109/TKDE.2008.239.
[33] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognit. Lett., vol. 27, no. 8, pp. 861–874, 2006, doi: https://doi.org/10.1016/j.patrec.2005.10.010.
[34] M. Mosbach, M. Andriushchenko, and D. Klakow, “On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines,” in Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021, doi:
https://doi.org/10.48550/arXiv.2006.04884.
[35] S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, 2023, doi: https://doi.org/10.1016/j.patter.2023.100804.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
