Exposing Relational Identity Leakage in CAPEC Attack Pattern Classification with XGBoost
DOI:
https://doi.org/10.29207/q7qmhe27Keywords:
CAPEC classification, XGBoost, data leakage , feature representation , VULDAT datasetAbstract
Machine learning models for cybersecurity taxonomy classification are frequently trained on relational identity features (such as CWE, Tactic, and Technique identifiers) that may encode dataset-specific associations rather than transferable attack semantics, yet this risk is rarely assessed alongside label formation and missing-value handling in a single, integrated design. This study evaluates how feature representation, CAPEC label formation strategy, and imputation method jointly affect the classification of CAPEC attack patterns using XGBoost on the VULDAT dataset (610 CVE samples, 21 candidate features). Eighteen experimental combinations, crossing three feature scenarios, three imputation methods, and two label formation strategies, were evaluated using five-fold stratified group cross-validation, dummy-classifier and label-shuffle sanity checks, gain-based feature importance, out-of-fold error analysis, and the Wilcoxon signed-rank test. Retaining relational identity features yielded 0.9398-0.9418 mean accuracy and 0.8793-0.8816 F1-Macro, whereas keeping only structural relational counts caused accuracy to collapse to 0.6136 (F1-Macro 0.4110). Relational identity features accounted for 65.2% of gain-based importance, indicating that reported performance depends substantially on deterministic taxonomy identifiers rather than solely on relational complexity. Fixed first-listed CAPEC labeling consistently outperformed frequency-based labeling, while the imputation method had a negligible effect. These findings offer a reusable, leakage-aware evaluation protocol for CVE-CWE-CAPEC-ATT&CK classification research.
References
[1] A. Pinto, L.-C. Herrera, Y. Donoso, and J. A. Gutierrez, "Survey on Intrusion Detection Systems Based on Machine Learning Techniques for the Protection of Critical Infrastructure," Sensors, vol. 23, no. 5, p. 2415, Feb. 2023, doi: 10.3390/s23052415.
[2] R. Kaur, D. Gabrijelčič, and T. Klobučar, "Artificial intelligence for cybersecurity: Literature review and future research directions," Information Fusion, vol. 97, p. 101804, Sep. 2023, doi: 10.1016/j.inffus.2023.101804.
[3] K. Kanakogi et al., "Tracing CVE Vulnerability Information to CAPEC Attack Patterns Using Natural Language Processing Techniques," Information, vol. 12, no. 8, p. 298, Jul. 2021, doi: 10.3390/info12080298.
[4] R. Othman, B. Rossi, and B. Russo, "Cybersecurity Defenses: Exploration of CVE Types Through Attack Descriptions," in 2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, Aug. 2024, pp. 415-418, doi: 10.1109/SEAA64295.2024.00069.
[5] K. Kanakogi et al., "Comparative Evaluation of NLP-Based Approaches for Linking CAPEC Attack Patterns from CVE Vulnerability Information," Applied Sciences, vol. 12, no. 7, p. 3400, Mar. 2022, doi: 10.3390/app12073400.
[6] O. Grigorescu, A. Nica, M. Dascalu, and R. Rughinis, "CVE2ATT&CK: BERT-Based Mapping of CVEs to MITRE ATT&CK Techniques," Algorithms, vol. 15, no. 9, p. 314, Aug. 2022, doi: 10.3390/a15090314.
[7] R. Othman, "Predicting Known Vulnerabilities from Attack Descriptions Using Sentence Transformers," Ph.D. dissertation, Free Univ. of Bozen-Bolzano, Bolzano, Italy, 2026, doi: 10.48550/arXiv.2602.22433.
[8] I. Kawaminami, S. Shao, S. Hariri, and C. Tunc, "Attack Classification and Response Framework (ACR) Based on Machine Learning and CAPEC Ontology," IEEE Access, vol. 14, pp. 11560-11581, 2026, doi: 10.1109/ACCESS.2026.3653365.
[9] C. Bonhomme and A. Dulaunoy, "Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion," arXiv:2607.25572, Jul. 2026, doi: 10.48550/arXiv.2607.25572.
[10] S. Kapoor and A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns, vol. 4, no. 9, p. 100804, Sep. 2023, doi: 10.1016/j.patter.2023.100804.
[11] M. A. Bouke and A. Abdullah, "An empirical study of pattern leakage impact during data preprocessing on machine learning-based intrusion detection models reliability," Expert Systems with Applications, vol. 230, p. 120715, Nov. 2023, doi: 10.1016/j.eswa.2023.120715.
[12] M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, "Data leakage inflates prediction performance in connectome-based machine learning models," Nature Communications, vol. 15, no. 1, p. 1829, Feb. 2024, doi: 10.1038/s41467-024-46150-w.
[13] T. Thomas and E. Rajabi, "A systematic review of machine learning-based missing value imputation techniques," Data Technologies and Applications, vol. 55, no. 4, pp. 558-585, Aug. 2021, doi: 10.1108/DTA-12-2020-0298.
[14] J. Li et al., "Comparison of the effects of imputation methods for missing data in predictive modelling of cohort study datasets," BMC Medical Research Methodology, vol. 24, no. 1, p. 41, Feb. 2024, doi: 10.1186/s12874-024-02173-x.
[15] L. Sasse et al., "Overview of leakage scenarios in supervised machine learning," Journal of Big Data, vol. 12, no. 1, p. 135, May 2025, doi: 10.1186/s40537-025-01193-8.
[16] H. Zouhri, A. Idri, and A. Ratnani, "Evaluating the impact of filter-based feature selection in intrusion detection systems," International Journal of Information Security, vol. 23, no. 2, pp. 759-785, Apr. 2024, doi: 10.1007/s10207-023-00767-y.
[17] S. K. Siahroudi and D. Kudenko, "An effective single-model learning for multi-label data," Expert Systems with Applications, vol. 232, p. 120887, Dec. 2023, doi: 10.1016/j.eswa.2023.120887.
[18] T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System," in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, New York, NY, USA: ACM, Aug. 2016, pp. 785-794, doi: 10.1145/2939672.2939785.
[19] C. E. L. Asry, I. Benchaji, S. Douzi, and B. E. L. Ouahidi, "Enhancing cybersecurity: A high-performance intrusion detection approach through boosting minority class recognition," PLoS ONE, vol. 20, no. 3, p. e0317346, Mar. 2025, doi: 10.1371/journal.pone.0317346.
[20] F. T. Liu, K. M. Ting, and Z.-H. Zhou, "Isolation-Based Anomaly Detection," ACM Transactions on Knowledge Discovery from Data, vol. 6, no. 1, pp. 1-39, Mar. 2012, doi: 10.1145/2133360.2133363.
[21] W. Chua et al., "Web Traffic Anomaly Detection Using Isolation Forest," Informatics, vol. 11, no. 4, p. 83, Nov. 2024, doi: 10.3390/informatics11040083.
[22] J. Allgaier and R. Pryss, "Cross-Validation Visualized: A Narrative Guide to Advanced Methods," Machine Learning and Knowledge Extraction, vol. 6, no. 2, pp. 1378-1388, Jun. 2024, doi: 10.3390/make6020065.
[23] S. Riyanto, I. S. Sitanggang, T. Djatna, and T. D. Atikah, "Comparative Analysis using Various Performance Metrics in Imbalanced Data for Multi-class Text Classification," International Journal of Advanced Computer Science and Applications, vol. 14, no. 6, 2023, doi: 10.14569/IJACSA.2023.01406116.
[24] D. Berrar, "Using p-values for the comparison of classifiers: pitfalls and alternatives," Data Mining and Knowledge Discovery, vol. 36, no. 3, pp. 1102-1139, May 2022, doi: 10.1007/s10618-022-00828-1.
[25] V. Lumumba, D. Kiprotich, M. Mpaine, N. Makena, and M. Kavita, "Comparative Analysis of Cross-Validation Techniques: LOOCV, K-folds Cross-Validation, and Repeated K-folds Cross-Validation in Machine Learning Models," American Journal of Theoretical and Applied Statistics, vol. 13, no. 5, pp. 127-137, Oct. 2024, doi: 10.11648/j.ajtas.20241305.13.
[26] J. S. Aguilar-Ruiz and M. Michalak, "Classification performance assessment for imbalanced multiclass data," Scientific Reports, vol. 14, no. 1, p. 10759, May 2024, doi: 10.1038/s41598-024-61365-z.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Systems Engineering and Information Technology (JOSEIT)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
