Comparing Data Preprocessing Strategy on T5 Architecture to Classify ICD-10 Diagnosis

Authors

  • I Gusti Ngurah Lanang Wijayakusuma Universitas Udayana
  • Made Sudarma Universitas Udayana
  • I Ketut Gede Darma Putra Universitas Udayana
  • Oka Sudana Universitas Udayana
  • Dian Astutik Universitas Udayana

DOI:

https://doi.org/10.29207/resti.v9i5.6919

Keywords:

ICD-10 classification, medical NLP, preprocessing strategy, T5, text-to-text transformer

Abstract

Manual ICD-10 coding in healthcare systems remains time-consuming, error-prone, and inefficient, particularly in resource-constrained settings. This study investigates the effect of various preprocessing strategies on the performance of the Text-to-Text Transfer Transformer (T5) model for primary diagnosis classification using structured clinical data. A total of 7,263 clinical records were collected from two high-density regions in Bali (Badung and Gianyar) between January 2023 and March 2024, then converted into descriptive text prompts for model training. Four experimental scenarios combined variations of input features and label configurations, comparing T5 with Oversampling against T5 with Easy Data Augmentation (EDA) plus Oversampling. Results showed that T5 with Random Oversampling consistently outperformed the EDA-based configuration across all scenarios, with performance gaps ranging from 8% to 19%. Scenario 4, which excluded body system features and the semantically overlapping E860 label, achieved the highest balance, reaching 84.7% accuracy, 85.1% precision, 84.7% recall, and 84.3% F1-score. Conversely, the EDA-based approach reduced training time by up to 72%, indicating a clear trade-off between performance and efficiency. Both configurations frequently misclassified semantically similar codes within the same ICD-10 categories, underscoring the difficulty of distinguishing clinically related diagnoses. Overall, the results suggest that careful selection of preprocessing strategies can enhance transformer-based medical text classification, while striking a balance between model performance and training efficiency. This work may serve as an initial reference for developing more efficient semi-automated medical coding systems in the Indonesian regional healthcare context.

Downloads

Download data is not yet available.

References

K. M. Mitura et al., “ICD-10 Classification in the Practice of Emergency Medical Teams: New Insights,” Emerg Med Int, vol. 2024, no. 1, p. 8506561, 2024, doi: https://doi.org/10.1155/2024/8506561.

Y. Hong and M. L. Zeng, “International Classification of Diseases (ICD),” Knowledge Organization, vol. 49, no. 7, pp. 496–528, 2022, doi: 10.5771/0943-7444-2022-7-496.

Z. Ma, J. Li, M. Sufyan, and J. Li, “Automated ICD Coding for Primary Diagnosis Based on Graph Convolution Network,” Conf Proc IEEE Int Conf Syst Man Cybern, pp. 3216–3221, 2023, doi: 10.1109/SMC53992.2023.10394578.

P. Nie, H. Wu, and Z. Cai, “Towards Automatic ICD Coding via Label Graph Generation,” Mathematics, vol. 12, no. 15, p. 2398, Aug. 2024, doi: 10.3390/MATH12152398.

K. Burks, J. Shields, J. Evans, J. Plumley, J. Gerlach, and S. Flesher, “A systematic review of outpatient billing practices,” SAGE Open Med, vol. 10, p. 20503121221099020, 2022, doi: 10.1177/20503121221099021.

J. Quint and A. Brownrigg, “What Should Clinicians Know About How Coding Influences Epidemiological Research?,” AMA J Ethics, vol. 27, no. 1, pp. E51–E57, Jan. 2025, doi: 10.1001/AMAJETHICS.2025.51.

C. Crema, G. Attardi, D. Sartiano, and A. Redolfi, “Natural language processing in clinical neuroscience and psychiatry: A review,” Front Psychiatry, vol. 13, 2022, doi: 10.3389/fpsyt.2022.946387.

J. H. B. Masud, C.-C. Kuo, C.-Y. Yeh, H.-C. Yang, and M.-C. Lin, “Applying Deep Learning Model to Predict Diagnosis Code of Medical Records,” Diagnostics, vol. 13, no. 13, 2023, doi: 10.3390/diagnostics13132297.

J. Jasmir, W. Riyadi, S. R. Agustini, Y. Arvita, D. Meisak, and L. Aryani, “Bidirectional Long Short-Term Memory and Word Embedding Feature for Improvement Classification of Cancer Clinical Trial Document,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 6, no. 4, pp. 505–510, Aug. 2022, doi: 10.29207/RESTI.V6I4.4005.

S. Ayanouz, A. abdelhakim Boudhir, and M. Ben Ahmed, “Unleashing the Power of NLP and Transformers: A Game-Changer in Medical Research and Clinical Practice and a revolution of Medical Text Analysis: Case Study: Cancer report classification by priority,” in Proceedings of the 6th International Conference on Networking, Intelligent Systems & Security, in NISS ’23. New York, NY, USA: Association for Computing Machinery, 2023. doi: 10.1145/3607720.3607787.

F. Alafari, M. Driss, and A. Cherif, “Advances in natural language processing for healthcare: A comprehensive review of techniques, applications, and future directions,” Comput Sci Rev, vol. 56, p. 100725, 2025, doi: 10.1016/j.cosrev.2025.100725.

M. H. Popescu, K. Roitero, S. Travasci, and V. Della Mea, “Automatic Assignment of ICD-10 Codes to Diagnostic Texts using Transformers Based Techniques,” in 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI), 2021, pp. 188–192. doi: 10.1109/ICHI52183.2021.00037.

E. Al-Bashabsheh, A. Alaiad, M. Al-Ayyoub, O. Beni-Yonis, R. A. Zitar, and L. Abualigah, “Improving clinical documentation: automatic inference of ICD-10 codes from patient notes using BERT model,” Journal of Supercomputing, vol. 79, no. 11, pp. 12766–12790, Jul. 2023, doi: 10.1007/S11227-023-05160-Z.

I. Aden, C. H. T. Child, and C. C. Reyes-Aldasoro, “International Classification of Diseases Prediction from MIMIIC-III Clinical Text Using Pre-Trained ClinicalBERT and NLP Deep Learning Models Achieving State of the Art,” Big Data and Cognitive Computing, vol. 8, no. 5, p. 47, May 2024, doi: 10.3390/BDCC8050047.

H. Chen, “Comparison of Large Language And Vision Models on Representative Downstream Tasks,” 2023 International Conference on Image Processing, Computer Vision and Machine Learning, ICICML 2023, pp. 307–311, 2023, doi: 10.1109/ICICML60161.2023.10424913.

S. Elmitwalli, J. Mehegan, A. Gallagher, and R. Alebshehy, “Enhancing sentiment and intent analysis in public health via fine-tuned Large Language Models on tobacco and e-cigarette-related tweets,” Front Big Data, vol. 7, Nov. 2024, doi: 10.3389/FDATA.2024.1501154.

Q. Zhao, Y. Xia, Y. Long, G. Xu, and J. Wang, “Leveraging sensory knowledge into Text-to-Text Transfer Transformer for enhanced emotion analysis,” Inf Process Manag, vol. 62, no. 1, p. 103876, Jan. 2025, doi: 10.1016/J.IPM.2024.103876.

A. Marshan, A. N. Almutairi, A. Ioannou, D. Bell, A. Monaghan, and M. Arzoky, “MedT5SQL: a transformers-based large language model for text-to-SQL conversion in the healthcare domain,” Front Big Data, vol. 7, Jun. 2024, doi: 10.3389/FDATA.2024.1371680.

M. Siino, I. Tinnirello, and M. La Cascia, “Is text preprocessing still worth the time? A comparative survey on the influence of popular preprocessing methods on Transformers and traditional classifiers,” Inf Syst, vol. 121, p. 102342, Mar. 2024, doi: 10.1016/J.IS.2023.102342.

H. S. Woo, J. M. Kim, and W. G. Lee, “Validation of Text Data Preprocessing Using a Neural Network Model,” Math Probl Eng, vol. 2020, no. 1, p. 1958149, Jan. 2020, doi: 10.1155/2020/1958149.

K. Phorah, M. Sumbwanyambe, and M. Sibiya, “Systematic Literature Review on Data Preprocessing for Improved Water Potability Prediction: A Study of Data Cleaning, Feature Engineering, and Dimensionality Reduction Techniques,” Nanotechnol Percept, vol. 20, no. S11, pp. 133–151, Sep. 2024, doi: 10.62441/NANO-NTP.V20IS11.1945.

C. P. Chai, “Comparison of text preprocessing methods,” Nat Lang Eng, vol. 29, no. 3, pp. 509–553, May 2023, doi: 10.1017/S1351324922000213.

S. Z. Ridoy, J. Sultana, Z. F. Ria, M. A. Uddin, M. H. Rahman, and R. M. Rahman, “An Efficient Text Cleaning Pipeline for Clinical Text for Transformer Encoder Models,” in 2024 IEEE 12th International Conference on Intelligent Systems (IS), 2024, pp. 1–9. doi: 10.1109/IS61756.2024.10705199.

B. Kar and B. K. Sarkar, “A Hybrid Feature Reduction Approach for Medical Decision Support System,” Math Probl Eng, vol. 2022, no. 1, Jan. 2022, doi: 10.1155/2022/3984082.

D. Dhinakaran, L. Srinivasan, S. Edwin Raja, K. Valarmathi, and M. Gomathy Nayagam, “Synergistic feature selection and distributed classification framework for high-dimensional medical data analysis,” MethodsX, vol. 14, Jun. 2025, doi: 10.1016/J.MEX.2025.103219.

C. Mugisha and I. Paik, “Comparison of Neural Language Modeling Pipelines for Outcome Prediction from Unstructured Medical Text Notes,” IEEE Access, vol. 10, pp. 16489–16498, 2022, doi: 10.1109/ACCESS.2022.3148279.

M. Salmi, D. Atif, D. Oliva, A. Abraham, and S. Ventura, “Handling imbalanced medical datasets: review of a decade of research,” Artif Intell Rev, vol. 57, no. 10, pp. 1–57, Oct. 2024, doi: 10.1007/S10462-024-10884-2.

A. Ghavidel and P. Pazos, “Machine learning (ML) techniques to predict breast cancer in imbalanced datasets: a systematic review,” Journal of Cancer Survivorship, vol. 19, no. 1, pp. 270–294, Feb. 2023, doi: 10.1007/S11764-023-01465-3.

Y. Ren, D. Wu, Y. Tong, A. López-DeFede, and S. Gareau, “Issue of Data Imbalance on Low Birthweight Baby Outcomes Prediction and Associated Risk Factors Identification: Establishment of Benchmarking Key Machine Learning Models With Data Rebalancing Strategies,” J Med Internet Res, vol. 25, p. e44081, May 2023, doi: 10.2196/44081.

D. Fechner, S. Reid, and S. Dolnicar, “Tourism and Emerging Infectious Diseases: More Connections Than First Meet the Eye,” J Travel Res, vol. 62, no. 5, pp. 935–948, May 2023, doi: 10.1177/00472875221127718.

T. R. Rao, B. Pratiksha, C. Navyasri, D. J. Kalita, and S. Alam, “Gastroenteritis: A Comprehensive Review,” International Journal of Current Science Research and Review, vol. 07, no. 10, Oct. 2024, doi: 10.47191/IJCSRR/V7-I10-07.

K. A. Houston et al., “Gastroenteritis aggressive versus slow treatment for rehydration (GASTRO): A phase II rehydration trial for severe dehydration: WHO plan C versus slow rehydration,” BMC Med, vol. 17, no. 1, Jul. 2019, doi: 10.1186/S12916-019-1356-Z.

L. Hartling, S. Bellemare, N. Wiebe, K. F. Russell, T. P. Klassen, and W. R. Craig, “Oral versus intravenous rehydration for treating dehydration due to gastroenteritis in children,” Cochrane Database of Systematic Reviews, vol. 2006, no. 3, Jul. 2006, doi: 10.1002/14651858.CD004390.PUB2.

M. Mujahid et al., “Data oversampling and imbalanced datasets: an investigation of performance for machine learning and feature engineering,” J Big Data, vol. 11, no. 1, pp. 1–32, Dec. 2024, doi: 10.1186/S40537-024-00943-4.

Z. Cai, Z. Li, Y. Liu, L. Guo, and Y. Song, “Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss,” Apr. 2025, doi: 10.48550/arXiv.2505.00021.

S. Henning, W. Beluch, A. Fraser, and A. Friedrich, “A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein, Eds., Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 523–540. doi: 10.18653/v1/2023.eacl-main.38.

J. Wei and K. Zou, “EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 6382–6388. doi: 10.18653/v1/D19-1670.

D. Setiyarini, T. B. Adji, and I. Hidayah, “Evaluating Performance of Transformer Models for Dialogue Summarization: A Comparison of T5-Base, T5-Small, and BART-Base,” in 2024 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), 2024, pp. 184–190. doi: 10.1109/COMNETSAT63286.2024.10862384.

Downloads

Published

2025-10-24

Issue

Section

Artificial Intelligence

How to Cite

Lanang Wijayakusuma, I. G. N., Sudarma, M. ., Darma Putra, I. K. G. ., Sudana, O., & Astutik, D. (2025). Comparing Data Preprocessing Strategy on T5 Architecture to Classify ICD-10 Diagnosis. Jurnal RESTI (Rekayasa Sistem Dan Teknologi Informasi), 9(5), 1109-1120. https://doi.org/10.29207/resti.v9i5.6919