CVI-Validated Indo-Transformer Framework  for Intelligent Cooperative Supervision

Syahroni Hidayat; Afriani Fajar Navissaturrisqi; Feddy Setio  Pribadi

doi:10.29207/resti.v10i4.7606

Authors

Syahroni Hidayat Universitas Negeri Semarang
Afriani Fajar Navissaturrisqi Universitas Negeri Semarang
Feddy Setio Pribadi Universitas Negeri Semarang

DOI:

https://doi.org/10.29207/resti.v10i4.7606

Keywords:

cooperative supervision, GenAI, IndoBERT, stratified k-fold, text classification

Abstract

Cooperative supervision reports contain complex narrative structures and overlapping administrative terminology, complicating automatic classification into governance, risk profile, financial performance, and capital adequacy. Reliable automation is particularly important for accelerating the analysis of supervisory findings while addressing limited labeled data and imbalanced categories. This study aimed to develop and externally evaluate a text-classification framework combining quantitatively validated Generative Artificial Intelligence (GenAI) labeling with conventional and Transformer-based models. Data comprised 294 preprocessed sentences collected from the Department of Cooperatives, Small and Medium Enterprises, Industry, and Trade of Semarang Regency during 2023–2025. Few-shot annotations were generated using ChatGPT, Gemini, Perplexity, and DeepSeek, and three-model combinations were evaluated using the Content Validity Index (CVI); majority voting from the best combination established ground truth. TF-IDF with Logistic Regression and Support Vector Machine served as baselines, whereas IndoBERT and IndoRoBERTa represented contextual models. Performance was assessed through stratified five-fold cross-validation and external testing on 58 unseen sentences. ChatGPT–Gemini–Perplexity achieved the highest Scale-Level CVI of 0.898. IndoBERT obtained the best cross-validated F1-score of 0.9099, exceeding IndoRoBERTa (0.8217), Logistic Regression (0.8004), and SVM (0.7863). On unseen data, IndoBERT retained an F1-score of 0.862, compared with 0.759 for IndoRoBERTa. These findings demonstrate that CVI-validated ensemble GenAI can construct consistent labels for low-resource administrative texts and that IndoBERT provides the strongest and most stable generalization for cooperative supervision classification. The framework offers a practical basis for scalable annotation and reliable automated support for evidence-based supervisory decision-making.

Downloads

Download data is not yet available.

References

No. 25 Undang-Undang, “Undang-Undang RI No. 25 Tahun 1992 Tentang Perkoperasian Indonesia [Law of the Republic of Indonesia No. 25 of 1992 on Cooperatives],” Peraturan Bpk, no. 25, pp. 1–57, 1992.

S. A. Abdul Hakim Muhiddin, “Momentum Hari Koperasi, untuk perkuat partisipasi dan aset koperasi [Cooperative Day Momentum to Strengthen Participation and Cooperative Assets],” Antara. Accessed: Nov. 20, 2025. [Online]. Available: https://www.antaranews.com/berita/4960893/momentum-hari-koperasi-untuk-perkuat-partisipasi-dan-aset-koperasi

K. K. dan UKKM, “Peraturan Menteri Koperasi dan UKM Nomor 9 Tahun 2020 tentang Pengawasan Koperasi [Regulation of the Minister of Cooperatives and SMEs No. 9 of 2020 on Cooperative Supervision],” Peraturan Menteri Koperasi dan UKM Nomor 9 Tahun 2020 tentang Pengawasan Koperasi, no. 1202, pp. 1–53, 2020.

I. Chalkidis et al., “LexGLUE: A Benchmark Dataset for Legal Language Understanding in English,” Proceedings ofthe 60th Annual Meeting ofthe Association for Computational Linguistics, vol. 1, pp. 4310–4330, 2021, doi: https://doi.org/10.18653/v1/2022.acl-long.297.

H. S. Mawar and M. Rahardi, “Sentiment Analysis of the TPKS Law on Twitter: A Comparative Study of Classification Algorithm Performance,” Journal of Applied Informatics and Computing, vol. 9, no. 6, pp. 3087–3096, 2025, doi: https://doi.org/10.30871/jaic.v9i6.11503.

N. M. Gardazi, A. Daud, M. K. Malik, A. Bukhari, T. Alsahfi, and B. Alshemaimri, “BERT applications in natural language processing: a review,” Artif. Intell. Rev., vol. 58, no. 6, 2025, doi: https://doi.org/10.1007/s10462-025-11162-5.

S. Raza, M. Garg, D. J. Reji, S. R. Bashir, and C. Ding, “NBIAS: A natural language processing framework for BIAS identification in text,” Expert Syst. Appl., vol. 237, no. PB, p. 121542, 2024, doi: https://doi.org/10.1016/j.eswa.2023.121542.

S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, and T. Hashimoto, “Whose Opinions Do Language Models Reflect?,” in Proceedings of the 40th International Conference on Machine Learning (ICML), 2023, pp. 29971–30004. https://dl.acm.org/doi/10.5555/3618408.3619652

T. Y. C. Tam et al., “A framework for human evaluation of large language models in healthcare derived from literature review,” npj Digital Medicine, vol. 7, no. 1, 2024, doi: https://doi.org/10.1038/s41746-024-01258-7.

T. Miftahushudur, H. M. Sahin, B. Grieve, and H. Yin, “A Survey of Methods for Addressing Imbalance Data Problems in Agriculture Applications,” Remote Sensing, vol. 17, no. 3, pp. 1–31, 2025, doi: https://doi.org/10.3390/rs17030454.

S. Szeghalmy and A. Fazekas, “A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning,” Sensors, vol. 23, no. 4, 2023, doi: https://doi.org/10.3390/s23042333.

I. R. Huriah, A. I. S. Widianingrum, Azlina, and Taslim, “Optimasi Augmentasi Data Berbasis Synonym Replacement pada Klasifikasi Teks Berita Menggunakan Neural Network [Optimization of Synonym-Based Data Augmentation in News Text Classification Using Neural Networks],” Komputa : Jurnal Ilmiah Komputer dan Informatika, vol. 14, no. 1, pp. 100–107, 2025, doi: https://doi.org/10.34010/komputa.v14i1.15339

S. Fuchs, J. Schnellbach, H. Wittges, and H. Krcmar, “Human vs. Automated data annotation: Labeling the data set for an ML-driven support ticket classifier,” Data Knowl. Eng., vol. 162, p. 102534, 2026, doi: https://doi.org/10.1016/j.datak.2025.102534.

F. Gilardi, M. Alizadeh, and M. Kubli, “ChatGPT outperforms crowd workers for text-annotation tasks,” Proc. Natl. Acad. Sci. U. S. A., vol. 120, no. 30, pp. 1–3, 2023, doi: https://doi.org/10.1073/pnas.2305016120.

D. F. Polit and C. T. Beck, “The Content Validity Index: Are You Sure You Know What’s Being Reported? Critique and Recommendations,” Res. Nurs. Health, vol. 29, no. 5, pp. 489–497, 2006, doi: https://doi.org/10.1002/nur.20147.

M. S. B. Yusoff, “ABC of Content Validation and Content Validity Index Calculation,” Education in Medicine Journal, vol. 11, no. 2, pp. 49–54, 2019, doi: https://doi.org/10.21315/eimj2019.11.2.6.

S. P. Octarini, A. Y. Zakiyyah, and K. Purwandari, “Comparison of IndoBERT and SVM Algorithm to Perform Aspect Based Sentiment Analysis using Hierarchical Dirichlet Process,” Engineering, MAthematics and Computer Science Journal (EMACS), vol. 7, no. 3, pp. 363–370, 2025, doi: https://doi.org/10.21512/emacsjournal.v7i3.13493.

F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference, pp. 757–770, 2020, doi: https://doi.org/10.18653/v1/2020.coling-main.66.

T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 2623–2631. doi: https://doi.org/10.1145/3292500.3330701.

M. Mosbach, M. Andriushchenko, and D. Klakow, “On the Stability of Fine-Tuning BERT: Misconceptions, Explanations, and Strong Baselines,” in International Conference on Learning Representations (ICLR), Vienna (Virtual), 2021. https://doi.org/10.48550/arXiv.2006.04884

H. Kaur, H. S. Pannu, and A. K. Malhi, “A systematic review on imbalanced data challenges in machine learning: Applications and solutions,” ACM Comput. Surv., vol. 52, no. 4, 2020, doi: https://doi.org/10.1145/3343440.

M. Das, S. Kamalanathan, and P. Alphonse, “A Comparative Study on TF-IDF feature weighting method and its analysis using unstructured dataset,” in COLINS-2021: 5th International Conference on Computational Linguistics and Intelligent Systems, 2021, pp. 98–107. https://doi.org/10.48550/arXiv.2308.04037.

S. Samidi and D. Fatmawati, “Sentiment Classification of IT Service Feedback via TF-IDF,” CogITo Smart Journal, vol. 10, no. 2, pp. 403–417, 2024, doi: https://doi.org/10.31154/cogito.v10i2.701.403-417.

V. B. Lestari and C. A. Hutagalung, “Evaluation of TF-IDF Extraction Techniques in Sentiment Analysis of Indonesian-Language Marketplaces Using SVM, Logistic Regression, and Naive Bayes,” Journal of Computer Science and Applications, vol. 8, no. 1, pp. 22–2025, 2025, doi: https://doi.org/10.21009/j-koma.v8i1.05.

V. Ayumi, M. Purba, and S. Mailana, “Klasifikasi Teks Umpan Balik Kompetensi Sosial Dosen di Perguruan Tinggi Menggunakan Word2Vec dan CNN-1D [Text Classification of Lecturer Social Competency Feedback Using Word2Vec and CNN-1D],” JSAI (Journal Scientific and Applied Informatics), vol. 8, no. 2, pp. 564–569, 2025, doi: https://doi.org/10.36085/jsai.v8i2.8764.

W. Trisnawati and A. Wibowo, “Sentiment Analysis of ICT Service User Using Naive Bayes Classifier and Svm Methods with TF-IDF Text Weighting”, J. Tek. Inform. (JUTIF), vol. 5, no. 3, pp. 709–719, 2024. doi: https://doi.org/10.52436/1.jutif.2024.5.3.1784.

S. Atikah Permata Sari, “Deteksi Fraud Kartu Kredit Dengan Logistic Regression, Random Forest, Dan Gradient Boosting [Credit Card Fraud Detection Using Logistic Regression, Random Forest, and Gradient Boosting],” Jurnal Informatika Terpadu, vol. 6, no. 1, pp. 29–37, 2025, doi: https://doi.org/10.54914/jit.v11i2.1826.

A. Yenkikar, C. N. Babu, and D. J. Hemanth, “Semantic relational machine learning model for sentiment analysis using cascade feature selection and heterogeneous classifier ensemble,” PeerJ Comput. Sci., vol. 8, pp. 1–34, 2022, doi: https://doi.org/10.7717/peerj-cs.1100.

R. Guido, S. Ferrisi, D. Lofaro, and D. Conforti, “An Overview on the Advancements of Support Vector Machine Models in Healthcare Applications: A Review,” Information (Switzerland), vol. 15, no. 4, 2024, doi: https://doi.org/10.3390/info15040235.

E. C. Garrido-Merchan, R. Gozalo-Brizuela, and S. Gonzalez-Carvajal, “Comparing BERT Against Traditional Machine Learning Models in Text Classification,” Journal of Computational and Cognitive Engineering, vol. 2, no. 4, pp. 352–356, 2023, doi: https://doi.org/10.47852/bonviewJCCE3202838.

H. Zhang and M. O. Shafiq, “Survey of transformers and towards ensemble learning using transformers for natural language processing,” J. Big Data, vol. 11, no. 1, 2024, doi: https://doi.org/10.1186/s40537-023-00842-0.

P. Sridevi, Z. Arefin, and S. I. Ahamed, “An integrated machine learning and hyperparameter optimization framework for noninvasive creatinine estimation using photoplethysmography signals,” Healthcare Analytics, vol. 7, no. April, p. 100395, 2025, doi: https://doi.org/10.1016/j.health.2025.100395.

E. Yulianti and N. K. Nissa, “ABSA of Indonesian customer reviews using IndoBERT: single-sentence and sentence-pair classification approaches,” Bulletin of Electrical Engineering and Informatics, vol. 13, no. 5, pp. 3579–3589, 2024, doi: https://doi.org/10.11591/eei.v13i5.8032.

D. Suhartono, M. R. N. Majiid, and R. Fredyan, Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia, vol. 29, no. 16. Springer US, 2024. doi: https://doi.org/10.1007/s10639-024-12717-9.

G. Sri Harsh, Y. Sai Sri Vivek, P. Maneesha, S. K. Rout, S. Ranjith Reddy, and B. K. Sethi, “Automated Interview Evaluation System Using RoBERTa Technology,” 2024 1st International Conference on Cognitive, Green and Ubiquitous Computing, IC-CGU 2024, pp. 1–8, 2024, doi: https://doi.org/10.1109/IC-CGU58078.2024.10530789.

F. Dwi Sukma, R. Wijaya, and A. Romadhony, “Speech to Text Correction for Indonesian Early Marriage Counseling Chatbots Using IndoRoBERTa and Mistral-7B,” Journal on Computing, vol. 10, no. 1, pp. 72–87, 2025, doi: https://doi.org/10.21108/indojc.v10i1.9708.

Q. Li et al., “A Survey on Text Classification: From Traditional to Deep Learning,” ACM Trans. Intell. Syst. Technol., vol. 13, no. 2, 2022, doi: https://doi.org/10.1145/3495162.

I. D. Mienye and T. G. Swart, “Ensemble Large Language Models: A Survey,” Information (Switzerland), vol. 16, no. 8, 2025, doi: https://doi.org/10.3390/info16080688.

H. Seo et al., “Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy,” Applied Sciences (Switzerland), vol. 15, no. 2, pp. 1–35, 2025, doi: https://doi.org/10.3390/app15020671.

M. Buda, A. Maki, and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, pp. 249–259, 2018, doi: https://doi.org/10.1016/j.neunet.2018.07.011.

T. Abedin, H. Xu, and S. Uddin, “The impact of K selection in K fold cross-validation on bias and variance in supervised learning models,” Sci. Rep., vol. 16, no. 1, pp. 1–13, 2026, doi: https://doi.org/10.1038/s41598-026-37247-x.

B. M. Hussein and S. M. Shareef, “An Empirical Study on the Correlation between Early Stopping Patience and Epochs in Deep Learning,” ITM Web of Conferences, vol. 64, p. 01003, 2024, doi: https://doi.org/10.1051/itmconf/20246401003.

M. Belkin, D. Hsu, S. Ma, and S. Mandal, “Reconciling modern machine-learning practice and the classical bias–variance trade-off,” Proc. Natl. Acad. Sci. U. S. A., vol. 116, no. 32, pp. 15849–15854, 2019, doi: https://doi.org/10.1073/pnas.1903070116.

I. Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, Polosukhin, “Attention Is All You Need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. https://doi.org/10.48550/arXiv.1706.03762.

H. Lu, L. Ehwerhemuepha, and C. Rakovski, “A comparative study on deep learning models for text classification of unstructured medical notes with various levels of class imbalance,” BMC Med. Res. Methodol., vol. 22, no. 1, pp. 1–12, 2022, doi: https://doi.org/10.1186/s12874-022-01665-y.