Reliable Indonesian Academic Service Chatbot Using Retrieval-Augmented Generation and Adaptive Refusal

Jarot Budiasto; Tatik Melinda Tallulembang; Selfina Pare; Muhammad Hasbi

doi:10.29207/resti.v10i5.8018

Authors

Jarot Budiasto Universitas Musamus
Tatik Melinda Tallulembang Universitas Musamus
Selfina Pare Universitas Musamus
Muhammad Hasbi Universitas Musamus

DOI:

https://doi.org/10.29207/resti.v10i5.8018

Keywords:

academic service chatbot, Indonesian NLP, refusal mechanism, retrieval-augmented generation, semantic retrieval

Abstract

Academic service information is often stored in lengthy, formal, and distributed university documents, making accurate access difficult. Generative chatbots can improve access but may answer unsupported questions unless retrieval and refusal are controlled. This study proposes a reliability-oriented Retrieval-Augmented Generation (RAG) chatbot for Indonesian academic services integrating BGE-M3, FAISS, adaptive refusal thresholding, rule-based guardrails, Qwen2.5, and retry-on-refusal. A controlled experiment used 12 official documents and 160 Indonesian questions across eight categories. The final configuration achieved Precision@5 of 0.7250, Recall@5 of 0.9125, MRR of 0.8940, nDCG@5 of 0.8888, and mean semantic similarity of 0.8516. All 16 expected-refusal questions were refused and none of the 144 answerable questions was falsely refused, yielding observed refusal accuracy of 1.0000 and a false refusal rate of 0.0000. Retry-on-refusal reduced the false refusal rate from 0.0556 to 0.0000 and increased answerable-only similarity from 0.8150 to 0.8392. These results provide initial evidence that combining retrieval confidence, guardrails, and retry supports more reliable behavior within the controlled benchmark. Because the refusal subset was small and institution-specific, larger evaluations using naturally occurring student queries are required before inferring operational reliability.

Downloads

Download data is not yet available.

References

[1] A. Vaswani et al., “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, in NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, pp. 6000–6010. doi: 10.5555/3295222.3295349.

[2] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, in NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020, pp. 9459–9474. doi: 10.5555/3495724.3496517.

[3] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang, “REALM: retrieval-augmented language model pre-training,” in Proceedings of the 37th International Conference on Machine Learning, in ICML’20. JMLR.org, 2020. doi: 10.5555/3524938.3525306.

[4] V. Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu, Eds., Online: Association for Computational Linguistics, Nov. 2020, pp. 6769–6781. doi: 10.18653/v1/2020.emnlp-main.550.

[5] J. Johnson, M. Douze, and H. Jégou, “Billion-Scale Similarity Search with GPUs,” IEEE Trans. Big Data, vol. 7, no. 3, pp. 535–547, 2021, doi: 10.1109/TBDATA.2019.2921572.

[6] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3982–3992. doi: 10.18653/v1/D19-1410.

[7] T. Gao, X. Yao, and D. Chen, “SimCSE: Simple Contrastive Learning of Sentence Embeddings,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W. Yih, Eds., Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 6894–6910. doi: 10.18653/v1/2021.emnlp-main.552.

[8] P. A. S. Artayasa, A. S. Kusuma, and P. Sugiartawan, “Enhancing Academic Chatbot Accuracy With Retrieval-Augmented Generation in Higher Education,” in 2025 IEEE International Symposium on Consumer Technology (ISCT), 2025, pp. 427–432. doi: 10.1109/ISCT66099.2025.11297210.

[9] L. S. Hartono, E. I. Setiawan, and V. Singh, “Retrieval Augmented Generation-Based Chatbot for prospective and current university students,” Int. J. Eng. Sci. Inf. Technol, vol. 5, no. 3, pp. 268–277, 2025, doi: 10.52088/ijesty.v5i3.951.

[10] M. Khasanova Zafar kizi and Y. Suh, “Design and Performance Evaluation of LLM-Based RAG Pipelines for Chatbot Services in International Student Admissions,” Electronics (Basel)., vol. 14, no. 15, 2025, doi: 10.3390/electronics14153095.

[11] S. Neupane et al., “From Questions to Insightful Answers: Building an Informed Chatbot for University Resources,” in 2024 IEEE Frontiers in Education Conference (FIE), Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2024, pp. 1–9. doi: 10.1109/FIE61694.2024.10892994.

[12] M. L. Husain, Y. Wibisono, and A. Anisyah, “Development of an Academic Services Chatbot Based on Retrieval-Augmented Generation (RAG),” Brilliance: Research of Artificial Intelligence, vol. 5, no. 2, pp. 727–735, Aug. 2025, doi: 10.47709/brilliance.v5i2.6719.

[13] J. Swacha and M. Gracel, “Retrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications,” Applied Sciences, vol. 15, no. 8, 2025, doi: 10.3390/app15084234.

[14] Z. Li, Z. Wang, W. Wang, K. Hung, H. Xie, and F. L. Wang, “Retrieval-augmented generation for educational application: A systematic survey,” Computers and Education: Artificial Intelligence, vol. 8, p. 100417, 2025, doi: 10.1016/j.caeai.2025.100417.

[15] D. Baur, J. Ansorg, C.-E. Heyde, and A. Voelker, “Development and Evaluation of a Retrieval-Augmented Generation Chatbot for Orthopedic and Trauma Surgery Patient Education: Mixed-Methods Study,” JMIR AI, vol. 4, p. e75262, 2025, doi: 10.2196/75262.

[16] H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of Retrieval-Augmented Generation: A Survey,” in Big Data, W. Zhu, H. Xiong, X. Cheng, L. Cui, Z. Dou, J. Dong, S. Pang, L. Wang, L. Kong, and Z. Chen, Eds., Singapore: Springer Nature Singapore, 2025, pp. 102–120. doi: 10.1007/978-981-96-1024-2_8.

[17] J. Liu, R. Ding, L. Zhang, P. Xie, and F. Huang, “Cofe-rag: A comprehensive full-chain evaluation framework for retrieval-augmented generation with enhanced data diversity,” arXiv preprint arXiv:2410.12248, 2024, doi: 10.48550/arXiv.2410.12248.

[18] S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAs: Automated Evaluation of Retrieval Augmented Generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, N. Aletras and O. De Clercq, Eds., St. Julians, Malta: Association for Computational Linguistics, Mar. 2024, pp. 150–158. doi: 10.18653/v1/2024.eacl-demo.16.

[19] Y. Lyu et al., “CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models,” ACM Trans. Inf. Syst., vol. 43, no. 2, Jan. 2025, doi: 10.1145/3701228.

[20] P. Zhao et al., “Retrieval-Augmented Generation for AI-Generated Content: A Survey,” Data Sci. Eng., vol. 11, no. 1, pp. 1–29, 2026, doi: 10.1007/s41019-025-00335-5.

[21] Y. Hu and Y. Lu, “Rag and rau: A survey on retrieval-augmented language model in natural language processing,” arXiv preprint arXiv:2404.19543, 2024, doi: 10.48550/arXiv.2404.19543.

[22] Y. Zhou et al., “Trustworthiness in retrieval-augmented generation systems: A survey,” arXiv preprint arXiv:2409.10102, 2024, doi: 10.48550/arXiv.2409.10102.

[23] Z. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Comput. Surv., vol. 55, no. 12, Mar. 2023, doi: 10.1145/3571730.

[24] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation,” in Findings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and V. Srikumar, Eds., Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 2318–2335. doi: 10.18653/v1/2024.findings-acl.137.

[25] C. A. Gomez-Cabello et al., “Comparative Evaluation of Advanced Chunking for Retrieval-Augmented Generation in Large Language Models for Clinical Decision Support,” Bioengineering, vol. 12, no. 11, 2025, doi: 10.3390/bioengineering12111194.

[26] R. Akkiraju et al., “Facts about building retrieval augmented generation-based chatbots,” arXiv preprint arXiv:2407.07858, 2024, doi: 10.48550/arXiv.2407.07858.

[27] A. James, M. Trovati, and S. Bolton, “Retrieval-Augmented Generation to Generate Knowledge Assets and Creation of Action Drivers,” Applied Sciences, vol. 15, no. 11, 2025, doi: 10.3390/app15116247.

[28] A. Ammar, A. Koubaa, O. Nacar, and W. Boulila, “Optimizing retrieval-augmented generation: analysis of hyperparameter impact on performance and efficiency,” arXiv preprint arXiv:2505.08445, 2025, doi: 10.48550/arXiv.2505.08445.

[29] B. Saha, U. Saha, and M. Z. Malik, “QuIM-RAG: Advancing Retrieval-Augmented Generation With Inverted Question Matching for Enhanced QA Performance,” IEEE Access, vol. 12, pp. 185401–185410, 2024, doi: 10.1109/ACCESS.2024.3513155.

[30] Y. Yu et al., “RankRAG: unifying context ranking with retrieval-augmented generation in LLMs,” in Proceedings of the 38th International Conference on Neural Information Processing Systems, in NIPS ’24. Red Hook, NY, USA: Curran Associates Inc., 2024. doi: 10.5555/3737916.3741766.

[31] A. Abdallah, M. Abdalla, B. Piryani, J. Mozafari, M. Ali, and A. Jatowt, “RerankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedback,” in Proceedings of the 34th ACM International Conference on Information and Knowledge Management, in CIKM ’25. New York, NY, USA: Association for Computing Machinery, 2025, pp. 6593–6597. doi: 10.1145/3746252.3761484.

[32] Y. Katsis et al., “mtRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems,” Trans. Assoc. Comput. Linguist., vol. 13, pp. 784–808, 2025, doi: 10.1162/tacl.a.19.

[33] G. Mialon et al., “Augmented language models: a survey,” arXiv preprint arXiv:2302.07842, 2023, doi: 10.48550/arXiv.2302.07842.

[34] V. Raina and M. Gales, “Question-Based Retrieval using Atomic Units for Enterprise RAG,” in Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), M. Schlichtkrull, Y. Chen, C. Whitehouse, Z. Deng, M. Akhtar, R. Aly, Z. Guo, C. Christodoulopoulos, O. Cocarascu, A. Mittal, J. Thorne, and A. Vlachos, Eds., Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 219–233. doi: 10.18653/v1/2024.fever-1.25.

[35] A. Yang et al., “Qwen2.5 Technical Report,” arXiv preprint arXiv:2412.15115, 2025, doi: 10.48550/arXiv.2412.15115.