Optimizing Transformer Model FlanT5 for Multi-Question Answering  with Context-Aware Learning Rate

Tri Lathif Mardi Suryanto; Aji Prasetya Wibawa; Hariyono Hariyono; Andrew  Nafalski

doi:10.29207/resti.v9i6.6985

Authors

Tri Lathif Mardi Suryanto Universitas Pembangunan Nasional Veteran Jawa Timur
Aji Prasetya Wibawa Universitas Negeri Malang
Hariyono Hariyono Universitas Pembangunan Nasional Veteran Jawa Timur
Andrew Nafalski University of South Australia

DOI:

https://doi.org/10.29207/resti.v9i6.6985

Keywords:

context-aware learning rate, flanT5, NLP, questions answers system, transformer

Abstract

This study investigates the performance of FlanT5-based transformer models in handling Multiple-Question Answering (M-QA) tasks, in which multiple semantically related questions must be addressed with a single cohesive answer. Unlike traditional QA systems that focus on one-to-one question-answer pairs, the M-QA approach challenges the model to understand contextual relationships across several questions tied to the same topic. A custom dataset was developed with shared context, grouped questions, and a unified answer to train and evaluate the model. The FlanT5 architecture was fine-tuned using different learning rates (0.0001, 0.0002, 0.0003) to explore the effect of training configurations on model performance. The evaluation was conducted using the ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum metrics. The results indicate that a learning rate of 0.0003 provides the optimal performance, achieving a ROUGE-Lsum score of 0.7390. This study confirms the capability of instruction-tuned transformers to manage complex summarization scenarios that require contextual coherence. The findings are relevant for real-world applications such as intelligent digital assistants, clinical decision support, and educational chatbots. Furthermore, this study emphasizes the importance of hyperparameter tuning in improving the effectiveness of question-driven summarization systems for scalable and efficient deployment.

Downloads

Download data is not yet available.

References

S. S. Lakkimsetty, S. V. Latchireddy, S. M. Lakkoju, G. R. Manukonda, and R. V. V. M. Krishna, “Fine-Tuned Transformer Models for Question Answering,” in 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), IEEE, Jul. 2023, pp. 1–5. doi: 10.1109/ICCCNT56998.2023.10307046.

A. Rawat and S. Singh Samant, “Comparative Analysis of Transformer based Models for Question Answering,” in 2022 2nd International Conference on Innovative Sustainable Computational Technologies (CISCT), IEEE, Dec. 2022, pp. 1–6. doi: 10.1109/CISCT55310.2022.10046525.

D. Khashabi et al., “UNIFIEDQA: Crossing format boundaries with a single QA system,” Find. Assoc. Comput. Linguist. Find. ACL EMNLP 2020, pp. 1896–1907, 2020, doi: 10.18653/v1/2020.findings-emnlp.171.

J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu, “PEGASUS: Pre-Training with extracted gap-sentences for abstractive summarization,” 37th Int. Conf. Mach. Learn. ICML 2020, vol. PartF16814, pp. 11265–11276, 2020.

P. Tummala and C. K. Roa, “Exploring T5 and RGAN for Enhanced Sarcasm Generation in NLP,” IEEE Access, vol. 12, pp. 88642–88657, 2024, doi: 10.1109/ACCESS.2024.3416692.

B. Guan, X. Zhu, and S. Yuan, “A T5-based interpretable reading comprehension model with more accurate evidence training,” Inf. Process. Manag., vol. 61, no. 2, p. 103584, Mar. 2024, doi: 10.1016/j.ipm.2023.103584.

S. Soltanmohammadi, Z. U. Shamszaman, G. Shatha, and M. Jamila, “Job-Matching Chatbots Powered by T5: A Comparative Performance Study with GPT-2,” Proc. - Int. Conf. Dev. eSystems Eng. DeSE, pp. 325–330, Mar. 2024, doi: 10.1109/DESE60595.2023.10468896.

T. L. M. Suryanto, A. P. Wibawa, H. Hariyono, and A. Nafalski, “Comparative Performance of Transformer Models for Cultural Heritage in NLP Tasks,” Adv. Sustain. Sci. Eng. Technol., vol. 7, no. 1, p. 0250115, Jan. 2025, doi: 10.26877/asset.v7i1.1211.

S. Wiegreffe, O. Tafjord, Y. Belinkov, H. Hajishirzi, and A. Sabharwal, “Answer, Assemble, Ace: Understanding How Lms Answer Multiple Choice Questions,” ICLR 2025, pp. 1–25, 2025.

I. D. Raharjo and Egia Rosi Subhiyakto, “Implementing Long Short Term Memory (LSTM) in Chatbots for Multi Usaha Raya,” Adv. Sustain. Sci. Eng. Technol., vol. 6, no. 4, p. 02404018, Oct. 2024, doi: 10.26877/asset.v6i4.934.

A. Janson, “How to leverage anthropomorphism for chatbot service interfaces: The interplay of communication style and personification,” Comput. Human Behav., p. 107954, 2023, doi: 10.1016/j.chb.2023.107954.

P. D. Larasati, A. Irawan, S. Anwar, M. F. Mulya, M. A. Dewi, and I. Nurfatima, “Chatbot helpdesk design for digital customer service.” Accessed: Dec. 20, 2024. [Online]. Available: https://pubs2.ascee.org/index.php/aet/article/view/684

C. V. Misischia, F. Poecze, and C. Strauss, “Chatbots in customer service: Their relevance and impact on service quality,” Procedia Comput. Sci., vol. 201, no. C, pp. 421–428, 2022, doi: 10.1016/j.procs.2022.03.055.

S. Kesarwani, Titiksha, and S. Juneja, “Student Chatbot System: A Review on Educational Chatbot,” in 2023 7th International Conference on Trends in Electronics and Informatics (ICOEI), IEEE, Apr. 2023, pp. 1578–1583. doi: 10.1109/ICOEI56765.2023.10125876.

D. P. P. Villanueva and I. Aguilar-Alonso, “A Chatbot as a Support System for Educational Institutions,” in 2021 62nd International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS), IEEE, Oct. 2021, pp. 1–6. doi: 10.1109/ITMS52826.2021.9615271.

S. Hussain, S. H. Al-Hashmi, M. H. Malik, and S. I. Ali Kazmi, “Chatbot in E-learning,” SHS Web Conf., vol. 156, p. 01002, Jan. 2023, doi: 10.1051/shsconf/202315601002.

H. Yu, C. Yu, Z. Wang, D. Zou, and H. Qin, “Enhancing Healthcare through Large Language Models: A Study on Medical Question Answering,” 2024, [Online]. Available: http://arxiv.org/abs/2408.04138

Y. W. Sung, D. S. Park, and C. G. Kim, “A Study of BERT-Based Classification Performance of Text-Based Health Counseling Data,” 2022, doi: 10.32604/cmes.2022.022465.

A. Babu and S. B. Boddu, “BERT-Based Medical Chatbot: Enhancing Healthcare Communication through Natural Language Understanding,” Explor. Res. Clin. Soc. Pharm., vol. 13, p. 100419, Mar. 2024, doi: 10.1016/j.rcsop.2024.100419.

N. Nascimento, P. Alencar, C. Lucena, and D. Cowan, “A Context-Aware Machine Learning-based Approach A Context-Aware Machine Learning-based Approach,” CASCON’2018, Oct. 2018, Markham, Ontario Canada, no. July 2020, 2018.

T. L. M. Suryanto, A. P. Wibawa, H. Hariyono, and H. Shili, “Contextual Relevance-Driven Question Answering Generation: Experimental Insights Using Transformer-Based Models,” Int. J. Eng. Sci. Inf. Technol., vol. 5, no. 4, pp. 103–113, Jul. 2025, doi: 10.52088/ijesty.v5i4.989.

P. K. Rachabatuni, F. Principi, P. Mazzanti, and M. Bertini, “Context-aware chatbot using MLLMs for Cultural Heritage,” in Proceedings of the ACM Multimedia Systems Conference 2024 on ZZZ, New York, NY, USA: ACM, Apr. 2024, pp. 459–463. doi: 10.1145/3625468.3652193.

D. A. Sulistyo, D. D. Prasetya, F. A. Ahda, and A. P. Wibawa, “Pivoted Low Resource Multilingual Translation with NER Optimization,” ACM Trans. Asian Low-Resource Lang. Inf. Process., vol. 24, no. 5, pp. 1–16, May 2025, doi: 10.1145/3727876.

Y. Wu and L. Liu, “Selecting and Composing Learning Rate Policies for Deep Neural Networks,” ACM Trans. Intell. Syst. Technol., vol. 14, no. 2, Feb. 2023, doi: 10.1145/3570508/ASSET/CC4BF25D-3B4F-4EAF-BD15-33E5CBB999E0/ASSETS/GRAPHIC/TIST-2020-12-0363-F07.JPG.

C. Xu, S. Liu, Z. Yang, Y. Huang, and K. K. Wong, “Learning Rate Optimization for Federated Learning Exploiting Over-the-Air Computation,” IEEE J. Sel. Areas Commun., vol. 39, no. 12, pp. 3742–3756, Dec. 2021, doi: 10.1109/JSAC.2021.3118402.

G. Retsinas, G. Sfikas, P. P. Filntisis, and P. Maragos, “Newton-Based Trainable Learning Rate,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Jun. 2023, pp. 1–5. doi: 10.1109/ICASSP49357.2023.10094576.

A. Mah, C. E. M. Golden, and C. M. Constantinople, “Dopamine transients encode reward prediction errors independent of learning rates,” Cell Rep., vol. 43, no. 10, p. 114840, Oct. 2024, doi: 10.1016/J.CELREP.2024.114840.

K. Lv, H. Yan, Q. Guo, H. Lv, and X. Qiu, “AdaLomo: Low-memory Optimization with Adaptive Learning Rate,” Oct. 2023, Accessed: Dec. 13, 2024. [Online]. Available: https://arxiv.org/abs/2310.10195v3

M. A. Hannan et al., “SOC Estimation of Li-ion Batteries with Learning Rate-Optimized Deep Fully Convolutional Network,” IEEE Trans. Power Electron., vol. 36, no. 7, pp. 7349–7353, Jul. 2021, doi: 10.1109/TPEL.2020.3041876.

Y. Wu et al., “Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks,” in 2019 IEEE International Conference on Big Data (Big Data), IEEE, Dec. 2019, pp. 1971–1980. doi: 10.1109/BigData47090.2019.9006104.

M. Choi, “An Empirical Study on the Optimal Batch Size for the Deep Q-Network,” 2019, pp. 73–81. doi: 10.1007/978-3-319-78452-6_8.

Z. Zhuang, M. Liu, A. Cutkosky, and F. Orabona, “Understanding AdamW through Proximal Methods and Scale-Freeness,” no. 2019, 2022, [Online]. Available: http://arxiv.org/abs/2202.00089

R. Llugsi, S. El Yacoubi, A. Fontaine, and P. Lupera, “Comparison between Adam, AdaMax and Adam W optimizers to implement a Weather Forecast based on Neural Networks for the Andean city of Quito,” in 2021 IEEE Fifth Ecuador Technical Chapters Meeting (ETCM), IEEE, Oct. 2021, pp. 1–6. doi: 10.1109/ETCM53643.2021.9590681.

Shem L. Gonzales, “Enhancing Image Classification Performance: A Comparative Analysis of Optimization Algorithms,” Int. J. Adv. Res. Sci. Commun. Technol., pp. 641–645, Jun. 2023, doi: 10.48175/IJARSCT-11918.

S. Sanniboina, S. Trivedi, and S. Vijayaraghavan, “LoRE: Logit-Ranked Retriever Ensemble for Enhancing Open-Domain Question Answering,” Oct. 2024, Accessed: Dec. 12, 2024. [Online]. Available: https://arxiv.org/abs/2410.10042v1

S. Kumar and A. Solanki, “ROUGE-SS: A New ROUGE Variant for Evaluation of Text Summarization,” Authorea Prepr., Jul. 2023, doi: 10.22541/AU.168984209.92955863/V1.

H. S. Ali, L. M. Kefali, S. Parida, and S. R. Dash, “Amharic ATS - A Comparison Between Graph Based and Statistical Based Approach using Rouge Metric and Human Evaluation,” 2022 OPJU Int. Technol. Conf. Emerg. Technol. Sustain. Dev. OTCON 2022, 2023, doi: 10.1109/OTCON56053.2023.10114029.

A. Auriemma Citarella, M. Barbella, M. G. Ciobanu, F. De Marco, L. Di Biasi, and G. Tortora, “Rouge Metric Evaluation for Text Summarization Techniques.” 2024. doi: 10.2139/ssrn.4753220.

M. Barbella and G. Tortora, “Rouge Metric Evaluation for Text Summarization Techniques,” SSRN Electron. J., May 2022, doi: 10.2139/SSRN.4120317.

Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” no. 1, 2019, [Online]. Available: http://arxiv.org/abs/1907.11692

C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res., vol. 21, pp. 1–67, 2020.

H. W. Chung et al., “Scaling Instruction-Finetuned Language Models,” pp. 1–54, 2022, [Online]. Available: http://arxiv.org/abs/2210.11416

S. Narayan et al., “Conditional Generation with a Question-Answering Blueprint,” Trans. Assoc. Comput. Linguist., vol. 11, pp. 974–996, Dec. 2023, doi: 10.1162/tacl_a_00583.