BERT Model Fine-tuned for Scientific Document Classification and Recommendation
DOI:
https://doi.org/10.29207/resti.v9i4.6789Keywords:
BERT, fine-tuning, recommendation system, cosine similarity, document classificationAbstract
The increasing number of academic documents requires efficient and accurate classification and recommendation systems to assist in retrieving relevant information. This system is built using the "bert-base-uncased” model from Hugging Face, which has been fine-tuned to improve the classification accuracy and relevance of document recommendations. The dataset used consists of 2.000 academic documents in the field of computer science, with features including titles, abstracts, and keywords, which were combined into a single input for the model. Document similarity is measured using cosine similarity, resulting in recommendations based on semantic proximity. Unlike traditional approaches, which rely primarily on word frequency or surface-level matching, the proposed method leverages BERT’s contextual embeddings to capture deeper semantic meanings and relationships between documents. This allows for more accurate classification and more context-aware recommendations. Evaluation results show that the best model configuration (learning rate 3e-5, batch size 32, optimizer AdamW) achieved 89.5% training accuracy and an F1-score of 0.8947, while testing yielded 91% accuracy and 90% F1-score. The recommendation system consistently produced Precision@k values above 92% for k between 5 and 30, with Recall@k reaching 1.0 as k increased. These results indicate that the system not only performs reliably in classifying complex academic texts but also effectively recommends contextually relevant documents. This integrated approach shows strong potential for enhancing academic document retrieval and supports the development of semantically aware information management systems.
Downloads
References
A. S. Rahmatullah, E. Mulyasa, S. Syahrani, F. Pongpalilu, and R. E. Putri, “Digital Era 4.0: The Contribution to Education and Student Psychology,” Linguistics and Culture Review, vol. 6, no. 3, pp. 89–107, 2022, doi: 10.21744/lingcure.v6ns3.2064.
R. I. Mukhamediev et al., “Review of Artificial Intelligence and Machine Learning Technologies: Classification, Restrictions, Opportunities and Challenges,” Mathematics, vol. 10, no. 2552, pp. 1–25, 2022, doi: 10.3390/math10152552.
M. Musaev, M. Rakhmatullaev, S. Normatov, K. Shukurov, and M. Abdullaeva, “Integrated Intelligent System for Scientific and Educational Information Retrieval,” in Proceedings of the 15th International Scientific and Practical Conference, Rezekne Academy of Technologies, 2024, pp. 212–219. doi: 10.17770/etr2024vol2.8028.
E. F. Okagbue et al., “A comprehensive overview of artificial intelligence and machine learning in education pedagogy: 21 Years (2000–2021) of research indexed in the scopus database,” Social Sciences and Humanities Open, vol. 8, 2023, doi: 10.1016/j.ssaho.2023.100655.
K. A. Hambarde and H. Proenca, “Information Retrieval: Recent Advances and Beyond,” IEEE Access, vol. 11, pp. 76581–76604, 2023, doi: 10.1109/ACCESS.2023.3295776.
M. Zhang, J. Fan, A. Sharma, and A. Kukkar, “Data mining applications in university information management system development,” Journal of Intelligent Systems, vol. 31, pp. 207–220, 2022, doi: 10.1515/jisys-2022-0006.
C. K. Kreutz and R. Schenkel, “Scientific paper recommendation systems: a literature review of recent publications,” International Journal on Digital Libraries, vol. 23, pp. 335–369, 2022, doi: 10.1007/s00799-022-00339-w.
H. Ezaldeen, S. K. Bisoy, R. Misra, and R. Alatrash, “Semantics aware intelligent framework for content-based e-learning recommendation,” Natural Language Processing Journal, vol. 3, pp. 1–16, 2023, doi: 10.1016/j.nlp.2023.100008.
S. Jha, C. V. Mahamuni, and I. K. Garewal, “Natural Language Processing: A Literature Survey of Approaches, Applications, Current Trends, and Future Directions,” in 2024 Asian Conference on Intelligent Technologies, ACOIT 2024, Institute of Electrical and Electronics Engineers Inc., 2024. doi: 10.1109/ACOIT62457.2024.10941573.
N. M. Gardazi, A. Daud, M. K. Malik, A. Bukhari, T. Alsahfi, and B. Alshemaimri, “BERT applications in natural language processing: a review,” Artif Intell Rev, vol. 58, no. 166, pp. 1–49, 2025, doi: 10.1007/s10462-025-11162-5.
A. Rahali and M. A. Akhloufi, “End-to-End Transformer-Based Models in Textual-Based NLP,” AI, vol. 4, pp. 54–110, 2023, doi: 10.3390/ai4010004.
E. Kotei and R. Thirunavukarasu, “ASystematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning,” Information, vol. 14, pp. 1–25, 2023, doi: 10.3390/info14030187.
H. Wang, J. Li, H. Wu, E. Hovy, and Y. Sun, “Pre-Trained Language Models and Their Applications,” Engineering, vol. 25, pp. 51–65, 2023, doi: 10.1016/j.eng.2022.04.024.
W. Zhang, Y. Yang, T. Akilan, Q. M. J. Wu, and T. Liu, “Fast Transfer Learning Method Using Random Layer Freezing and Feature Refinement Strategy,” IEEE Trans Cybern, vol. 55, no. 1, pp. 234–246, 2024, doi: 10.1109/TCYB.2024.3483068.
L. Wang et al., “Parameter-efficient fine-tuning in large language models: a survey of methodologies,” Artif Intell Rev, vol. 58, pp. 1–64, 2025, doi: 10.1007/s10462-025-11236-4.
A. H. Salman and W. Al-Jawher, “A New Multi-class Classification Method Based on Machine Learning to Document Classification,” in 16th International Conference on Developments in eSystems Engineering (DeSE), IEEE Xplore, 2023, pp. 605–610. doi: 10.1109/DeSE60595.2023.10468839.
R. K. Shah, S. Kumar, and Shashank, “Multilabel News Category Classification using Machine Learning,” in Proceedings of the 8th International Conference on Communication and Electronics Systems (ICCES 2023), IEEE Xplore, 2023, pp. 1245–1250. doi: 10.1109/ICCES57224.2023.10192826.
M. Memoon, M. Umar, S. Rani, M. Khalid, and S. Shaheen, “Twitter News Classification Using Machine Learning,” in 2024 International Conference on Engineering & Computing Technologies (ICECT), IEEE Xplore, 2024, pp. 1–7. doi: 10.1109/ICECT61618.2024.10581173.
A. Onan and H. Alhumyani, “Contextual Hypergraph Networks for Enhanced Extractive Summarization: Introducing Multi-Element Contextual Hypergraph Extractive Summarizer (MCHES),” Applied Sciences, vol. 14, pp. 1–25, 2024, doi: 10.3390/app14114671.
A. R. I. Fauzy and E. B. Setiawan, “Detecting Fake News on Social Media Combined with the CNN Methods,” JURNAL RESTI (RekayasaSistemdanTeknologiInformasi), vol. 7, no. 2, pp. 271–277, 2023, doi: 10.29207/resti.v7i1.4889.
S. A. Zikrina and Fitriyani, “Advancing Hate Speech Detection in Indonesian Language Using Graph Neural Networks and TF-IDF,” Jurnal RESTI, vol. 9, no. 1, pp. 137–145, 2025, doi: 10.29207/resti.v9i1.6179.
A. A. Nabil, D. Das, M. S. Salim, S. Arifeen, and H. M. A. Fattah, “Bangla Emergency Post Classification on Social Media using Transformer Based BERT Models,” in 6th International Conference on Electrical Information and Communication Technology (EICT), IEEE Xplore, 2023, pp. 1–6. doi: 10.1109/EICT61409.2023.10427900.
B. Yu, C. Deng, and L. Bu, “Policy Text Classification Algorithm Based on Bert,” in 11th International Conference of Information and Communication Technology (ICTech), IEEE Xplore, 2022, pp. 488–491. doi: 10.1109/ICTech55460.2022.00103.
R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. A. Almazroi, “A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification,” J Healthc Eng, vol. 2022, pp. 1–17, 2022, doi: 10.1155/2022/3498123.
M. Bilal and A. A. Almazroi, “Effectiveness of Fine-tuned BERT Model in Classification of Helpful and Unhelpful Online Customer Reviews,” Electronic Commerce Research, vol. 23, pp. 2737–2757, 2022, doi: 10.1007/s10660-022-09560-w.
M. Chiny, M. Chihab, O. Bencharef, and Y. Chihab, “Netflix Recommendation System based on TF-IDF and Cosine Similarity Algorithms,” in 2nd International Conference on Big Data, Modelling and Machine Learning (BML 2021), SCITEPRESS, 2022, pp. 15–20. doi: 10.5220/0010727500003101.
S. Al-Otaibi et al., “Cosine similarity-based algorithm for social networking recommendation,” International Journal of Electrical and Computer Engineering (IJECE), vol. 12, no. 2, pp. 1881–1892, 2022, doi: 10.11591/ijece.v12i2.pp1881-1892.
C. G. Reswara, J. Nicolas, M. Ananta, and F. I. Kurniadi, “Anime Recommendation System Using Bert and Cosine Similarity,” in 4th International Conference on Artificial Intelligence and Data Sciences: Discovering Technological Advancement in Artificial Intelligence and Data Science (AiDAS), IEEE Xplore, 2023, pp. 109–113. doi: 10.1109/AiDAS60501.2023.10284693.
S. Sankar, H. Muslihuddeen, S. Ostwal, P. Sathvika, and A. K. Madasamy, “CISER: Customized Institute Specific Search Engine for Retrieving Research Papers,” in 4th International Conference on Innovative Trends in Information Technology (ICITIIT), IEEE Xplore, 2023, pp. 1–5. doi: 10.1109/ICITIIT57246.2023.10068620.
W. I Alhababi and A. W Mulhim, “What are the Embedded Systems,” Int J Innov Sci Res Technol, vol. 9, no. 9, pp. 1585–1587, 2024, doi: 10.38124/ijisrt/ijisrt24sep953.
N. Mayasari, E. Andriani, and A. R. Hibrida, “Revolutionizing Business Operations: A Bibliometric Analysis of Enterprise Systems and Organizational Efficiency,” The Eastasouth Journal of Information System and Computer Science, vol. 1, no. 1, pp. 45–54, 2023, doi: 10.58812/esiscs.v1i01.134.
P. K. Hristova, M. Ivanovic, and B. Diri, “Special issue: Innovations in Intelligent Systems and Applications,” Journal of Information and Telecommunication, vol. 6, no. 1, pp. 1–5, 2022, doi: 10.1080/24751839.2021.1989242.
M. P. Geetha and D. K. Renuka, “Improving the performance of aspect based sentiment analysis using fine-tuned Bert Base Uncased model,” International Journal of Intelligent Networks, vol. 2, pp. 64–69, 2021, doi: 10.1016/j.ijin.2021.06.005.
K. N. Prasanthi, R. E. Madhavi, D. N. S. Sabarinadh, and B. Sravani, “A Novel Approach for Sentiment Analysis on social media using BERT & ROBERTA Transformer-Based Models,” in International Conference for Convergence in Technology (I2CT), IEEE Xplore, 2023, pp. 1–6. doi: 10.1109/I2CT57861.2023.10126206.
X. Li, X. Wang, and H. Liu, “Research on fine-tuning strategy of sentiment analysis model based on BERT,” in 2024 10th International Conference on Smart Computing and Communication (ICSCC), IEEE Xplore, May 2021, pp. 798–802. doi: 10.1109/CISCE52179.2021.9445882.
P. Nabila and E. B. Setiawan, “Adam and AdamW Optimization Algorithm Application on BERT Model for Hate Speech Detection on Twitter,” in 2024 International Conference on Data Science and Its Applications (ICoDSA), IEEE Xplore, 2024, pp. 346–351. doi: 10.1109/ICoDSA62899.2024.10651619.
R. Shaharao et al., “Semantic Similarity in Multi-Source Information Retrieval for Improving Learner Performance,” in International Conference on Smart Computing and Communication (ICSCC), IEEE Xplore, 2024, pp. 1–6. doi: 10.1109/ICSCC62041.2024.10690443.
M. Heydarian, T. E. Doyle, and R. Samavi, “MLCM: Multi-Label Confusion Matrix,” IEEE Access, vol. 10, pp. 19083–19095, 2022, doi: 10.1109/ACCESS.2022.3151048.
B. D. Wijanarko et al., “AI-Based Feature Extraction and Cosine Similarity for Automation of Student Learning Assessment,” in 24th International Arab Conference on Information Technology (ACIT), IEEE Xplore, 2023. doi: 10.1109/ACIT58888.2023.10453927.
M. S. Sungkar, R. Wulandari, S. Syamsidar, W. Firdaus, A. Andiyan, and R. Rahim, “Enhancing E-commerce Recommendation Accuracy Using KNN and Hybrid Approaches: An Empirical Study,” in Advances in Intelligent Manufacturing and Robotics, Springer, 2023, pp. 293–300. doi: 10.1007/978-981-99-8498-5_22.
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
