Behavioral Analysis of Semantic Similarity Metrics under Transformer-Based Representations

Authors

  • Musthofa Galih Pradana Universitas Pembangunan Nasional Veteran Jakarta
  • Nindy Irzavika Universitas Pembangunan Nasional Veteran Jakarta
  • Nurhuda Maulana Universitas Pembangunan Nasional Veteran Jakarta
  • Syaila Ananta Karenina Universitas Pembangunan Nasional Veteran Jakarta
  • Salma Ashiila Rabbani Universitas Pembangunan Nasional Veteran Jakarta

DOI:

https://doi.org/10.29207/resti.v10i2.7490

Keywords:

cosine similarity, knowledge extraction, rank shift analysis, semantic similarity, transformer-based embeddings

Abstract

Knowledge extraction has several approaches such as traditional approaches that rely on lexical representation capabilities, one of which is TF-IDF whose implementation can be combined with several classic similarity metrics such as cosine similarity and dice coefficient similarity. In addition to applying the lexical representation approach, this study tries to apply it to a more modern type of representation, namely transformer-embedding-based contextual representation. The data used in this study is abstract document data of students' theses. The findings of the study show that contextual embedding changes the behavior of similarity values. The results of the analysis showed an average ranking shift of 6.70 positions. The test results showed a weak rating correlation value (Spearman = 0.22; Kendall = 0.146), and the high-ranking alignment measured with NDCG (0.97), which shows structural differences in the order of similarity between lexical and contextual representations. Other findings also show that the gap in ranking produced by the two representations used is quite far due to the difference in the mechanism and working pattern of the two representations that are far different, the selection of the type of representation must be on the characteristics of the data to be processed, if looking at the character of the text data in academic documents, the selection of contextual representations based on transformer embedding will be more suitable with contextual understanding to avoid the use of variations in words avoid plagiarism detection when applying a semantic-based representation approach.

Downloads

Download data is not yet available.

References

M. Botto-Tobar, A. Serebrenik, and M. G. J. Van den Brand, “Cross-Language Plagiarism Detection: Methods, Tools, and Challenges: A Systematic Review,” Int J Adv Sci Eng Inf Technol, vol. 12, no. 2, pp. 589–599, Apr. 2022, doi: https://doi.org/10.18517/ijaseit.12.2.14711.

G. B. Vitor, A. C. Victorino, and J. V. Ferreira, “Modeling evidential grids using semantic context information for dynamic scene perception,” Knowl Based Syst, vol. 215, p. 106777, Mar. 2021, doi: https://doi.org/10.1016/j.knosys.2021.106777.

B. Klimova, M. Pikhart, A. D. Benites, C. Lehr, and C. Sanchez-Stockhammer, “Neural machine translation in foreign language teaching and learning: a systematic review,” Educ Inf Technol (Dordr), vol. 28, no. 1, pp. 663–682, Jan. 2023, doi: https://doi.org/10.1007/s10639-022-11194-2.

X. Huang, C. Wang, X. Tian, Z. Li, C. Zhao, and M. Xiao, “A Multi-Hop Semantic Communication System Enhanced by Semantic Importance,” IEEE Access, vol. 13, pp. 140685–140693, 2025, doi: https://doi.org/10.1109/ACCESS.2025.3593341.

D. Chandrasekaran and V. Mago, “Evolution of Semantic Similarity A Survey,” ACM Comput Surv, vol. 54, no. 2, pp. 1–37, Mar. 2022, doi: https://doi.org/10.1145/3440755

P. Yang, H. Wang, J. Yang, Z. Qian, Y. Zhang, and X. Lin, “Deep Learning Approaches for Similarity Computation: A Survey,” IEEE Trans Knowl Data Eng, vol. 36, no. 12, pp. 7893–7912, Dec. 2024, doi: https://doi.org/10.1109/TKDE.2024.3422484.

N. M. Gardazi, A. Daud, M. K. Malik, A. Bukhari, T. Alsahfi, and B. Alshemaimri, “BERT applications in natural language processing: a review,” Artif Intell Rev, vol. 58, no. 6, p. 166, Mar. 2025, doi: https://doi.org/10.1007/s10462-025-11162-5.

M. Galih Pradana, N. Irzavika, N. Maulana, J. Mu, and K. Wari, “Performance Improvement of Cosine Similarity Algorithm with Bidirectional Encoder Representations from Transformers on Abstract Document Similarity Detection,” JOIV : International Journal on Informatics Visualization, vol. 9, no. 2, pp. 824–830, Mar. 2025, doi: http://dx.doi.org/10.62527/joiv.9.2.2853.

S. Zoupanos, S. Kolovos, A. Kanavos, O. Papadimitriou, and M. Maragoudakis, “Efficient comparison of sentence embeddings,” in Proceedings of the 12th Hellenic Conference on Artificial Intelligence, New York, NY, USA: ACM, Sep. 2022, pp. 1–6. doi: https://doi.org/10.1145/3549737.354975.

T. Gao, X. Yao, and D. Chen, “SimCSE: Simple Contrastive Learning of Sentence Embeddings,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 6894–6910. doi: https://doi.org/10.18653/v1/2021.emnlp-main.552.

K. Abdalgader, A. A. Matroud, and K. Hossin, “Experimental study on short-text clustering using transformer-based semantic similarity measure,” PeerJ Comput Sci, vol. 10, p. e2078, May 2024, doi: https://doi.org/10.7717/peerj-cs.2078.

H. Licht, “Cross-Lingual Classification of Political Texts Using Multilingual Sentence Embeddings,” Political Analysis, vol. 31, no. 3, pp. 366–379, Jul. 2023, doi: https://doi.org/10.1017/pan.2022.29.

F. F. Ijebu, Y. Liu, C. Sun, and P. U. Usip, “Soft cosine and extended cosine adaptation for pre-trained language model semantic vector analysis,” Appl Soft Comput, vol. 169, p. 112551, Jan. 2025, doi: https://doi.org/10.1016/j.asoc.2024.112551.

N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 3980–3990. doi: https://doi.org/10.18653/v1/D19-1410.

K. Nassiri and M. Akhloufi, “Transformer models used for text-based question answering systems,” Applied Intelligence, vol. 53, no. 9, pp. 10602–10635, May 2023, doi: https://doi.org/10.1007/s10489-022-04052-8.

P. Jiang and X. Cai, “A Survey of Text-Matching Techniques,” Information, vol. 15, no. 6, p. 332, Jun. 2024, doi: https://doi.org/10.3390/info15060332.

L. Zhang, “Features extraction based on Naive Bayes algorithm and TF-IDF for news classification,” PLoS One, vol. 20, no. 7, p. e0327347, Jul. 2025, doi: https://doi.org/10.1371/journal.pone.0327347.

H. D. Abubakar and M. Umar, “Sentiment Classification: Review of Text Vectorization Methods: Bag of Words, Tf-Idf, Word2vec and Doc2vec,” SLU Journal of Science and Technology, vol. 4, no. 1 & 2, pp. 27–33, Aug. 2022, doi: https://doi.org/10.56471/slujst.v4i.266.

S. Akuma, T. Lubem, and I. T. Adom, “Comparing Bag of Words and TF-IDF with different models for hate speech detection from live tweets,” International Journal of Information Technology, vol. 14, no. 7, pp. 3629–3635, Dec. 2022, doi: https://doi.org/10.1007/s41870-022-01096-4.

F. Lan, “Research on Text Similarity Measurement Hybrid Algorithm with Term Semantic Information and TF-IDF Method,” Advances in Multimedia, vol. 2022, pp. 1–11, Apr. 2022, doi: https://doi.org/10.1155/2022/7923262.

D. Meidelfi, - Yulherniwati, I. Rahmayuni, T. Hidayat, and D. Chandra, “TF-IDF Implementation for Similarity Checker on The Final Project Title,” International Journal of Advanced Science Computing and Engineering, vol. 3, no. 1, pp. 40–52, 2021, doi: https://doi.org/10.30630/ijasce.3.1.3.

V. Raina et al., “Tackling Bias in the Dice Similarity Coefficient: Introducing NDSC for White Matter Lesion Segmentation,” in 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), IEEE, Apr. 2023, pp. 1–5. doi: https://doi.org/10.1109/ISBI53787.2023.10230755.

U. ur Rehman and T. Mahmood, “The generalized dice similarity measures for bipolar complex fuzzy set and its applications to pattern recognition and medical diagnosis,” Computational and Applied Mathematics, vol. 41, no. 6, p. 265, Sep. 2022, doi: https://doi.org/10.1007/s40314-022-01948-6.

A. Levy, B. R. Shalom, and M. Chalamish, “A guide to similarity measures and their data science applications,” J Big Data, vol. 12, no. 1, p. 188, Jul. 2025, doi: https://doi.org/10.1186/s40537-025-01227-1.

B. Pavlyshenko and M. Stasiuk, “SEMANTIC SIMILARITY ANALYSIS USING TRANSFORMER-BASED SENTENCE EMBEDDINGS,” Electronics and Information Technologies, vol. 30, p. 803, 2025, doi: https://doi.org/10.30970/eli.30.4.

D. Wang and S. Zhang, “Large language models in medical and healthcare fields: applications, advances, and challenges,” Artif Intell Rev, vol. 57, no. 11, p. 299, Sep. 2024, doi: https://10.1007/s10462-024-10921-0.

S. Agarwala, A. Anagawadi, and R. M. Reddy Guddeti, “Detecting Semantic Similarity of Documents Using Natural Language Processing,” Procedia CIRP, vol. 189, pp. 128–135, 2021, doi: https://doi.org/10.1016/j.procs.2021.05.076.

B. Ihnaini, B. Abuhaija, E. A. Mills, and M. Mahmuddin, “Semantic similarity on multimodal data: A comprehensive survey with applications,” Journal of King Saud University - Computer and Information Sciences, vol. 36, no. 10, p. 102263, Dec. 2024, doi: https://doi.org/10.1016/j.jksuci.2024.102263.

J. Guan, B. Chen, and S. Yu, “A hybrid similarity model for mitigating the cold-start problem of collaborative filtering in sparse data,” Expert Syst Appl, vol. 249, p. 123700, Sep. 2024, doi: https://doi.org/10.1016/j.eswa.2024.123700.

S. W. Brown, J. Bonn, G. Kazeminejad, A. Zaenen, J. Pustejovsky, and M. Palmer, “Semantic Representations for NLP Using VerbNet and the Generative Lexicon,” Front Artif Intell, vol. 5, Apr. 2022, doi: https://doi.org/10.3389/frai.2022.821697.

A. M. Elkhatat, “Evaluating the authenticity of ChatGPT responses: a study on text-matching capabilities,” International Journal for Educational Integrity, vol. 19, no. 1, p. 15, Aug. 2023, doi: https://10.1007/s40979-023-00137-0.

W. Mu and K. H. Lim, “Modelling Text Similarity: A Survey,” in Proceedings of the International Conference on Advances in Social Networks Analysis and Mining, New York, NY, USA: ACM, Nov. 2023, pp. 698–705. doi: https://doi.org/10.1145/3625007.3627305.

Downloads

Published

2026-04-20

Issue

Section

Artificial Intelligence

How to Cite

Pradana, M. G., Irzavika, N. ., Maulana, N. ., Karenina, S. A. ., & Rabbani , S. A. . (2026). Behavioral Analysis of Semantic Similarity Metrics under Transformer-Based Representations. Jurnal RESTI (Rekayasa Sistem Dan Teknologi Informasi), 10(2), 332-340. https://doi.org/10.29207/resti.v10i2.7490