Benchmarking Transformer Models for Sentiment Analysis on YouTube User Comments

Li-Hua Li; Agus Cahyo Nugroho

doi:10.29207/resti.v10i5.7752

Authors

Li-Hua Li Chaoyang University of Technology
Agus Cahyo Nugroho Universitas Katolik Soegijapranata

DOI:

https://doi.org/10.29207/resti.v10i5.7752

Keywords:

bert, distilbert, roberta, sentiment analysis, twitter-roberta

Abstract

Social media has become one of the most important platforms for public expression due to its ability to provide users with an unprecedented scale for sharing opinions, sentiments, and experiences. Consequently, social media platforms provide valuable insights into public perception and user sentiment across various topics. This study presents a comparative evaluation of five transformer-based models, namely DistilBERT, BERT, BERTweet, RoBERTa, and Twitter-RoBERTa, on a cleaned and preprocessed YouTube Comments Dataset for sentiment analysis tasks. All models were evaluated under identical experimental settings and preprocessing procedures to reduce textual noise and improve data quality. Each transformer model was fine-tuned under common training conditions to assess its effectiveness in handling informal and user-generated social media content. Experimental results show that Twitter-RoBERTa outperforms other approaches and achieves the best overall accuracy: 90.13% (F1-score: 90.01%), demonstrating its ability to learn from contextual nuances and social media-specific language patterns. In addition, McNemar statistical analysis further supports the statistical significance of many performance differences between the models evaluated including those between Twitter-RoBERTa and smaller general-purpose models such as DistilBERT and BERT. These results demonstrate the value of using domain-specific pretraining on noisy social media text to improve sentiment classification performance and that transformer architectures built with fine-tuning in mind can effectively capture platform specific aspects of linguistics. In summary, this work presents a strong benchmark for transformer-based-sentiment-analysis on YouTube comments and valuable information for future research in social media sentiment analysis.

Downloads

Download data is not yet available.

References

[1] D. Drašković and S. Milanović, “Aspect-based sentiment analysis of user-generated content from a microblogging platform,” Journal of Big Data, vol. 12, no. 1, p. 186, 2025, https://doi.org/10.1186/s40537-025-01244-0.

[2] N. M. Gardazi, A. Daud, M. K. Malik, A. Bukhari, T. Alsahfi, and B. Alshemaimri, "BERT applications in natural language processing: a review," Artif. Intell. Rev., vol. 58, no. 6, pp. 1–49, 2025, https://doi.org/10.1007/s10462-025-11162-5.

[3] C. Liu, A. Arulappan, R. Naha, A. Mahanti, J. Kamruzzaman, and I. H. Ra, “Large language models and sentiment analysis in financial markets: A review, datasets, and case study,” IEEE Access, vol. 12, pp. 134041–134061, 2024, https://doi.org/10.1109/ACCESS.2024.3445413.

[4] A. Alamsyah and Y. Sagama, "Empowering Indonesian internet users: An approach to counter online toxicity and enhance digital well-being," Intell. Syst. Appl., vol. 22, p. 200394, 2024, https://doi.org/10.1016/j.iswa.2024.200394.

[5] G. Fenza, V. Loia, C. Stanzione, and M. Di Gisi, "Robustness of models addressing Information Disorder: A comprehensive review and benchmarking study," Neurocomputing, p. 127951, 2024, https://doi.org/10.1016/j.neucom.2024.127951.

[6] B. Liu, “Sentiment analysis: A fascinating problem,” in Sentiment Analysis and Opinion Mining, Cham, Switzerland: Springer International Publishing, 2022, pp. 1–8, https://doi.org/10.1007/978-3-031-02145-9_1.

[7] M. Kastrati, Z. Kastrati, A. Shariq Imran, and M. Biba, “Leveraging distant supervision and deep learning for Twitter sentiment and emotion classification,” Journal of Intelligent Information Systems, vol. 62, no. 4, pp. 1045–1070, 2024, https://doi.org/10.1007/s10844-024-00845-0.

[8] S. Yuan, Z. Zou, H. Zhou, Y. Ren, J. Wu, and K. Yan, “SECNN: Squeeze-and-excitation convolutional neural network for sentence classification,” IEEE Access, vol. 13, pp. 42858–42865, 2025, https://ieeexplore.ieee.org/document/11070808.

[9] Q. Guo, Z. He, and Z. Wang, “Assessing the effectiveness of long short-term memory and artificial neural network in predicting daily ozone concentrations in Liaocheng City,” Scientific Reports, vol. 15, no. 1, p. 6798, 2025, https://doi.org/10.1038/s41598-025-91329-w.

[10] M. Kowsher, A. A. Sami, N. J. Prottasha, M. S. Arefin, P. K. Dhar, and T. Koshiba, “Bangla-BERT: Transformer-based efficient model for transfer learning and language understanding,” IEEE Access, vol. 10, pp. 91855–91870, 2022, https://doi.org/10.1109/ACCESS.2022.3197662.

[11] A. Benayas, M. A. Sicilia, and M. Mora-Cantallops, “A comparative analysis of encoder-only and decoder-only models in intent classification and sentiment analysis: Navigating the trade-offs in model size and performance,” Language Resources and Evaluation, vol. 59, no. 3, pp. 2007–2030, 2025, https://doi.org/10.1007/s10579-024-09796-y.

[12] K. Du, F. Xing, R. Mao, and E. Cambria, “Financial sentiment analysis: Techniques and applications,” ACM Computing Surveys, vol. 56, no. 9, pp. 1–42, 2024, https://doi.org/10.1145/3649451.

[13] J. Liang, L. Cao, S. Madden, Z. Ives, and G. Li, “RITA: Group attention is all you need for timeseries analytics,” Proceedings of the ACM on Management of Data, vol. 2, no. 1, pp. 1–28, 2024, https://doi.org/10.1145/3639317.

[14] B. Tahayna and R. K. Ayyasamy, "Applying English idiomatic expressions to classify deep sentiments in COVID-19 tweets," Comput. Syst. Sci. Eng., vol. 47, no. 1, 2023, http://dx.doi.org/10.32604/csse.2023.036648.

[15] J. Sirusstara, N. Alexander, A. Alfarisy, S. Achmad, and R. Sutoyo, “Clickbait headline detection in Indonesian news sites using robustly optimized BERT pre-training approach (RoBERTa),” in Proc. 2022 3rd Int. Conf. Artificial Intelligence and Data Sciences (AiDAS), 2022, pp. 1–6, IEEE, https://doi.org/10.1109/AiDAS56890.2022.9918678.

[16] W. M. N. W. B. Manike, R. A. Wijesinghe, and A. M. T. N. Adasuriya, “Optimizing DistilBERT for low-resource sentiment analysis: A study on global unstructured L1-norm pruning and accuracy-efficiency trade-offs,” in Proc. 2026 6th Int. Conf. Advanced Research in Computing (ICARC), 2026, pp. 1–6, IEEE, https://doi.org/10.1109/ICARC68737.2026.11453647.

[17] H. Su, J. Kasai, C. H. Wu, W. Shi, T. Wang, J. Xin, et al., “Selective annotation makes language models better few-shot learners,” arXiv preprint arXiv:2209.01975, 2022, https://doi.org/10.48550/arXiv.2209.01975.

[18] L. Gorenstein, E. Konen, M. Green, and E. Klang, “Bidirectional encoder representations from transformers in radiology: A systematic review of natural language processing applications,” Journal of the American College of Radiology, vol. 21, no. 6, pp. 914–941, 2024, https://doi.org/10.1016/j.jacr.2024.01.012.

[19] M. S. Islam, L. Xiangdong, and J. Ahmed, “BERT: Advancements in language understanding for different NLP tasks: Challenges and future perspectives,” Journal of Electrical Systems and Information Technology, vol. 13, no. 1, Art. no. 49, 2026, https://doi.org/10.1186/s43067-026-00341-1.

[20] B. Li, C. Li, J. Sun, X. Zeng, X. Chen, and J. Zheng, “ERNIE-UIE: Advancing information extraction in Chinese medical knowledge graph,” PLOS ONE, vol. 20, no. 5, p. e0325082, 2025, https://doi.org/10.1371/journal.pone.0325082.

[21] B. Wang, Q. Xie, J. Pei, Z. Chen, P. Tiwari, Z. Li, and J. Fu, “Pre-trained language models in biomedical domain: A systematic survey,” ACM Computing Surveys, vol. 56, no. 3, pp. 1–52, 2023, https://doi.org/10.1145/3611651.

[22] D. Vithanage, P. Yu, L. Wang, and C. Deng, “Contextual word embedding for biomedical knowledge extraction: a rapid review and case study,” Journal of Healthcare Informatics Research, vol. 8, no. 1, pp. 158–179, 2024, https://doi.org/10.1007/s41666-023-00157-y.

[23] Y. Z. Vakili, A. Fallah, and H. Sajedi, “Distilled BERT model in natural language processing,” in Proc. 2024 14th Int. Conf. Computer and Knowledge Engineering (ICCKE), 2024, pp. 243–250, IEEE, https://doi.org/10.1109/ICCKE65377.2024.10874673.

[24] M. Soyeb, M. Mahdin, S. S. Sadaf, M. M. Azwad, and A. Abrar, “TYMBERT: Tiny yet mighty—fine-tuned and compressed TinyBERT for high-accuracy, low-resource NLP,” in Proc. 2025 2nd Int. Conf. Next-Generation Computing, IoT and Machine Learning (NCIM), 2025, pp. 1–6, IEEE, https://doi.org/10.1109/NCIM65934.2025.11160230.

[25] G. Tucudean, M. Bucos, B. Dragulescu, and C. D. Caleanu, “Natural language processing with transformers: a review,” PeerJ Computer Science, vol. 10, p. e2222, 2024, https://doi.org/10.7717/peerj-cs.2222.

[26] A. B. Alawi and F. Bozkurt, “A hybrid machine learning model for sentiment analysis and satisfaction assessment with Turkish universities using Twitter data,” Decision Analytics Journal, vol. 11, p. 100473, 2024, https://doi.org/10.1016/j.dajour.2024.100473.

[27] S. S. Tiwari, H. R. Goyal, A. Shrivastava, K. Yadav, and A. L. N. Rao, “Clinical text analysis with natural language processing: A BERT-based approach,” in Proc. 2024 Int. Conf. Communication, Computer Sciences and Engineering (IC3SE), 2024, pp. 1382–1388, IEEE, https://doi.org/10.1109/IC3SE62002.2024.10593504.

[28] N. Kumar, P. N. Bhat, B. R. Pushpa, and N. R. Sreekumar, “Hate speech detection and analysis using NLP: A hybrid DB-BiLSTM approach,” in Proc. 2025 Control Instrumentation System Conference (CISCON), 2025, pp. 1–6, IEEE, https://doi.org/10.1109/CISCON66933.2025.11337424.

[29] S. Zhu, S. Gu, S. Li, L. Xu, and D. Xiong, “Mining parallel sentences from internet with multi-view knowledge distillation for low-resource language pairs,” Knowledge and Information Systems, vol. 66, no. 1, pp. 187–209, 2024, https://doi.org/10.1007/s10115-023-01925-3.

[30] S. Chen, Y. Zhang, and Q. Yang, “Multi-task learning in natural language processing: An overview,” ACM Computing Surveys, vol. 56, no. 12, pp. 1–32, 2024, https://doi.org/10.1145/3663363.

[31] K. L. Tan, C. P. Lee, K. M. Lim, and K. S. M. Anbananthen, “Sentiment analysis with ensemble hybrid deep learning model,” IEEE Access, vol. 10, pp. 103694–103704, 2022, https://doi.org/10.1109/ACCESS.2022.3210182.

[32] M. AnsariShiri and N. Mansouri, “A comparative evaluation of transformer models for medical abstract classification,” Multimedia Tools and Applications, vol. 85, no. 3, p. 183, 2026, https://doi.org/10.1007/s11042-026-21140-w.

[33] M. Siino, M. Falco, D. Croce, and P. Rosso, “Exploring LLMs applications in law: A literature review on current legal NLP approaches,” IEEE Access, vol. 13, pp. 18253–18276, 2025, https://doi.org/10.1109/ACCESS.2025.3533217.

[34] F. Carneiro, D. Vianna, J. Carvalho, A. Plastino, and A. Paes, “BERTweet.BR: A pre-trained language model for tweets in Portuguese,” Neural Computing and Applications, vol. 37, no. 6, pp. 4363–4385, 2025, https://doi.org/10.1007/s00521-024-10711-3.

[35] A. Maghsoudi et al., “Sentiment Analysis of Insomnia-Related Tweets via a Combination of Transformers Using Dempster-Shafer Theory: Pre– and Peri–COVID-19 Pandemic Retrospective Study,” J. Med. Internet Res., vol. 24, no. 12, p. e41517, Dec. 2022. https://doi.org/10.2196/41517.

[36] M. A. Jahin et al., "A hybrid transformer and attention based recurrent neural network for robust and interpretable sentiment analysis of tweets," Scientific Reports, vol. 14, no. 1, p. 24882, 2024, https://doi.org/10.1038/s41598-024-76079-5.

[37] L. He, "Enhanced twitter sentiment analysis with dual joint classifier integrating RoBERTa and BERT architectures,"

Frontiers in Physics, vol. 12, p. 1477714, 2024, doi: 10.3389/fphy.2024.1477714.

[38] D. Q. Nguyen, T. Vu, and A. T. Nguyen, “BERTweet: A pre-trained language model for English Tweets,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Oct. 2020, pp. 9–14, https://doi.org/10.18653/v1/2020.emnlp-demos.2

[39] F. Barbieri et al., “TweetEval: Unified benchmark and comparative evaluation for tweet classification,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, https://doi.org/10.18653/v1/2020.findings-emnlp.148.

[40] S. Minaee, N. Kalchbrenner, Erik Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep learning-based text classification: A comprehensive review,” ACM Computing Surveys (CSUR), vol. 54, no. 3, pp. 1–40, 2021, https://doi.org/10.1145/3439726.