Indonesian Hate Speech Detection under Class Imbalance Using a Soft-Voting Ensemble of IndoBERTweet and IndoRoBERTa
DOI:
https://doi.org/10.29207/resti.v10i3.7681Keywords:
hate speech detection, ensemble learning, imbalanced data, indonesian text, transformer modelsAbstract
Hate speech detection on Indonesian social media remains challenging due to the coexistence of formal and highly colloquial language, as well as the moderate class imbalance typical of real-world datasets. Models trained under these conditions often skew toward the majority class and generalize poorly across linguistic registers. This study investigates whether a simple, training-free model-level ensemble can improve Indonesian hate speech detection under such conditions without resampling the data. IndoBERTweet and IndoRoBERTa, pretrained respectively on informal Twitter text and broader formal corpora, serve as complementary base models, and their class probabilities are combined through equal-weight soft voting. On the Indonesian Hate Speech Superset (N = 14,306), evaluated across five random seeds with paired significance testing, the soft-voting ensemble attains a macro-averaged F1 of 0.898 ± 0.003 and a macro recall of 0.899 ± 0.003. It significantly outperforms a TF-IDF SVM baseline and the IndoRoBERTa base model, while showing no significant difference from the stronger IndoBERTweet base model and a trained logistic-regression stacking ensemble. Notably, the ensemble matches the stacking ensemble without any additional training stage or meta-learner, and a calibration analysis shows it improves probability calibration over both base models. These results indicate that equal-weight probability averaging is a simple, reproducible, and competitive strategy for Indonesian hate speech detection under moderate class imbalance.
Downloads
References
S. Stieglitz and B. Ross, “The Impact of Social Media on Social Cohesion: A Double-Edged Sword,” MaC, vol. 10, no. 2, pp. 104–107, May 2022, doi: 10.17645/mac.v10i2.5792.
J. B. Walther, “Social media and online hate,” Current Opinion in Psychology, vol. 45, p. 101298, Jun. 2022, doi: 10.1016/j.copsyc.2021.12.010.
M. Retta, “A pragmatic and discourse analysis of hate words on social media,” Internet Pragmatics, vol. 6, no. 2, pp. 197–218, Dec. 2023, doi: 10.1075/ip.00096.ret.
D. Hartmann, A. Oueslati, D. Staufer, L. Pohlmann, S. Munzert, and H. Heuer, “Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Yokohama Japan: ACM, Apr. 2025, pp. 1–26. doi: 10.1145/3706598.3713998.
J. Gracia-Calandín and L. Suárez-Montoya, “The eradication of hate speech on social media: a systematic review,” JICES, vol. 21, no. 4, pp. 406–421, Nov. 2023, doi: 10.1108/JICES-11-2022-0098.
C. Silva, “Fighting Against Hate Speech: A Case for Harnessing Interactive Digital Counter-Narratives,” in Interactive Storytelling, vol. 14383, L. Holloway-Attaway and J. T. Murray, Eds., in Lecture Notes in Computer Science, vol. 14383. , Cham: Springer Nature Switzerland, 2023, pp. 159–174. doi: 10.1007/978-3-031-47655-6_10.
A. M. Kumar and S. Gupta, “Governance of Social Media Platforms: A Literature Review,” PAJAIS, vol. 15, pp. 56–86, Mar. 2023, doi: 10.17705/1pais.15103.
A. Rawat, S. Kumar, and S. S. Samant, “Hate speech detection in social media: Techniques, recent trends, and future challenges,” WIREs Computational Stats, vol. 16, no. 2, p. e1648, Mar. 2024, doi: 10.1002/wics.1648.
F. Alkomah and X. Ma, “A Literature Review of Textual Hate Speech Detection Methods and Datasets,” Information, vol. 13, no. 6, p. 273, May 2022, doi: 10.3390/info13060273.
A. Gandhi et al., “Hate speech detection: A comprehensive review of recent works,” Expert Systems, vol. 41, no. 8, p. e13562, Aug. 2024, doi: 10.1111/exsy.13562.
E. W. Pamungkas, D. G. P. Putri, and A. Fatmawati, “Hate Speech Detection in Bahasa Indonesia: Challenges and Opportunities,” IJACSA, vol. 14, no. 6, 2023, doi: 10.14569/IJACSA.2023.01406125.
M. O. Ibrohim and I. Budi, “Hate speech and abusive language detection in Indonesian social media: Progress and challenges,” Heliyon, vol. 9, no. 8, p. e18647, Aug. 2023, doi: 10.1016/j.heliyon.2023.e18647.
H. Margono, M. Saud, and A. Ashfaq, “Dynamics of hate speech in social media: insights from Indonesia,” GKMC, Aug. 2024, doi: 10.1108/GKMC-11-2023-0464.
A. D. Sanya and L. H. Suadaa, “Handling Imbalanced Dataset on Hate Speech Detection in Indonesian Online News Comments,” in 2022 10th International Conference on Information and Communication Technology (ICoICT), Bandung, Indonesia: IEEE, Aug. 2022, pp. 380–385. doi: 10.1109/ICoICT55009.2022.9914883.
S. Riyadi, A. Divayu Andriyani, and S. Noraini Sulaiman, “Improving Hate Speech Detection Using Double-Layers Hybrid CNN-RNN Model on Imbalanced Dataset,” IEEE Access, vol. 12, pp. 159660–159668, 2024, doi: 10.1109/ACCESS.2024.3487433.
R. Narula and P. Chaudhary, “A comprehensive review on detection of hate speech for multi-lingual data,” Soc. Netw. Anal. Min., vol. 14, no. 1, p. 244, Jan. 2025, doi: 10.1007/s13278-024-01401-y.
J.-G. Gaudreault and P. Branco, “Empirical analysis of performance assessment for imbalanced classification,” Mach Learn, vol. 113, no. 8, pp. 5533–5575, Aug. 2024, doi: 10.1007/s10994-023-06497-5.
C. Padurariu and M. E. Breaban, “Dealing with Data Imbalance in Text Classification,” Procedia Computer Science, vol. 159, pp. 736–745, 2019, doi: 10.1016/j.procs.2019.09.229.
D. Dablain, B. Krawczyk, and N. V. Chawla, “DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data,” IEEE Trans. Neural Netw. Learning Syst., vol. 34, no. 9, pp. 6390–6404, Sep. 2023, doi: 10.1109/TNNLS.2021.3136503.
Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-Balanced Loss Based on Effective Number of Samples,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA: IEEE, Jun. 2019, pp. 9260–9269. doi: 10.1109/CVPR.2019.00949.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal Loss for Dense Object Detection,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, Oct. 2017, pp. 2999–3007. doi: 10.1109/ICCV.2017.324.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” in Proceedings of the 34th International Conference on Machine Learning, D. Precup and Y. W. Teh, Eds., in Proceedings of Machine Learning Research, vol. 70. PMLR, Aug. 2017, pp. 1321–1330. [Online]. Available: https://proceedings.mlr.press/v70/guo17a.html
M. Subramanian, V. Easwaramoorthy Sathiskumar, G. Deepalakshmi, J. Cho, and G. Manikandan, “A survey on hate speech detection and sentiment analysis using machine learning and deep learning models,” Alexandria Engineering Journal, vol. 80, pp. 110–121, Oct. 2023, doi: 10.1016/j.aej.2023.08.038.
S. S. Roy, A. Roy, P. Samui, M. Gandomi, and A. H. Gandomi, “Hateful Sentiment Detection in Real-Time Tweets: An LSTM-Based Comparative Approach,” IEEE Trans. Comput. Soc. Syst., vol. 11, no. 4, pp. 5028–5037, Aug. 2024, doi: 10.1109/TCSS.2023.3260217.
S. Nouas, F. Boumahdi, A. Madani, and F. Berrhail, “Cyberbullying: A BERT Bi-LSTM Solution for Hate Speech Detection,” in Proceedings of the International Conference on Applied Cybersecurity (ACS) 2023, H. Zantout and H. Ragab Hassen, Eds., in Lecture Notes in Networks and Systems, vol. 760. Cham: Springer Nature Switzerland, 2023, pp. 57–65. doi: 10.1007/978-3-031-40598-3_7.
M. Tonneau, D. Liu, S. Fraiberger, R. Schroeder, S. Hale, and P. Röttger, “From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets,” in Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), Mexico City, Mexico: Association for Computational Linguistics, 2024, pp. 283–311. doi: 10.18653/v1/2024.woah-1.23.
N. I. Pratiwi, I. Budi, and I. Alfina, “Hate Speech Detection on Indonesian Instagram Comments using FastText Approach,” in 2018 International Conference on Advanced Computer Science and Information Systems (ICACSIS), Yogyakarta: IEEE, Oct. 2018, pp. 447–450. doi: 10.1109/ICACSIS.2018.8618182.
M. O. Ibrohim and I. Budi, “Multi-label Hate Speech and Abusive Language Detection in Indonesian Twitter,” in Proceedings of the Third Workshop on Abusive Language Online, Florence, Italy: Association for Computational Linguistics, 2019, pp. 46–57. doi: 10.18653/v1/W19-3506.
I. Alfina, R. Mulia, M. I. Fanany, and Y. Ekanata, “Hate speech detection in the Indonesian language: A dataset and preliminary study,” in 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS), Bali: IEEE, Oct. 2017, pp. 233–238. doi: 10.1109/ICACSIS.2017.8355039.
A. Muhamad Faza, Y. Sibaroni, and S. S. Prasetiyowati, “A Comparative Study on Handling Imbalanced Data in Indonesian Hate Speech Detection Using FastText and BiLSTM,” ijoict, vol. 11, no. 2, pp. 136–149, Dec. 2025, doi: 10.21108/ijoict.v11i2.9513.
G. Ramos et al., “A comprehensive review on automatic hate speech detection in the age of the transformer,” Soc. Netw. Anal. Min., vol. 14, no. 1, p. 204, Oct. 2024, doi: 10.1007/s13278-024-01361-3.
A. Albladi et al., “Hate Speech Detection Using Large Language Models: A Comprehensive Review,” IEEE Access, vol. 13, pp. 20871–20892, 2025, doi: 10.1109/ACCESS.2025.3532397.
P. A. Mufva, K. H. Chandra, K. F. Aji, I. A. Iswanto, and S. Joddy, “Performance comparison of deep learning approaches for Indonesian twitter hate speech detection using IndoBERTweet embedding,” Procedia Computer Science, vol. 269, pp. 1663–1671, 2025, doi: 10.1016/j.procs.2025.09.109.
M. Ridha, D. Nurjanah, and M. Rakha, “Multilabel Classification Abusive Language and Hate Speech on Indonesian Twitter Using Transformer Model: IndoBERTweet & IndoRoBERTa,” in 2024 International Conference on Intelligent Cybernetics Technology & Applications (ICICyTA), Bali, Indonesia: IEEE, Dec. 2024, pp. 48–54. doi: 10.1109/ICICYTA64807.2024.10912874.
F. Koto, J. H. Lau, and T. Baldwin, “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021, pp. 10660–10668. doi: 10.18653/v1/2021.emnlp-main.833.
“flax-community/indonesian-roberta-base · Hugging Face.” Accessed: Mar. 30, 2026. [Online]. Available: https://huggingface.co/flax-community/indonesian-roberta-base
A. M. U. D. Khanday, S. T. Rabani, Q. R. Khan, and S. H. Malik, “Detecting twitter hate speech in COVID-19 era using machine learning and ensemble learning techniques,” International Journal of Information Management Data Insights, vol. 2, no. 2, p. 100120, Nov. 2022, doi: 10.1016/j.jjimei.2022.100120.
P. Garg, M. K. Sharma, and P. Kumar, “Improving Hate Speech Classification Through Ensemble Learning and Explainable AI Techniques,” Arab J Sci Eng, vol. 50, no. 15, pp. 11631–11644, Aug. 2025, doi: 10.1007/s13369-024-09540-2.
O. Sagi and L. Rokach, “Ensemble learning: A survey,” WIREs Data Min & Knowl, vol. 8, no. 4, p. e1249, Jul. 2018, doi: 10.1002/widm.1249.
T. Huang, Q. She, and J. Zhang, “BoostingBERT:Integrating Multi-Class Boosting into BERT for NLP Tasks,” 2020, arXiv. doi: 10.48550/ARXIV.2009.05959.
M. A. Ganaie, M. Hu, A. K. Malik, M. Tanveer, and P. N. Suganthan, “Ensemble deep learning: A review,” Engineering Applications of Artificial Intelligence, vol. 115, p. 105151, Oct. 2022, doi: 10.1016/j.engappai.2022.105151.
A. Mohammed and R. Kora, “An effective ensemble deep learning framework for text classification,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 10, pp. 8825–8837, Nov. 2022, doi: 10.1016/j.jksuci.2021.11.001.
M. U. Salur and İ. Aydın, “A soft voting ensemble learning-based approach for multimodal sentiment analysis,” Neural Comput & Applic, vol. 34, no. 21, pp. 18391–18406, Nov. 2022, doi: 10.1007/s00521-022-07451-7.
P. Narwariya and S. K. Tiwari, “A robust multi-class text classification framework using TF-IDF features and soft voting ensembles,” in Recent Advances in Computational Methods in Science and Technology, CRC Press, 2026.
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
