An Applied Evaluation of Multi-Label Hate Speech Detection for Indonesian Digital Platforms
DOI:
https://doi.org/10.29207/resti.v10i3.7591Keywords:
hate speech, indonesian text, multi-label, text filtering, transformersAbstract
The rapid growth of user-generated content on digital platforms has increased the difficulty of moderating hate-related expressions, particularly in linguistically diverse environments such as Indonesian social media. Automated hate speech detection systems are therefore expected to operate not only with reliable predictive behavior but also with practical efficiency under large-scale deployment conditions. This study reports an applied evaluation of Transformer-based models for multi-label hate speech detection on Indonesian digital platforms. Rather than introducing a new classification architecture, the work focuses on assessing multiple pretrained language models within a unified and reproducible evaluation framework. The analysis examines overall model behavior, per-label performance tendencies, inference efficiency, and common error patterns under realistic multi-label settings. The results indicate that IndoBERT-based models (indobenchmark/indobert-base-p1 and cahya/bert-base-indonesian) achieved the strongest predictive performance for multi-label hate speech detection, although performance differences across the evaluated Transformer models remained relatively incremental. Experimental results show that the best-performing model achieved a macro-F1 score of 0.9742 and a micro-F1 score of 0.9749, while other Transformer models demonstrated competitive performance with macro-F1 values ranging from 0.955 to 0.964. In terms of efficiency, distilled models provided faster inference (approximately 5–7 ms per sample) compared to full-size models (8–11 ms per sample), highlighting a practical trade-off between predictive performance and computational cost. These findings emphasize the importance of practical evaluation strategies and suggest that flexible model configurations are more suitable than reliance on a single high-capacity model in real-world moderation systems.
Downloads
References
J. Marolla-Gajardo and I. Riquelme Plaza, “Teacher Education, Diversity, and the Prevention of Hate Speech: Ethical and Political Foundations for Inclusive Citizenship,” Societies, vol. 15, no. 5. p. 139, 2025. doi: 10.3390/soc15050139.
B. Vidgen and L. Derczynski, “Directions in abusive language training data, a systematic review: Garbage in, garbage out.,” PLoS One, vol. 15, no. 12, p. e0243300, 2020, doi: 10.1371/journal.pone.0243300.
P. Fortuna and S. Nunes, “A Survey on Automatic Detection of Hate Speech in Text,” ACM Comput. Surv., vol. 51, pp. 1–30, Jul. 2018, doi: 10.1145/3232676.
L. Floridi et al., “AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations,” Minds Mach., vol. 28, no. 4, pp. 689–707, 2018, doi: 10.1007/s11023-018-9482-5..
B. Mathew, P. Saha, C. Biemann, P. Goyal, and A. Mukherjee, “HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection,” Proc. AAAI Conf. Artif. Intell., vol. 35, pp. 14867–14875, May 2021, doi: 10.1609/aaai.v35i17.17745.
Ziqi Zhang and Lei Luo, “Hate speech detection: A solved problem? The challenging case of long tail on Twitter,” Semant. Web, vol. 10, no. 5, pp. 925–945, Sep. 2019, doi: 10.3233/SW-18033.
M. Mozafari, R. Farahbakhsh, and N. Crespi, “A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media BT - Complex Networks and Their Applications VIII,” H. Cherifi, S. Gaito, J. F. Mendes, E. Moro, and L. M. Rocha, Eds., Cham: Springer International Publishing, 2020, pp. 928–940.10.1007/978-3-030-36687-2_77
B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Suzhou, China: Association for Computational Linguistics, 2020, pp. 843–857. doi: 10.18653/v1/2020.aacl-main.85.
Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient Transformers: A Survey,” ACM Comput. Surv., vol. 55, no. 109, pp. 1–28, Sep. 2020, doi: doi: 10.1145/3530811..
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” CoRR, vol. abs/1910.01108, 2019, [Online]. Available: http://arxiv.org/abs/1910.01108
M.-L. Zhang and Z.-H. Zhou, “A Review on Multi-Label Learning Algorithms,” IEEE Trans. Knowl. Data Eng., vol. 26, no. 8, pp. 1819–1837, 2014, doi: 10.1109/TKDE.2013.39..
F. M. Plaza-Del-Arco, M. D. González, L. Ureña-López, and M. Martín-Valdivia, “Integrating implicit and explicit linguistic phenomena via multi-task learning for offensive language detection,” Knowledge-Based Syst., vol. 258, p. 109965, Oct. 2022, doi: 10.1016/j.knosys.2022.109965.
M. O. Ibrohim and I. Budi, “Multi-label Hate Speech and Abusive Language Detection in Indonesian Twitter,” in Proceedings of the Third Workshop on Abusive Language Online, Florence, Italy: Association for Computational Linguistics, 2019, pp. 46–57. doi: 10.18653/v1/W19-3506.
T. Wolf et al., “Transformers: State-of-the-Art Natural Language Processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Q. Liu and D. Schlangen, Eds., Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45. doi: 10.18653/v1/2020.emnlp-demos.6..
S. Shaik and C. S P, “NLP and Machine Learning for Sentiment Analysis in COVID-19 Tweets: A Comparative Study,” EAI Endorsed Trans. Pervasive Heal. Technol., vol. 10 SE-Research article, doi: 10.4108/eetpht.10.7051.
D. W. Brata, A. Djunaidy, and D. O. Siahaan, “Comparison of Deep Learning Methods in Detecting Hate Speech in Indonesian Tweets,” in Proceedings of the 8th International Conference on Sustainable Information Engineering and Technology, in SIET ’23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 58–63. doi: 10.1145/3626641.3626925.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proceedings of the 2019 Conference of the North, Association for Computational Linguistics, Jun 2019, pp. 4171–4186. 2018. doi: 10.18653/v1/N19-1423.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, D. Scott, N. Bel, and C. Zong, Eds., Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 757–770. doi: 10.48550/arXiv.2011.00677.
F. Koto, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W. Yih, Eds., Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 10660–10668. doi: 10.18653/v1/2021.emnlp-main.833.
A. Paszke et al., “PyTorch: an imperative style, high-performance deep learning library,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA: Curran Associates Inc., 2019. doi: 10.48550/arXiv.1912.01703.
T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated Hate Speech Detection and the Problem of Offensive Language,” Proc. Int. AAAI Conf. Web Soc. Media, vol. 11, pp. 512–515, May 2017, doi: 10.1609/icwsm.v11i1.14955.
Z. Waseem and D. Hovy, “Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter,” in Proceedings of the {NAACL} Student Research Workshop, J. Andreas, E. Choi, and A. Lazaridou, Eds., San Diego, California: Association for Computational Linguistics, Jun. 2016, pp. 88–93. doi: 10.18653/v1/N16-2013.
A. Schmidt and M. Wiegand, “A Survey on Hate Speech Detection using Natural Language Processing,” in Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, L.-W. Ku and C.-T. Li, Eds., Valencia, Spain: Association for Computational Linguistics, Apr. 2017, pp. 1–10. doi: 10.18653/v1/w17-1101.
P. Badjatiya, S. Gupta, M. Gupta, and V. Varma, “Deep Learning for Hate Speech Detection in Tweets,” in Proceedings of the 26th International Conference on World Wide Web Companion, in WWW ’17 Companion. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2017, pp. 759–760. doi: 10.1145/3041021.3054223.
M. Weber, C. Viehmann, M. Ziegele, and C. Schemer, “Online Hate Does Not Stay Online – How Implicit and Explicit Attitudes Mediate the Effect of Civil Negativity and Hate in User Comments on Prosocial Behavior,” Comput. Human Behav., vol. 104, p. 106192, 2020, doi: https://doi.org/10.1016/j.chb.2019.106192.
C. Nobata, J. Tetreault, A. Thomas, Y. Mehdad, and Y. Chang, “Abusive Language Detection in Online User Content,” in Proceedings of the 25th International Conference on World Wide Web, in WWW ’16. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2016, pp. 145–153. doi: 10.1145/2872427.2883062.
V. Basile et al., “SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter,” in Proceedings of the 13th International Workshop on Semantic Evaluation, J. May, E. Shutova, A. Herbelot, X. Zhu, M. Apidianaki, and S. M. Mohammad, Eds., Minneapolis, Minnesota, USA: Association for Computational Linguistics, Jun. 2019, pp. 54–63. doi: 10.18653/v1/S19-2007.
A.-M. Founta et al., “Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior,” Proc. Int. AAAI Conf. Web Soc. Media, vol. 12, no. 1, Feb. 2018, doi: 10.48550/arXiv.1802.00393.
M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar, “Predicting the Type and Target of Offensive Posts in Social Media,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds., Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 1415–1420. doi: 10.18653/v1/N19-1144.
S. Rosenthal, N. Farra, and P. Nakov, “SemEval-2017 Task 4: Sentiment Analysis in Twitter,” in Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), S. Bethard, M. Carpuat, M. Apidianaki, S. M. Mohammad, D. Cer, and D. Jurgens, Eds., Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 502–518. doi: 10.18653/v1/S17-2088.
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
