An Optimized Hyperparameter Tuning for Improved Hate Speech Detection with Multilayer Perceptron
Abstract
Hate speech classification is a critical task in the domain of natural language processing, aiming to mitigate the negative impacts of harmful content on digital platforms. This study explores the application of a Multilayer Perceptron (MLP) model for hate speech classification, utilizing Bag of Words (BoW) for feature extraction. The hypothesis posits that hyperparameter tuning through sophisticated optimization techniques will significantly improve model performance. To validate this hypothesis, we employed two distinct hyperparameter tuning approaches: Random Search and Optuna. Random Search provides a straightforward yet effective means of exploring the hyperparameter space, while Optuna offers a more sophisticated, optimization-based approach to hyperparameter selection. The study involved training the MLP model on a labeled dataset is based on crawling results on the Twitter platform of hate speech and non-hate speech overall total dataset is 13.169, followed by evaluation using standard metrics. Our experimental results demonstrate the comparative effectiveness of these two hyperparameter tuning methods. Notably, the MLP model tuned with Optuna achieved a higher F1-score of 81.49%, compared to 79.70% achieved with Random Search, indicating the superior performance of Optuna in optimizing the hyperparameters. These results were obtained through extensive cross-validation to ensure robustness and generalizability. The findings underscore the importance of optimized hyperparameters in developing robust hate speech classification systems. The superior perform ance of Optuna highlights its potential for broader application in other machine learning tasks requiring hyperparameter optimization. This improvement enables more reliable and efficient automated moderation, which is crucial for the integrity and security of digital communication platforms such as Twitter.
Downloads
References
J. Kansok-Dusche et al., “A Systematic Review on Hate Speech among Children and Adolescents: Definitions, Prevalence, and Overlap with Related Phenomena,” Oct. 01, 2023, SAGE Publications Ltd. doi: 10.1177/15248380221108070.
M. Anand, K. B. Sahay, M. A. Ahmed, D. Sultan, R. R. Chandan, and B. Singh, “Deep learning and natural language processing in computation for offensive language detection in online social networks by feature selection and ensemble classification techniques,” Theor Comput Sci, vol. 943, pp. 203–218, Jan. 2023, doi: 10.1016/j.tcs.2022.06.020.
B. Elisa Shearer, A. Mitchell, J. Research Elisa Shearer, R. Associate Hannah Klein, and C. Manager, “FOR MEDIA OR OTHER INQUIRIES,” 2021. [Online]. Available: www.pewresearch.org
B. Mathew, P. Saha, S. M. Yimam, C. Biemann, P. Goyal, and A. Mukherjee, “HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection,” Dec. 2020, [Online]. Available: http://arxiv.org/abs/2012.10289
“Kemp, S. (2021). https://datareportal.com/reports/digital-2021-global-overviewreport.”
Y. Wang, J. Guo, C. Yuan, and B. Li, “Sentiment Analysis of Twitter Data,” Nov. 01, 2022, MDPI. doi: 10.3390/app122211775.
R. Kumar, A. K. Ojha, S. Malmasi, and M. Zampieri, “Benchmarking Aggression Identification in Social Media,” 2018. [Online]. Available: https://competitions.codalab.org/
M. Subramanian, V. Easwaramoorthy Sathiskumar, G. Deepalakshmi, J. Cho, and G. Manikandan, “A survey on hate speech detection and sentiment analysis using machine learning and deep learning models,” Oct. 01, 2023, Elsevier B.V. doi: 10.1016/j.aej.2023.08.038.
T. Elansari, H. Bourray, and M. Ouanan, “Modeling of Multilayer Perceptron Neural Network Hyperparameter Optimization and Training,” 2023, doi: 10.21203/rs.3.rs-2570112/v1.
M. O. Ibrohim and I. Budi, “Multi-label Hate Speech and Abusive Language Detection in Indonesian Twitter,” 2019. [Online]. Available: https://www.komnasham.go.id/index.php/
E. Utami, Rini, A. F. Iskandar, and S. Raharjo, “Multi-Label Classification of Indonesian Hate Speech Detection Using One-vs-All Method,” in Proceedings - 2021 IEEE 5th International Conference on Information Technology, Information Systems and Electrical Engineering: Applying Data Science and Artificial Intelligence Technologies for Global Challenges During Pandemic Era, ICITISEE 2021, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 78–82. doi: 10.1109/ICITISEE53823.2021.9655883.
N. Azmi Verdikha, R. Habid, and A. Johar Latipah, “Analisis DistilBERT dengan Support Vector Machine (SVM) untuk Klasifikasi Ujaran Kebencian pada Sosial Media Twitter,” METIK JURNAL, vol. 7, no. 2, pp. 101–110, Dec. 2023, doi: 10.47002/metik.v7i2.583.
A. M. U. D. Khanday, S. T. Rabani, Q. R. Khan, and S. H. Malik, “Detecting twitter hate speech in COVID-19 era using machine learning and ensemble learning techniques,” International Journal of Information Management Data Insights, vol. 2, no. 2, Nov. 2022, doi: 10.1016/j.jjimei.2022.100120.
R. W. Acuña Caicedo, J. M. Gómez Soriano, and H. A. Melgar Sasieta, “Bootstrapping semi-supervised annotation method for potential suicidal messages,” Apr. 01, 2022, Elsevier B.V. doi: 10.1016/j.invent.2022.100519.
“Retracted: Analysing Hate Speech against Migrants and Women through Tweets Using Ensembled Deep Learning Model,” Comput Intell Neurosci, vol. 2023, pp. 1–1, Oct. 2023, doi: 10.1155/2023/9781063.
K. Shaker, “Optimizing Sentiment Big Data Classification Using Multilayer Perceptron,” Anbar Journal of Engineering Sciences, vol. 13, no. 2, pp. 14–21, Nov. 2022, doi: 10.37649/aengs.2022.176353.
H. Mehta and K. Passi, “Social Media Hate Speech Detection Using Explainable Artificial Intelligence (XAI),” Algorithms, vol. 15, no. 8, Aug. 2022, doi: 10.3390/a15080291.
H. Cam, A. V. Cam, U. Demirel, and S. Ahmed, “Sentiment analysis of financial Twitter posts on Twitter with the machine learning classifiers,” Heliyon, vol. 10, no. 1, Jan. 2024, doi: 10.1016/j.heliyon.2023.e23784.
I. de Zarzà, J. de Curtò, and C. T. Calafate, “Optimizing Neural Networks for Imbalanced Data,” Electronics (Switzerland), vol. 12, no. 12, Jun. 2023, doi: 10.3390/electronics12122674.
W. F. Satrya, R. Aprilliyani, and E. H. Yossy, “Sentiment analysis of Indonesian police chief using multi-level ensemble model,” in Procedia Computer Science, Elsevier B.V., 2022, pp. 620–629. doi: 10.1016/j.procs.2022.12.177.
A. Toktarova et al., “Hate Speech Detection in Social Networks using Machine Learning and Deep Learning Methods.” [Online]. Available: www.ijacsa.thesai.org
B. Morris, “The components of the wired spanning forest are recurrent,” Probab Theory Relat Fields, vol. 125, no. 2, pp. 259–265, Feb. 2003, doi: 10.1007/s00440-002-0236-0.
S. Sadiq, A. Mehmood, S. Ullah, M. Ahmad, G. S. Choi, and B. W. On, “Aggression detection through deep neural model on Twitter,” Future Generation Computer Systems, vol. 114, pp. 120–129, Jan. 2021, doi: 10.1016/j.future.2020.07.050.
H. T. Vo, H. T. Ngoc, and L. Da Quach, “An Approach to Hyperparameter Tuning in Transfer Learning for Driver Drowsiness Detection Based on Bayesian Optimization and Random Search,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 4, pp. 828–837, 2023, doi: 10.14569/IJACSA.2023.0140492.
L. Yang and A. Shami, “On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice,” Jul. 2020, doi: 10.1016/j.neucom.2020.07.061.
Z. B. Zabinsky, “Random Search Algorithms,” 2009.
J. Bergstra, J. B. Ca, and Y. B. Ca, “Random Search for Hyper-Parameter Optimization Yoshua Bengio,” 2012. [Online]. Available: http://scikit-learn.sourceforge.net.
I. Jamaleddyn, R. El ayachi, and M. Biniz, “An improved approach to Arabic news classification based on hyperparameter tuning of machine learning algorithms,” Journal of Engineering Research (Kuwait), vol. 11, no. 2, Jun. 2023, doi: 10.1016/j.jer.2023.100061.
Y. Zhao, W. Zhang, and X. Liu, “Grid search with a weighted error function: Hyper-parameter optimization for financial time series forecasting,” Appl Soft Comput, vol. 154, Mar. 2024, doi: 10.1016/j.asoc.2024.111362.
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.10902
O. Dib, Z. Nan, and J. Liu, “Machine learning-based ransomware classification of Bitcoin transactions,” Journal of King Saud University - Computer and Information Sciences, vol. 36, no. 1, Jan. 2024, doi: 10.1016/j.jksuci.2024.101925.
Z. Car, S. Baressi Šegota, N. Anđelić, I. Lorencin, and V. Mrzljak, “Modeling the Spread of COVID-19 Infection Using a Multilayer Perceptron,” Comput Math Methods Med, vol. 2020, 2020, doi: 10.1155/2020/5714714.
R. Marco, S. S. S. Ahmad, and S. Ahmad, “An Improving Long Short Term Memory-Grid Search Based Deep Learning Neural Network for Software Effort Estimation,” International Journal of Intelligent Engineering and Systems, vol. 16, no. 4, pp. 164–180, 2023, doi: 10.22266/ijies2023.0831.14.
Copyright (c) 2024 Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
