Social Feature Integration for Entertainment Hoax Detection: Machine Learning and DistilBERT-Fusion
DOI:
https://doi.org/10.29207/resti.v10i4.7208Keywords:
DistilBERT, hoax detection, GossipCop, machine learning, social featuresAbstract
Detecting entertainment hoaxes remains a pressing challenge in the social media era, where short texts, provocative headlines, and rapid dissemination complicate verification. This study compares the effectiveness of classical machine learning models and the transformer DistilBERT, both with and without the integration of social features. The modified GossipCop dataset, consisting of 22,140 entries, includes news titles as the primary text representation and social features such as tweet count, tweet density, and viral indicators. Text was represented using TF-IDF for classical models and DistilBERT tokenization for the transformer, with performance evaluated under stratified 10-fold cross-validation. Results show that incorporating social features consistently improves classical models, with XGBoost + Social achieving the best performance (PR-AUC 0.84; F1-score 0.76), surpassing DistilBERT-Fusion (PR-AUC 0.79). McNemar’s test confirmed significant differences in error distributions, strengthening the reliability of these findings. To reduce reliance on a single empirical dataset, the social feature integration pipeline was additionally validated on the PolitiFact benchmark from a different domain, where incorporating minimal social signals yielded measurable and statistically significant gains over text-only models (Wilcoxon p = 0.0137), confirming the robustness of the social feature effect beyond the entertainment domain. Overall, the results highlight that for short-text entertainment news under the conditions examined in this study, boosting models enriched with social signals can outperform transformer-based approaches. While DistilBERT-Fusion provided competitive results, its improvement over text-only DistilBERT remained limited due to the short-text nature of the dataset, indicating opportunities for richer fusion strategies in future research.
Downloads
References
M. Rodríguez-Ibánez, A. Casánez-Ventura, F. Castejón-Mateos, and P.-M. Cuenca-Jiménez, “A review on sentiment analysis from social media platforms,” Expert Syst. Appl., vol. 223, p. 119862, Aug. 2023, doi: 10.1016/j.eswa.2023.119862.
B. T.K., C. S. R. Annavarapu, and A. Bablani, “Machine learning algorithms for social media analysis: A survey,” Comput. Sci. Rev., vol. 40, p. 100395, May 2021, https://doi.org/10.1016/j.cosrev.2021.100395.
M. Shahbazi and D. Bunker, “Social media trust: Fighting misinformation in the time of crisis,” Int. J. Inf. Manag., vol. 77, p. 102780, Aug. 2024, doi: 10.1016/j.ijinfomgt.2024.102780.
S. Muhammed T and S. K. Mathew, “The disaster of misinformation: a review of research in social media,” Int. J. Data Sci. Anal., vol. 13, no. 4, pp. 271–285, May 2022, doi: 10.1007/s41060-022-00311-6.
N. Cavus, M. Goksu, and B. Oktekin, “Real-time fake news detection in online social networks: FANDC Cloud-based system,” Sci. Rep., vol. 14, no. 1, p. 25954, Oct. 2024, doi: 10.1038/s41598-024-76102-9.
I. Ali, M. N. B. Ayub, P. Shivakumara, and N. F. B. M. Noor, “Fake News Detection Techniques on Social Media: A Survey,” Wirel. Commun. Mob. Comput., vol. 2022, no. 1, p. 6072084, Jan. 2022, doi: 10.1155/2022/6072084.
K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu, “FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media,” Mar. 27, 2019, arXiv: arXiv:1809.01286. doi: 10.48550/arXiv.1809.01286.
J. Y. Khan, Md. T. I. Khondaker, S. Afroz, G. Uddin, and A. Iqbal, “A benchmark study of machine learning models for online fake news detection,” Mach. Learn. Appl., vol. 4, p. 100032, Jun. 2021, doi: 10.1016/j.mlwa.2021.100032.
S. V. Balshetwar, A. Rs, and D. J. R, “Fake news detection in social media based on sentiment analysis using classifier techniques,” Multimed. Tools Appl., vol. 82, no. 23, pp. 35781–35811, Sep. 2023, doi: 10.1007/s11042-023-14883-3.
M. Nasser et al., “A systematic review of multimodal fake news detection on social media using deep learning models,” Results Eng., vol. 26, p. 104752, Jun. 2025, doi: 10.1016/j.rineng.2025.104752.
J. Alghamdi, Y. Lin, and S. Luo, “Towards COVID-19 fake news detection using transformer-based models,” Knowl.-Based Syst., vol. 274, p. 110642, Aug. 2023, doi: 10.1016/j.knosys.2023.110642.
M. S. Farooq, A. Naseem, F. Rustam, and I. Ashraf, “Fake news detection in Urdu language using machine learning,” PeerJ Comput. Sci., vol. 9, p. e1353, May 2023, doi: 10.7717/peerj-cs.1353.
S. Yakkundi, R. Patil, S. Sangani, R. H. Goudar, S. I. Goudar, and A. Qazi, “Exploring machine learning for fake news detection: techniques, tools, challenges, and future research directions,” Discov. Appl. Sci., vol. 7, no. 8, p. 873, Aug. 2025, doi: 10.1007/s42452-025-07548-3.
K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, “Fake News Detection on Social Media: A Data Mining Perspective,” Sep. 03, 2017, arXiv: arXiv:1708.01967. doi: 10.48550/arXiv.1708.01967.
D. Mouratidis, A. Kanavos, and K. Kermanidis, “From Misinformation to Insight: Machine Learning Strategies for Fake News Detection,” Information, vol. 16, no. 3, p. 189, Feb. 2025, doi: 10.3390/info16030189.
B. Şener, K. Acici, and E. Sümer, “Categorization of Alzheimer’s disease stages using deep learning approaches with McNemar’s test,” PeerJ Comput. Sci., vol. 10, p. e1877, Feb. 2024, doi: 10.7717/peerj-cs.1877.
S. Uddin and H. Lu, “Confirming the statistically significant superiority of tree-based machine learning algorithms over their counterparts for tabular data,” PLOS ONE, vol. 19, no. 4, p. e0301541, Apr. 2024, doi: 10.1371/journal.pone.0301541.
M. Adnan, A. A. S. Alarood, M. I. Uddin, and I. Ur Rehman, “Utilizing grid search cross-validation with adaptive boosting for augmenting performance of machine learning models,” PeerJ Comput. Sci., vol. 8, p. e803, Feb. 2022, doi: 10.7717/peerj-cs.803.
K. Ghosh, C. Bellinger, R. Corizzo, P. Branco, B. Krawczyk, and N. Japkowicz, “The class imbalance problem in deep learning,” Mach. Learn., vol. 113, no. 7, pp. 4845–4901, Jul. 2024, doi: 10.1007/s10994-022-06268-8.
S. K. Hamed, M. J. Ab Aziz, and M. R. Yaakub, “A review of fake news detection approaches: A critical analysis of relevant studies and highlighting key challenges associated with the dataset, feature representation, and data fusion,” Heliyon, vol. 9, no. 10, p. e20382, Oct. 2023, doi: 10.1016/j.heliyon.2023.e20382.
D. Z. Abidin, L. Afuan, A. N. Toscany, and Nurhadi, “A Comprehensive Benchmarking Pipeline for Transformer-Based Sentiment Analysis using Cross-Validated Metrics,” J. Tek. Inform. Jutif, vol. 6, no. 4, pp. 1797–1810, 2025, doi: https://doi.org/10.52436/1.jutif.2025.6.4.4894.
P. Guleria, J. Frnda, and P. N. Srinivasu, “NLP based text classification using TF-IDF enabled fine-tuned long short-term memory: An empirical analysis,” Array, vol. 27, p. 100467, Sep. 2025, doi: 10.1016/j.array.2025.100467.
A. Tariq, A. Mehmood, M. Elhadef, and M. U. G. Khan, “Adversarial Training for Fake News Classification,” IEEE Access, vol. 10, pp. 82706–82715, 2022, doi: 10.1109/ACCESS.2022.3195030.
A. Chadha and B. Kaushik, “A Hybrid Deep Learning Model Using Grid Search and Cross-Validation for Effective Classification and Prediction of Suicidal Ideation from Social Network Data,” New Gener. Comput., vol. 40, no. 4, pp. 889–914, Dec. 2022, doi: 10.1007/s00354-022-00191-1.
I. Mohiuddin and A. Almogren, “Ensemble techniques for detecting profile cloning attacks in online social networks,” PeerJ Comput. Sci., vol. 11, p. e3182, Sep. 2025, doi: 10.7717/peerj-cs.3182.
R. Scharlach and B. Hallinan, “The value affordances of social media engagement features,” J. Comput.-Mediat. Commun., vol. 28, no. 6, p. zmad040, Sep. 2023, doi: 10.1093/jcmc/zmad040.
S. K. Uppada, K. Manasa, B. Vidhathri, R. Harini, and B. Sivaselvan, “Novel approaches to fake news and fake account detection in OSNs: user social engagement and visual content centric model,” Soc. Netw. Anal. Min., vol. 12, no. 1, p. 52, Dec. 2022, doi: 10.1007/s13278-022-00878-9.
R. Patil, S. Boit, V. Gudivada, and J. Nandigam, “A Survey of Text Representation and Embedding Techniques in NLP,” IEEE Access, vol. 11, pp. 36120–36146, 2023, doi: 10.1109/ACCESS.2023.3266377.
L. R. Aini and Evi Yulianti, “Expertise Retrieval Using Adjusted TF-IDF and Keyword Mapping to ACM Classification Terms,” J. RESTI Rekayasa Sist. Dan Teknol. Inf., vol. 9, no. 3, pp. 497–505, May 2025, doi: 10.29207/resti.v9i3.6397.
A. Hartanto, Ema Utami, Arief Setyanto, and Kusrini, “Evaluating Transformer Models for Social Media Text-Based Personality Profiling,” J. RESTI Rekayasa Sist. Dan Teknol. Inf., vol. 9, no. 1, pp. 20–30, Jan. 2025, doi: 10.29207/resti.v9i1.6157.
L. Zhang, “Features extraction based on Naive Bayes algorithm and TF-IDF for news classification,” PLOS One, vol. 20, no. 7, p. e0327347, Jul. 2025, doi: 10.1371/journal.pone.0327347.
S. Kofi Akpatsa et al., “Online News Sentiment Classification Using DistilBERT,” J. Quantum Comput., vol. 4, no. 1, pp. 1–11, 2022, doi: 10.32604/jqc.2022.026658.
J. Alghamdi, S. Luo, and Y. Lin, “A comprehensive survey on machine learning approaches for fake news detection,” Multimed. Tools Appl., vol. 83, no. 17, pp. 51009–51067, Nov. 2023, doi: 10.1007/s11042-023-17470-8.
L. Jixian, A. Gang, S. Zhihao, and S. Xiaoqiang, “Social Media Multimodal Information Analysis based on the BiLSTM-Attention-CNN-XGBoost Ensemble Neural Network,” Int. J. Adv. Comput. Sci. Appl., vol. 13, no. 12, 2022, doi: 10.14569/IJACSA.2022.0131215.
M. Hussain et al., “Optimised knowledge distillation for efficient social media emotion recognition using DistilBERT and ALBERT,” Sci. Rep., vol. 15, no. 1, p. 30104, Aug. 2025, doi: 10.1038/s41598-025-16001-9.
F. E. Arévalo-Cordovilla and M. Peña, “Evaluating ensemble models for fair and interpretable prediction in higher education using multimodal data,” Sci. Rep., vol. 15, no. 1, p. 29420, Aug. 2025, doi: 10.1038/s41598-025-15388-9.
D. Chicco and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Min., vol. 16, no. 1, p. 4, Feb. 2023, doi: 10.1186/s13040-023-00322-4.
J. Miao and W. Zhu, “Precision–recall curve (PRC) classification trees,” Evol. Intell., vol. 15, no. 3, pp. 1545–1569, Sep. 2022, doi: 10.1007/s12065-021-00565-2.
R. Meléndez, R. Giraldo, and V. Leiva, “Sign, Wilcoxon and Mann-Whitney Tests for Functional Data: An Approach Based on Random Projections,” Mathematics, vol. 9, no. 1, p. 44, Dec. 2020, doi: 10.3390/math9010044.
M. Q. R. Pembury Smith and G. D. Ruxton, “Effective use of the McNemar test,” Behav. Ecol. Sociobiol., vol. 74, no. 11, p. 133, Nov. 2020, doi: 10.1007/s00265-020-02916-y.
Y. Atilkan et al., “Advancing Crayfish Disease Detection: A Comparative Study of Deep Learning and Canonical Machine Learning Techniques,” Appl. Sci., vol. 14, no. 14, p. 6211, Jul. 2024, doi: 10.3390/app14146211.
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
