Harnessing BERT for Semantic Understanding in Tourism Recommendation Engines
DOI:
https://doi.org/10.29207/resti.v9i4.6575Keywords:
BERT, classification, deep learning, fine-tuning, NLPAbstract
It will be necessary for attraction managers within hotels to track guests' lifestyles to keep the business running. Such an understanding may be achieved, for example by analyzing reviews on attractions to capture the attitudes of the visitors towards the services and business within the tourism industry. The approach utilizes web scraping to gather user-generated reviews, using text preprocessing, data pre-processing, and further improvement of the model using labelled sentiment data divided into three sentiment classes: positive, negative, or neutral. The dataset consisting of 908 reviews were divided in 70:15:15 ratio for training, validation and testing. Model performance was measured in terms of accuracy, precision, recall and F1-score. In this study, the BERT deep learning model is used to classify sentiments of Indonesian tourist. Using the SmallBERT variant fine-tuned on 515k reviews for 5 epochs, the model achieved 91.40% accuracy, 90.51% precision, recall, and F1 score. The results indicate a dominance of positive sentiments, visualized using tableau. This research provides a robust foundation for developing intelligent sentiment-based recommendation systems in the tourism sector and suggests future exploration using other transformer-based models such as GPT, T5, or BART for comparative analysis.
Downloads
References
K. Lei, C. Wen, and X. Wang, “RETRACTED: Research on the coordinated development of tourism economy based on embedded dynamic data,” Microprocess. Microsyst., vol. 82, p. 103933, 2021, doi: https://doi.org/10.1016/j.micpro.2021.103933.
G. E. Bayram, “Impact of Information Technology on Tourism,” in The Emerald Handbook of ICT in Tourism and Hospitality, A. Hassan and A. Sharma, Eds., Emerald Publishing Limited, 2020, pp. 243–257. doi: 10.1108/978-1-83982-688-720201015.
J. Ahn and D. Im, “Automatic Recommendation of Nearby Tourist Attractions related to Events,” vol. 21, no. 3, pp. 407–413, 2020, doi: https://doi.org/10.5762/KAIS.2020.21.3.407.
Y. Luo, X. Zhang, Y. Qin, Z. Yang, and Y. Liang, “Tourism Attraction Selection with Sentiment Analysis of Online Reviews Based on Probabilistic Linguistic Term Sets and the IDOCRIW-COCOSO Model,” Int. J. Fuzzy Syst., vol. 23, no. 1, pp. 295–308, 2021, doi: 10.1007/s40815-020-00969-9.
D. G. Sainger, “Sentiment Analysis - An Assessment of Online Public Opinion: A Conceptual Review,” Turkish J. Comput. Math. Educ., vol. 12, no. 5, pp. 1881–1887, 2021, doi: 10.17762/turcomat.v12i5.2266.
M. Qian, C. Newton, and D. Qian, “Cultural Understanding Using In-context Learning and Masked Language Modeling BT - HCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence,” C. Stephanidis, M. Kurosu, J. Y. C. Chen, G. Fragomeni, N. Streitz, S. Konomi, H. Degen, and S. Ntoa, Eds., Cham: Springer International Publishing, 2021, pp. 500–508.
L. Huang, W. Chen, and H. Qu, “Accelerating Transformer for Neural Machine Translation,” ACM Int. Conf. Proceeding Ser., pp. 191–197, 2021, doi: 10.1145/3457682.3457711.
N. Q. K. Le, Q. T. Ho, T. T. D. Nguyen, and Y. Y. Ou, “A transformer architecture based on BERT and 2D convolutional neural network to identify DNA enhancers from sequence information,” Brief. Bioinform., vol. 22, no. 5, pp. 1–7, 2021, doi: 10.1093/bib/bbab005.
R. K. Kaliyar, “A Multi-layer Bidirectional Transformer Encoder for Pre-trained Word Embedding: A Survey of BERT,” in 2020 10th International Conference on Cloud Computing, Data Science & Engineering (Confluence), 2020, pp. 336–340. doi: 10.1109/Confluence47617.2020.9058044.
Y. Higuchi, B. Yan, S. Arora, T. Ogawa, T. Kobayashi, and S. Watanabe, “BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model,” Find. Assoc. Comput. Linguist. EMNLP 2022, pp. 5515–5532, 2022, doi: 10.18653/v1/2022.findings-emnlp.402.
B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and L. Li, “On the sentence embeddings from pre-trained language models,” EMNLP 2020 - 2020 Conf. Empir. Methods Nat. Lang. Process. Proc. Conf., pp. 9119–9130, 2020, doi: 10.18653/v1/2020.emnlp-main.733.
A. Chowanda and Y. Muliono, “Indonesian Sentiment Analysis Model from Social Media by Stacking BERT and BI-LSTM,” in 2022 3rd International Conference on Artificial Intelligence and Data Sciences (AiDAS), 2022, pp. 278–282. doi: 10.1109/AiDAS56890.2022.9918717.
F. Koto, J. H. Lau, and T. Baldwin, “INDOBERTWEET: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” EMNLP 2021 - 2021 Conf. Empir. Methods Nat. Lang. Process. Proc., pp. 10660–10668, 2021, doi: 10.18653/v1/2021.emnlp-main.833.
I. Ramli, A. A. Krisnadhi, and R. E. Prasojo, “IndoKEPLER, IndoWiki, and IndoLAMA: A Knowledge-enhanced Language Model, Dataset, and Benchmark for the Indonesian Language,” in 2022 7th International Workshop on Big Data and Information Security (IWBIS), 2022, pp. 19–26. doi: 10.1109/IWBIS56557.2022.9924844.
J. He, A. Wumaier, Z. Kadeer, W. Sun, X. Xin, and L. Zheng, “A Local and Global Context Focus Multilingual Learning Model for Aspect-Based Sentiment Analysis,” IEEE Access, vol. 10, pp. 84135–84146, 2022, doi: 10.1109/ACCESS.2022.3197218.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” COLING 2020 - 28th Int. Conf. Comput. Linguist. Proc. Conf., pp. 757–770, 2020, doi: 10.18653/v1/2020.coling-main.66.
G. Pérez-Tapia, F. Almeida-García, and P. Mercadé-Melé, “The ‘Four Core Elements’ as a Measuring Instrument: From Simplicity to Complexity in Tourist Destination,” Economies, vol. 9, no. 2. 2021. doi: 10.3390/economies9020053.
M. Dini, I. Curina, B. Francioni, S. Hegner, and M. Cioppi, “Tourists’ satisfaction and sense of belonging in adopting responsible behaviors: the role of on-site and social media involvement in cultural tourism,” TQM J., vol. 35, no. 9, pp. 388–410, 2023, doi: 10.1108/TQM-03-2023-0085.
Z.-A. Torabi, A. A. Shalbafian, Z. Allam, Z. Ghaderi, B. Murgante, and A. R. Khavarian-Garmsir, “Enhancing Memorable Experiences, Tourist Satisfaction, and Revisit Intention through Smart Tourism Technologies,” Sustainability, vol. 14, no. 5. 2022. doi: 10.3390/su14052721.
A. Kamal, S. B. Sura, L. P. Hung, R. Astri, and T. Kurniati, “Empowerment of MSME crafts recommendation system with content-based filtering method on social commerce after the pandemic,” E3S Web Conf., vol. 464, 2023, doi: 10.1051/e3sconf/202346412001.
R. Astri, L. P. Hung, S. B. Sura, A. Kamal, and R. Yuliet, “Sentiment analysis using naive bayes for reviews of visitors to Padang City beach tourism after the COVID-19 pandemic,” E3S Web Conf., vol. 464, 2023, doi: 10.1051/e3sconf/202346406002.
S. Cahyawijaya et al., “IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation,” EMNLP 2021 - 2021 Conf. Empir. Methods Nat. Lang. Process. Proc., pp. 8875–8898, 2021, doi: 10.18653/v1/2021.emnlp-main.699.
A. Kathuria, A. Gupta, and R. K. Singla, “A Review of Tools and Techniques for Preprocessing of Textual Data BT - Computational Methods and Data Engineering,” V. Singh, V. K. Asari, S. Kumar, and R. B. Patel, Eds., Singapore: Springer Singapore, 2021, pp. 407–422.
Rianto, A. B. Mutiara, E. P. Wibowo, and P. I. Santosa, “Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation,” J. Big Data, vol. 8, no. 1, p. 26, 2021, doi: 10.1186/s40537-021-00413-1.
L. A. Mullen, K. Benoit, O. Keyes, D. Selivanov, and J. Arnold, “Fast, Consistent Tokenization of Natural Language Text,” J. Open Source Softw., vol. 3, no. 23, p. 655, 2018, doi: 10.21105/joss.00655.
S. Sarica and J. Luo, “Stopwords in technical language processing,” PLoS One, vol. 16, no. 8 August, pp. 1–13, 2021, doi: 10.1371/journal.pone.0254937.
M. Siino, I. Tinnirello, and M. La Cascia, “Is text preprocessing still worth the time? A comparative survey on the influence of popular preprocessing methods on Transformers and traditional classifiers,” Inf. Syst., vol. 121, p. 102342, 2024, doi: https://doi.org/10.1016/j.is.2023.102342.
M. Nesca, A. Katz, C. K. Leung, and L. M. Lix, “A scoping review of preprocessing methods for unstructured text data to assess data quality,” Int. J. Popul. Data Sci., vol. 7, no. 1, pp. 1–15, 2022, doi: 10.23889/ijpds.v7i1.1757.
F. Alrasheedi, X. Zhong, and P. C. Huang, “Padding Module: Learning the Padding in Deep Neural Networks,” IEEE Access, vol. 11, pp. 7348–7357, 2023, doi: 10.1109/ACCESS.2023.3238315.
M. Xiao et al., “Addressing Overfitting Problem in Deep Learning-Based Solutions for Next Generation Data-Driven Networks,” Wirel. Commun. Mob. Comput., vol. 2021, 2021, doi: 10.1155/2021/8493795.
L. Sartran, S. Barrett, A. Kuncoro, M. Stanojević, P. Blunsom, and C. Dyer, “Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale,” Trans. Assoc. Comput. Linguist., vol. 10, pp. 1423–1439, Dec. 2022, doi: 10.1162/tacl_a_00526.
E. Radiya-Dixit, “How fine can fine-tuning be? Learning efficient language models,” Proc. Mach. Learn. Res., vol. 108, pp. 2435–2443, 2020.
Z. Fu, H. Yang, A. M. C. So, W. Lam, L. Bing, and N. Collier, “On the Effectiveness of Parameter-Efficient Fine-Tuning,” Proc. 37th AAAI Conf. Artif. Intell. AAAI 2023, vol. 37, pp. 12799–12807, 2023, doi: 10.1609/aaai.v37i11.26505.
Q. Wan, C. Wan, K. Xiao, H. Xiong, D. Liu, and X. Liu, “Token-Event-Role Structure-based Multi-Channel Document-Level Event Extraction,” vol. 1, no. 1, 2023, [Online]. Available: http://arxiv.org/abs/2306.17733
A. S. Talaat, “Sentiment analysis classification system using hybrid BERT models,” J. Big Data, vol. 10, no. 1, 2023, doi: 10.1186/s40537-023-00781-w.
H. Zhong et al., “Adam revisited: a weighted past gradients perspective,” Front. Comput. Sci., vol. 14, Oct. 2020, doi: 10.1007/s11704-019-8457-x.
P. Bianchi, “Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization,” SIAM J. Optim., vol. 31, pp. 244–274, Jan. 2021, doi: 10.1137/19M1263443.
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;





