Spell Correction for the Minangkabau Language  Using BERT-Based Embeddings

Dewi Soyusiawaty; Abdul Fadlil; Sunardi

doi:10.29207/resti.v10i3.7182

Authors

Dewi Soyusiawaty Universitas Ahmad Dahlan
Abdul Fadlil Universitas Ahmad Dahlan
Sunardi Universitas Ahmad Dahlan

DOI:

https://doi.org/10.29207/resti.v10i3.7182

Keywords:

cosine similarity, IndoBERT, low-resource language, mBERT, minangkabau language

Abstract

Spell checking is an essential component of natural language processing, as it directly influences applications such as sentiment analysis, text classification, and machine translation. Developing a reliable system for low-resource languages like Minangkabau is challenging due to frequent spelling variations and limited annotated data. This study proposes a contextual spell correction model using pre-trained IndoBERT and multilingual BERT (mBERT) embeddings applied without additional training. The method masks misspelled words, extracts the contextual embedding of the [MASK] token, and compares it with candidate embeddings generated through dictionary filtering and Levenshtein Distance. Evaluation was conducted on the Spell Error Corpus for Minangkabau Language (SPEML), which includes insertion errors, deletion errors, substitution errors, transposition errors, punctuation errors, real-word errors, and loanword errors. Results show that mBERT consistently outperformed IndoBERT, achieving an average F1-score of 0.83 compared to 0.75. Statistical validation using paired t-test and Wilcoxon signed-rank test further confirmed that the performance difference between the two models was significant. Both models reached perfect scores (1.0) in real-word and loanword categories, and strong results in insertion_medium (0.97 for mBERT and 0.95 for IndoBERT). The lowest performance occurred in deletion_short (0.52 for IndoBERT) and long words cases (0.57 for mBERT). In addition, a small-scale external validation using 100 Twitter/X sentences was conducted to assess the applicability of the proposed method to real-world social media text. Overall, the findings confirm the effectiveness of contextual embeddings for Minangkabau spelling correction while highlighting challenges in long misspelled words, deletion errors, and informal real-world text.

Downloads

Download data is not yet available.

References

W. B. Yanfi Yanfi, Haryono Soeparno, Reina Setiawan, “Multi-Head_Attention_Based_Bidirectional_LSTM_for_Spelling_Error_Detection_in_the_Indonesian_Language.pdf,” 2024, Ieee Access. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10580948

F. Koto and I. Koto, “Towards Computational Linguistics in Minangkabau Language: Studies on Sentiment Analysis and Machine Translation.” [Online]. Available: https://id.wikimedia.org/wiki/

P. Etoori, M. Chinnakotla, and R. Mamidi, “Automatic spelling correction for resource-scarce languages using deep learning,” ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Student Research Workshop, pp. 146–152, 2018. [Online]. Available: https://doi.org/10.18653/v1/P18-3021

D. Hládek, J. Staš, and M. Pleva, “Survey of automatic spelling correction,” Electronics (Switzerland), vol. 9, no. 10, pp. 1–29, 2020. [Online]. Available: https://doi.org/10.3390/electronics9101670

Y. Hu, X. Jing, Y. Ko, and J. T. Rayz, “Misspelling correction with pre-trained contextual language model,” Proceedings of 2020 IEEE 19th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2020, pp. 144–149, 2020. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICCICC50026.2020.9450253

Ç. Çöltekin, A. S. Doğruöz, and Ö. Çetinoğlu, Resources for Turkish natural language processing: A critical survey, vol. 57, no. 1. Springer Netherlands, 2023. [Online]. Available: https://doi.org/10.1007/s10579-022-09605-4

P. A. Selvaraj, M. Jagadeesan, M. Harikrishnan, R. Vijayapriya, and K. Jayasudha, “Survey on Spell Checker for Tamil Language Using Natural Language Processing,” Journal of Pharmaceutical Negative Results ¦, vol. 13, no. 7, p. 2022. [Online]. Available: https://www.pnrjournal.com/index.php/home/article/view/1697

U. Liyanapathirana, K. Gunasinghe, and G. Dias, “SinSpell: A Comprehensive Spelling Checker for Sinhala,” Accelerating Higher Education Expansionand Development (AHEAD), 2021. Doi: 10.48550/arXiv.2107.02983

Y. Chaabi and F. Ataa Allah, “Amazigh spell checker using Damerau-Levenshtein algorithm and N-gram,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 8, pp. 6116–6124, 2022. doi: https://doi.org/10.1016/j.jksuci.2021.07.015

R. Aziz et al., “Real Word Spelling Error Detection and Correction for Urdu Language,” IEEE Access, vol. 11, no. September, pp. 100948–100962, 2023. doi: 10.1109/ACCESS.2023.3312730

S. Laaroussi, A. Yousfi, S. L. Aouragh, and S. O. El Alaoui, “Global spelling correction in context using language models: Application to the Arabic Language,” International Journal of Computing and Digital Systems, vol. 13, no. 1, pp. 361–370, 2023. doi: 10.12785/ijcds/130129

N. Mukazhanov, Z. Alibiyeva, A. Yerimbetova, A. Kassymova, and N. Alibiyeva, “Development of an augmented Damerau-Levenshtein method for correcting spelling errors in Kazakh texts,” Eastern-European Journal of Enterprise Technologies, vol. 5, no. 2(125), pp. 23–33, 2023. doi: 10.15587/1729-4061.2023.289187

T. Zesch, D. Gardner, and M. Bexte, “Transformer-Based Real-Word Spelling Error Feedback with Configurable Confusion Sets,” Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025). pp. 375–383, 2025. doi: 10.18653/v1/2025.bea-1.29

S. Kasmaiee, S. Kasmaiee, and M. Homayounpour, “Correcting spelling mistakes in Persian texts with rules and deep learning methods,” Sci Rep, vol. 13, no. 1, pp. 1–21, 2023. doi: https://doi.org/10.1038/s41598-023-47295-2

N. Madi and H. Al-Khalifa, “Error detection for Arabic text using neural sequence labeling,” Applied Sciences (Switzerland), vol. 10, no. 15, pp. 1–14, 2020. doi: https://doi.org/10.3390/app10155279

R. Phukan, M. Neog, P. J. Goutom, and N. Baruah, “Automated Spelling Error Detection in Assamese Texts using Deep Learning Approaches,” Procedia Comput Sci, vol. 235, pp. 1684–1694, 2024. doi: https://doi.org/10.1016/j.procs.2024.04.159

M. Arslan, H. Ghanem, S. Munawar, and C. Cruz, “A Survey on RAG with LLMs,” Procedia Comput Sci, vol. 246, no. C, pp. 3781–3790, 2024. doi: https://doi.org/10.1016/j.procs.2024.09.178

H. Zhang and M. O. Shafiq, “Survey of transformers and towards ensemble learning using transformers for natural language processing,” J Big Data, vol. 11, no. 1, 2024. doi: https://doi.org/10.1186/s40537-023-00842-0

C. Bryant, Z. Yuan, M. R. Qorib, H. Cao, H. T. Ng, and T. Briscoe, “Grammatical Error Correction: A Survey of the State of the Art,” Computational Linguistics, vol. 49, no. 3, pp. 643–701, 2023. doi: 10.48550/arXiv.2211.05166

S. Singh and S. Singh, "Systematic review of spell-checkers for highly inflectional languages," Artificial Intelligence Review, vol. 53, p. 4051–4092, 2020. doi: 10.1007/s10462-019-09787-4

S. Zhang, H. Huang, J. Liu, and H. Li, “Spelling error correction with soft-masked BERT,” Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 882–890, 2020. doi: 10.18653/v1/2020.acl-main.82

C. R. Rahman, M. D. H. Rahman, S. Zakir, M. Rafsan, and M. E. Ali, “BSpell: A CNN-Blended BERT Based Bangla Spell Checker,” BLP 2023 - 1st Workshop on Bangla Language Processing, Proceedings of the Workshop, pp. 85–93, 2023. doi: 10.18653/v1/2023.banglalp-1.2

T. Ji, H. Yan, and X. Qiu, “SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check,” EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pp. 3544–3551, 2021. doi: 10.18653/v1/2021.emnlp-main.287

A. Khattak, M. Z. Asghar, A. Saeed, I. A. Hameed, S. Asif Hassan, and S. Ahmad, “A survey on sentiment analysis in Urdu: A resource-poor language,” Egyptian Informatics Journal, vol. 22, no. 1, pp. 53–74, 2021. doi: https://doi.org/10.1016/j.eij.2020.04.003

Y. Yanfi, R. Setiawan, H. Soeparno, and W. Budiharto, “SPECIL: Spell Error Corpus for the Indonesian Language,” IEEE Access, vol. 11, pp. 93227–93237, 2023. doi: 10.1109/ACCESS.2023.3307712

H. L. H. S. W. Yanfi, Ford Lumban Gaol, Benfano Soewito, “Spell Checker for the Indonesian Language Extensive Review.pdf,” 2022, International Journal of Emerging Technology and Advanced Engineering. doi: 10.46338/ijetae0522_01

J. Bakagianni, K. Pouli, M. Gavriilidou, and J. Pavlopoulos, “A systematic survey of natural language processing for the Greek language,” Patterns, p. 101313, 2025. doi: https://doi.org/10.1016/j.patter.2025.101313

P. Kittiworapanya, N. Saelek, A. Lertpiya, and T. Chalothorn, “Survey of Query correction for Thai business-oriented information retrieval,” Proceedings - 2020 15th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2020, pp. 3–8, 2020. doi: 10.1109/iSAI-NLP51646.2020.9376809

M. Goonawardena, A. Kulatunga, R. Wickramasinghe, T. Weerasekara, H. De Silva, and S. Thelijjagoda, “Automated Spelling Checker And Grammatical Error Detection And Correction Model for Sinhala Language,” Proceedings - International Research Conference on Smart Computing and Systems Engineering, SCSE 2022, vol. 5, pp. 184–189, 2022, doi: 10.1109/SCSE56529.2022.9905126.

S. Abdulmalek, “Levenstein ’ s Algorithm On English and Arabic :,”2021 International Conference of Technology, Science and Administration (ICTSA). 2021. doi: 10.1109/ICTSA52017.2021.9406547

F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference, pp. 757–770, 2020. doi: 10.48550/arXiv.2011.00677

F. Koto, J. H. Lau, T. Baldwin. “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 10660–10668, 2021. doi: 10.18653/v1/2021.emnlp-main.833

S. S. Batais, “Consonantal Debuccalization and Deletion in Minangkabau,” Sage Open, vol. 14, no. 2, pp. 1–13, 2024. doi: https://doi.org/10.1177/21582440241240576

Z. Guo, Y. Ni, K. Wang, W. Zhu, and G. Xie, “Global Attention Decoder for Chinese Spelling Error Correction,” Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1419–1428, 2021. doi: 10.18653/v1/2021.findings-acl.122

and N. K. Vasyl Lytvyn, Petro Pukach , Victoria Vysotska, Myroslava Vovk, “Identification and Correction of Grammatical Errors in Ukrainian Texts Based on Machine Learning Technology,” Mdpi, vol. 11, no. 904, 2023. doi: https://doi.org/10.3390/math11040904

M. Dony and S. Yasin, “Koreksi Teks Untuk Kesalahan Penulisan Pada Artikel Berita Menggunakan Model Bahasa,” JITLIT, vol. 1, no. 2, pp. 33–38, 2024.

H. N. Trung, D. T. Ham, T. Huynh, and K. Hoang, “A Combination of BERT and Transformer for Vietnamese Spelling Correction,” Lecture Notes in Computer Science. Intelligent Information and Database Systems. pp.545-558. May 2024. doi: 10.1007/978-3-031-21743-2_43

C. G. Turhan, “Leveraging large language models for spelling correction in Turkish,” PeerJ Comput Sci, vol. 11, pp. 1–26, 2025. doi: 10.7717/peerj-cs.2889

D. Soyusiawaty, A. Fadlil, and S. Sunardi “Spell Error Corpus for Minang Language.” [Online]. Available: https://www.kaggle.com/datasets/dewisoyus/spell-error-corpus-for-minang-language

D. Soyusiawaty, A. Fadlil, and S. Sunardi, “Pengembangan Korpus Bahasa Minang pada Spell Error Corpus for Minang Language (SPEML),” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 11, no. 1, pp. 17–26, 2025. doi: https://doi.org/10.25077/TEKNOSI.v11i01.2025.17-26

C. Zhou et al., “Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors,” 2022, [Online]. Available: http://arxiv.org/abs/2204.12052