Hand Sign Recognition of Indonesian Sign Language System SIBI Using Inception V3 Image Embedding and Random Forest

Mayang Sari; Eko Rudiawan Jamzuri

doi:10.29207/resti.v9i2.6156

Mayang Sari Politeknik Negeri Batam
Eko Rudiawan Jamzuri Politeknik Negeri Batam

DOI: https://doi.org/10.29207/resti.v9i2.6156

Keywords: hand sign recognition, SIBI, Inception V3, image embedding, random forest

Abstract

This paper presents a sign language recognition system for the Indonesian Sign Language System SIBI using image embeddings combined with a Random Forest classifier. A dataset comprising 5280 images across 24 classes of SIBI alphabet symbols was utilized. Image features were extracted using the Inception V3 image embedding, and classification was performed using Random Forest algorithms. Model evaluation conducted through K-Fold cross-validation demonstrated that the proposed model achieved an accuracy of 59.00%, an F1-Score of 58.80%, a precision of 58.80%, and a recall of 59.00%. While the performance indicates room for improvement, this study lays the groundwork for enhancing sign language recognition systems to support the preservation and broader adoption of SIBI in Indonesia.

Downloads

Download data is not yet available.

References

H. Halim and L. Lina, “Aplikasi Pengidentifikasi Bahasa Isyarat Berdasarkan Gerak Tubuh Secara Real Time Menggunakan YOLO,” Simtek : jurnal sistem informasi dan teknik komputer, vol. 8, no. 2, Art. no. 2, Oct. 2023, doi: 10.51876/simtek.v8i2.215.

I. Solihin, “Kesetaraan Sosial Dalam Bahasa Isyarat Sebagai Identitas Tuli Di Komunitas Rumah Setara Palembang,” Eastasouth Journal of Effective Community Services, vol. 1, no. 03, Art. no. 03, Apr. 2023, doi: 10.58812/ejecs.v1i03.49.

E. Rakun, S. Muzahidin, I. S. A. Darmana, and W. Setiaji, "SIBI (Sign System Indonesian Language) Text-to-3D Animation Translation Mobile Application," International Journal of Advanced Computer Science and Applications (IJACSA), vol. 13, no. 9, Art. no. 9, Dec. 2022, doi: 10.14569/IJACSA.2022.0130950.

N. Palfreyman, "Social meanings of linguistic variation in BISINDO (Indonesian Sign Language)," Asia-Pacific Language Variation, vol. 6, no. 1, pp. 89–118, Jul. 2020, doi: 10.1075/aplv.00008.pal.

A. R. Ardiansyah, A. H. Nur’azizan, and R. Fernandis, “Implementasi Deteksi Bahasa Isyarat Tangan Menggunakan OpenCV dan MediaPipe,” Seminar Nasional Teknologi & Sains, vol. 3, no. 1, pp. 331–337, Jan. 2024.

I. I. Arifah, F. N. Fajri, and G. Q. O. Pratamasunu, "Deteksi Tangan Otomatis Pada Video Percakapan Bahasa Isyarat Indonesia Menggunakan Metode YOLO Dan CNN," Journal of Applied Informatics and Computing, vol. 6, no. 2, Art. no. 2, Nov. 2022, doi: 10.30871/jaic.v6i2.4694.

M. B. Subkhi, M. Y. Trinurais, R. K. A. Wibowo, and B. R. Prakosa, “Deteksi Bahasa Isyarat Berdasarkan SIBI (Sistem Bahasa Isyarat) menggunakan Transfer Learning,” Seminar Nasional Teknologi & Sains, vol. 3, no. 1, pp. 361–369, Jan. 2024, doi: 10.29407/stains.v3i1.4347.

M. F. Naufal and S. F. Kusuma, “Analisis Perbandingan Algoritma Machine Learning dan Deep Learning untuk Klasifikasi Citra Sistem Isyarat Bahasa Indonesia (SIBI),” Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 10, no. 4, Art. no. 4, 2024, doi: 10.25126/jtiik.20241046823.

Q. Dong, "Mobile Robot Vision Image Feature Recognition Method Based on Machine Vision," EAI Endorsed Trans Energy Web, vol. 10, Dec. 2023, doi: 10.4108/ew.3450.

J. Li, C. Li, J. Han, Y. Shi, G. Bian, and S. Zhou, "Robust Hand Gesture Recognition Using HOG-9ULBP Features and SVM Model," Electronics, vol. 11, no. 7, Art. no. 7, Jan. 2022, doi: 10.3390/electronics11070988.

N.-T. Do, S.-H. Kim, H.-J. Yang, and G.-S. Lee, "Robust Hand Shape Features for Dynamic Hand Gesture Recognition Using Multi-Level Feature LSTM," Applied Sciences, vol. 10, no. 18, Art. no. 18, Jan. 2020, doi: 10.3390/app10186293.

Y. Li and P. Zhang, "Static hand gesture recognition based on hierarchical decision and classification of finger features," Science Progress, vol. 105, no. 1, p. 003685042210863, Jan. 2022, doi: 10.1177/00368504221086362.

N. Gozzi et al., "Image Embeddings Extracted from CNNs Outperform Other Transfer Learning Approaches in Classification of Chest Radiographs," Diagnostics, vol. 12, no. 9, Art. no. 9, Sep. 2022, doi: 10.3390/diagnostics12092084.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, "Rethinking the Inception Architecture for Computer Vision," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, Jun. 2016, pp. 2818–2826. doi: 10.1109/CVPR.2016.308.

X. Fang et al., "Ultrasound image intelligent diagnosis in community-acquired pneumonia of children using convolutional neural network-based transfer learning," Front. Pediatr., vol. 10, Nov. 2022, doi: 10.3389/fped.2022.1063587.

J. Loyola-Fuentes, L. Pietrasanta, M. Marengo, and F. Coletti, "Machine Learning Algorithms for Flow Pattern Classification in Pulsating Heat Pipes," Energies, vol. 15, no. 6, Art. no. 6, Jan. 2022, doi: 10.3390/en15061970.

X. Fu, Y. Chen, J. Yan, Y. Chen, and F. Xu, "BGRF: A broad granular random forest algorithm," Journal of Intelligent & Fuzzy Systems, vol. 44, no. 5, pp. 8103–8117, Jan. 2023, doi: 10.3233/JIFS-223960.

D. Angelakis, E. Ventouras, and P. Asvestas, “Brain Signals to Actions Using Machine Learning,” Engineering Proceedings, vol. 50, no. 1, Art. no. 1, 2023, doi: 10.3390/engproc2023050007.

H. Bassi, R. P. Yim, J. Vendrow, R. Koduluka, C. Zhu, and H. Lyu, "Learning to predict synchronization of coupled oscillators on randomly generated graphs," Sci Rep, vol. 12, no. 1, p. 15056, Sep. 2022, doi: 10.1038/s41598-022-18953-8.

S. Akhtar et al., "Short-Term Load Forecasting Models: A Review of Challenges, Progress, and the Road Ahead," Energies, vol. 16, no. 10, Art. no. 10, Jan. 2023, doi: 10.3390/en16104060.

G. Alwafi and B. Fakieh, "A machine learning model to predict privacy fatigued users from social media personalized advertisements," Sci Rep, vol. 14, no. 1, p. 3685, Feb. 2024, doi: 10.1038/s41598-024-54078-w.

M. D. Chaity and J. van Aardt, "Exploring the Limits of Species Identification via a Convolutional Neural Network in a Complex Forest Scene through Simulated Imaging Spectroscopy," Remote Sensing, vol. 16, no. 3, Art. no. 3, Jan. 2024, doi: 10.3390/rs16030498.

B. Düzgün, A. Çayır, U. Ünal, and H. Dağ, "Network intrusion detection system by learning jointly from tabular and text-based features," Expert Systems, vol. 41, no. 4, p. e13518, 2024, doi: 10.1111/exsy.13518.

C.-S. Cheng, P.-W. Chen, Y.-C. Hsieh, and Y.-T. Wu, "Multivariate Process Control Chart Pattern Classification Using Multi-Channel Deep Convolutional Neural Networks," Mathematics, vol. 11, no. 15, Art. no. 15, Jan. 2023, doi: 10.3390/math11153291.

W. Peng and J. Huang, "An Approach to Incorporating Implicit Knowledge in Object Detection Models," Applied Sciences, vol. 14, no. 5, Art. no. 5, Jan. 2024, doi: 10.3390/app14052124.

S. Rhouas, A. E. Attaoui, and N. E. Hami, "Enhancing currency prediction in international e-commerce: Bayesian-optimized random forest approach using the Klarna dataset," International Journal of Electrical and Computer Engineering (IJECE), vol. 14, no. 3, Art. no. 3, Jun. 2024, doi: 10.11591/ijece.v14i3.pp3177-3186.