Real-Time Hand Gesture-Based Virtual Mouse System Using ESP32-CAM and OpenCV
DOI:
https://doi.org/10.29207/resti.v10i1.6609Keywords:
hand gesture recognition, image processing, IoT, real-time, virtual mouse controlAbstract
This research develops a virtual mouse control system that uses real-time hand gesture recognition implemented on an ESP32-CAM–based Internet of Things (IoT) platform. By leveraging OpenCV for image processing, the system translates hand gestures into corresponding mouse actions, including cursor movement, clicking, and scrolling. The study evaluates system performance under different lighting conditions and Wi-Fi speeds. Results show that higher Wi-Fi speeds significantly reduce latency, enabling smoother real time gesture recognition and high definition video output, while lower speeds lead to noticeable delays and reduced accuracy. The system successfully enables remote cursor control through camera captured hand gestures, supporting functions such as left click, right click, scrolling, and dragging. In latency tests performed with an internet speed of approximately 60 Mbps, the system achieved an average delay of about 50 milliseconds. Under optimal lighting conditions with minimal background interference, it accurately tracked hand movements and recognized gestures such as pointing, clicking, dragging, and scrolling in real time, achieving an accuracy rate of 95%. Despite its lower resolution compared to conventional webcams, the ESP32-CAM proves to be an effective solution for virtual mouse control, particularly in scenarios where high-resolution imaging is unnecessary. Its IoT capabilities support remote operation, allowing users to control the virtual mouse from a distance as long as both the ESP32-CAM and the computer remain connected. Overall, the findings highlight the ESP32-CAM based IoT platform as a viable alternative for gesture based interaction in real applications, although further enhancements are needed to improve performance in challenging environments.
Downloads
References
M. V. A, "Hand Gesture Recognition System as Virtual Mouse for HCI," International Journal for Research in Applied Science and Engineering Technology, vol. IX, no. 8, pp. 1098-1101, 2021. https://doi.org/10.22214/ijraset.2021.37503
B. Šumak, M. Špindler, M. Debeljak, M. Heričko and M. Pušnik, "An empirical evaluation of a hands-free computer interaction for users with motor disabilities," Journal of Biomedical Informatics, vol. XCVI, no. 1, p. 103249, 2019. https://doi.org/10.1016/j.jbi.2019.103249
M. MK and M. MS, "Virtual Mouse And Keyboard For Computer Interaction," international journal of engineering technology and management sciences, vol. VI, no. 4, pp. 205-213, 2022. https://doi.org/10.46647/ijetms.2022.v06i04.0036
G. Bhole, D. Bhingare, R. Bhise, S. Bhegade, S. Bhokare and A. Bhosle, "System Control using Hand Gesture," in 2023 International Conference for Advancement in Technology, ICONAT 2023, Goa, 2023. https://doi.org/10.1109/ICONAT57137.2023.10080493
S. S. Chakravarthi, B. N. K. Rao, N. P. Challa, R. Ranjana and A. Rai, "Gesture Recognition for Enhancing Human Computer Interaction," Journal of Scientific and Industrial Research, vol. LXXXII, no. 4, 2023. https://doi.org/10.56042/jsir.v82i04.72387
D. L. Falak, M. Khandait, S. M. Koteshwar, M. Upari and U. Singh, "Virtual Mouse Using Hand Gestures," International Research Journal of Modernization in Engineering Technology and Science, vol. IV, no. 11, pp. 81-85, 2022. https://doi.org/10.56726/irjmets30967
R. Annachhatre, M. Tamakuwala, P. Shinde, A. Jain and P. V. Kulkarni, "Virtual Mouse Using Hand Gesture Recognition - A Systematic Literature Review," in 2022 IEEE International Conference on Blockchain and Distributed Systems Security, ICBDS 2022, Pune, 2022. https://doi.org/10.1109/ICBDS53701.2022.9935944
N. Meghana, K. S. Lakshmi, M. N. L. Tejasree, K.Srujana and N.Ashok, "Gesture-Based Human-Computer Interaction," EPRA International Journal of Research & Development (IJRD), vol. VIII, no. 10, pp. 237-241, 2023. https://doi.org/10.36713/epra14757
C. Panagiotou, E. Faliagka, C. P. Antonopoulos and N. Voros, "Multidisciplinary ML Techniques on Gesture Recognition for People with Disabilities in a Smart Home Environment," AI, vol. VI, no. 1, pp. 1-17, 2025. https://doi.org/10.3390/ai6010017
V. Chang, R. O. Eniola, L. Golightly and Q. A. Xu, "An Exploration into Human–Computer Interaction: Hand Gesture Recognition Management in a Challenging Environment," SN Computer Science, vol. IV, p. 441, 2023. https://doi.org/10.1007/s42979-023-01751-y
S. Tirpude, D. Naidu, P. Rajkarne, S. Sarile, N. Saraf and R. Maheshwari, "Realtime Hand Gesture Recognition System for Human Computer Interaction," International Journal of Next-Generation Computing, vol. XIV, no. 1, pp. 264-269, 2023. https://doi.org/10.47164/ijngc.v14i1.1097
E. Fifi, E. Castrillón, G. Stöttinger, M. P. Cuellar and D. P. Morales, "Hand Gesture Recognition on Edge Devices: Sensor Technologies, Algorithms, and Processing Hardware," Sensors, vol. XXV, no. 6, p. 1687, 2025. https://doi.org/10.3390/s25061687
M. M. Rahman, A. Uzzaman, F. Khatun, M. Aktaruzzaman and N. Siddique, "A comparative study of advanced technologies and methods in hand gesture analysis and recognition systems," Expert Systems with Applications, vol. CCLXVI, p. 125929, 2025. https://doi.org/10.1016/j.eswa.2024.125929
V. L. Bhale, N. L. Sumnure, H. S. Maske and D. S. S. Patil, "Robot Car Using ESP32 Cam," International Journal of Advanced Research in Science, Communication and Technology (IJARSCT), vol. III, no. 1, pp. 494-497, 2023. https://doi.org/10.48175/IJARSCT-11280
V. V. T. Reddy, T. Dhyanchand, G. V. Krishna and S. Maheshwaram, "Virtual Mouse Control Using Colored Finger Tips and Hand Gesture Recognition," in Proceedings of 2020 IEEE-HYDCON International Conference on Engineering in the 4th Industrial Revolution, HYDCON 2020, Hyderabad, 2020. https://doi.org/10.1109/HYDCON48903.2020.9242677
S, S. S, R. B. S. and R. A. R, "Virtual mouse using hand gestures by skin recognition," Journal of Population Therapeutics and Clinical Pharmacology, vol. XXX, no. 7, pp. 251-258, 2023. https://doi.org/10.47750/jptcp.2023.30.07.030
M. Mishra, A. Gupta, V. Mehta and R. Ranjan, "Virtual Mouse Input Control using Hand Gestures," in 7th International Conference on Trends in Electronics and Informatics, ICOEI 2023 - Proceedings, Tirunelveli, 2023. https://doi.org/10.1109/ICOEI56765.2023.10125668
S. Ranjan, S. Anand, V. Gontia, A. Chandra, V. Sharan and S. SG, "Virtual Mouse Control using Hand Gesture," International Journal of Engineering Research in Computer Science and Engineering, vol. IX, no. 11, pp. 10-13, 2022. https://doi.org/10.36647/ijercse/09.11.art006
D. S. Tran, N. H. Ho, H. J. Yang, S. H. Kim and G. S. Lee, "Real-time virtual mouse system using RGB-D images and fingertip detection," Multimedia Tools and Applications, vol. LXXX, no. 7, p. 10473–10490, 2021. https://doi.org/10.1007/s11042-020-10156-5
J. D. Asmoro, A. T. Wibowo and M. Ridwan, "Virtual Mouse With Hand Gesture Recognition Based on Hand Landmark Model for Pointing Device," Jurnal Teknologi dan Sistem Informasi, vol. IX, no. 2, pp. 261-268, 2023. https://doi.org/10.33330/jurteksi.v9i2.2073
M. M. Sahiwala, S. R. Singh, D. Pawar, D. D. Jakasaniya and K. M. Patel, "Virtual Mouse using Coordinate Mapping," in 2nd International Conference on Sustainable Computing and Data Communication Systems, ICSCDS 2023 - Proceedings, Erode, 2023. https://doi.org/10.1109/ICSCDS56580.2023.10104598
T. K. Selvi, S. Sasirekha, M. Manikandan, M. O. Solomon and M. Vignesh, "Virtual Mouse using OpenCV and VNC," Journal of Information Technology and Digital World, vol. V, no. 2, pp. 169-179, 2023. https://doi.org/10.36548/jitdw.2023.2.007
S. Wattamwar, D. Ramgude and S. More, "Hand Gesture Based Virtual Mouse Using AI," Interantional Journal of Scientific Research in Engineering and Management, vol. VIII, no. 1, pp. 1-3, 2024. https://doi.org/10.55041/ijsrem28234
C. Saritha, Y. Phanitha and A. R. Krishna, "Virtual Mouse Operations Using Webcam," International Journal of Scientific Research in Science, Engineering and Technology, vol. X, no. 3, pp. 650-653, 2023. https://doi.org/10.32628/ijsrset23103152
S. Srivastava, R. Tiwari, A. Garg, Rishu and R. Tiwari, "Design and Development of Gesture Recognition Based Virtual Mouse System," in 2023 International Conference on Computational Intelligence, Communication Technology and Networking, CICTN 2023, Ghaziabad, 2023. https://doi.org/10.1109/CICTN57981.2023.10140390
Shukla, D. D. Katiyar and M. G. Goel, "Gesture Recognition-based AI Virtual Mouse," International Journal for Research in Applied Science and Engineering Technology, vol. X, no. 3, pp. 1583-1588, 2022. https://doi.org/10.22214/ijraset.2022.40937
R. Shukla, "AI Virtual Mouse Using Hand Gesture Recognition," International Journal of Research Publication and Reviews, vol. III, no. 10, pp. 168-173, 2022. https://doi.org/10.55248/gengpi.2022.3.10.13
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;





