Implementation of MobileNetV2 and OpenCV in a Real-Time Sign Language Recognition System

Marcellinus Ronan Narendra, Radius Tanone

Abstract


The main means of communication for the deaf is sign language, although communication hurdles are frequently caused by a lack of public comprehension. The goal of this project is to use the OpenCV library and MobileNetV2 architecture to create a real-time American Sign Language (ASL) recognition system. The ASL Alphabet from Kaggle, which has over 87,000 photos in 29 classes, is the dataset that was used. Convolutional Neural Networks (CNNs) based on MobileNetV2 with picture preprocessing and data augmentation were used to train the model. OpenCV was then used to combine the CNN with a camera for real-time implementation.

The evaluation results indicate an average F1-score of 83.6, recall of 83.8, and precision of 83.5. The system is responsive for direct interaction because it can operate at 18–22 frames per second (FPS) on a typical laptop. The system still has issues with complicated backdrops, low light levels, and gesture similarities between some letters. Overall, this study demonstrates the efficacy of MobileNetV2 and OpenCV in developing a lightweight, efficient real-time sign language recognition system that facilitates inclusive communication for individuals with hearing impairments.


Keywords


sign language; deep learning; MobileNetV2; OpenCV; real-time recognition

Full Text:

PDF

References


S. Apendi, C. Setianingsih, and M. W. Paryasto, “Deteksi Bahasa Isyarat Sistem Isyarat Bahasa Indonesia Menggunakan Metode Single Shot Multibox Detector | Apendi | eProceedings of Engineering,” Vol 10, No 1, vol. 10, no. 1, pp. 249–255, 2023, [Online]. Available: https://openlibrarypublications.telkomuniversity.ac.id/index.php/engineering/article/view/19322

R. Saputra et al., “Sistem Klasifikasi Alfabet Bahasa Isyarat Indonesia … 237,” pp. 237–248, 2025, [Online]. Available: https://data.mendeley.com/datasets/ywnjpbcz8m/1.

I. M. Pramono, Z. Niswati, and A. Agustina, “Model Penerjemah Bahasa Isyarat Indonesia Dengan Metode Convolutional Neural Network (Cnn),” Semnas Ristek (Seminar Nas. Ris. dan Inov. Teknol., vol. 8, no. 01, pp. 1–5, 2024, doi: 10.30998/semnasristek.v8i01.7124.

C. Han, J. Zhang, and H. Wu, “Fall Detection System Based on YOLO Algorithm and MobileNetV2 Model,” Proc. - 2024 10th Int. Conf. Syst. Informatics, ICSAI 2024, pp. 1–5, 2024, doi: 10.1109/ICSAI65059.2024.10893853.

C. A. Daniel, N. M. Liam, Y. Habchi, and S. Basnet, “Enhancing Language Learning with Real-Time Sign Language Recognition and Feedback,” URTC 2024 - 2024 IEEE MIT Undergrad. Res. Technol. Conf. Proc., pp. 1–5, 2024, doi: 10.1109/URTC65039.2024.10937578.

H. Y. A. Swasono, A. R. Himamunanto, and F. Maedjaja, “Implementasi YOLO11 dan OpenCV untuk Pengenalan Frasa dalam Video Real-Time Bahasa Isyarat Tangan,” MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 5, no. 3, pp. 1061–1073, 2025, doi: 10.57152/malcom.v5i3.2130.

F. Zaelani and Y. Miftahuddin, “Perbandingan Metode EfficientNetB3 dan MobileNetV2 Untuk Identifikasi Jenis Buah-buahan Menggunakan Fitur Daun,” J. Ilm. Teknol. Infomasi Terap., vol. 9, no. 1, pp. 1–11, 2022, doi: 10.33197/jitter.vol9.iss1.2022.911.

A. H. Gustsa and G. S. Permadi, “Sistem Deteksi Bahasa Isyarat Secara Realtime Dengan Tensorflow Object Detection dan Python Menggunakan Metode Convolutional Neural Network,” Inov. J. Ilm. Inov. …, vol. 7, no. 2, pp. 1–10, 2023, [Online]. Available: https://ejournal.unhasy.ac.id/index.php/inovate/article/view/4116

Agus Nugroho, R. Setiawan, A. Harris, and Beny, “Deteksi Bahasa Isyarat Bisindo Menggunakan Metode Machine Learning,” J. Process., vol. 18, no. 2, pp. 152–158, 2023, doi: 10.33998/processor.2023.18.2.1380.

A. R. Ardiansyah, A. H. Nur’azizan, and R. Fernandis, “Implementasi Deteksi Bahasa Isyarat Tangan Menggunakan OpenCV dan MediaPipe,” Stain. (Seminar Nas. Teknol. Sains), vol. 3, no. 1, pp. 331–337, 2024.

J. Eckardt-taing, “Real-Time American Sign Language Recognition Using Machine Learning,” 2024 2nd Int. Conf. Artif. Intell. Blockchain, Internet Things, pp. 1–5, doi: 10.1109/AIBThings63359.2024.10863041.

I. B. A. Peling, I. M. P. A. Ariawan, and G. B. Subiksa, “Deteksi Bahasa Isyarat Menggunakan Tensorflow Lite dan American Sign Language (ASL),” J. Krisnadana, vol. 3, no. 2, pp. 90–100, 2024, doi: 10.58982/krisnadana.v3i2.534.

A. Akhilesh, S. Sivakumar, M. A. R, G. Saranya, and K. Kumaran, “Real-Time Sign Language Interpretation and translation to speech Using CUDA and Machine Learning,” 2025 Int. Conf. Data Sci. Bus. Syst., pp. 1–5, 2025, doi: 10.1109/ICDSBS63635.2025.11031893.

S. Ashillah, S. Adventino Gulo, Y. Rahmi, and D. Yandra Niska, “Implementasi Algoritma Pengolahan Citra Digital untuk Perbaikan Kualitas Gambar,” J. SISKOM-KB (Sistem Komput. dan Kecerdasan Buatan), vol. 8, no. 3, pp. 274–280, 2025, doi: 10.47970/siskom-kb.v8i3.823.

J. Ulfah and N. Nurdin, “Implementasi Metode Deteksi Tepi Canny Untuk Menghitung Jumlah Uang Koin Dalam Gambar Menggunakan Opencv,” J. Inform. dan Tek. Elektro Terap., vol. 11, no. 3, pp. 420–426, 2023, doi: 10.23960/jitet.v11i3.3147.

G. F. Hanin and T. Dewayanto, “Peran Machine Learning Dan Deep Learning Dalam Pendeteksian Pencucian Uang-a Systematic Literature Review,” Diponegoro J. Account., vol. 13, no. 3, pp. 1–11, 2024, [Online]. Available: http://ejournal-s1.undip.ac.id/index.php/accounting

R. Muhammad, M. Y. Andi, and R. Abdul, “Deteksi Citra Daun Untuk Klasifikasi Penyakit PadiMenggunakan Pendekatan Deep Learning Dengan Model Cnn,” J. Teknol. Terpadu, vol. 10, no. 1, pp. 56–62, 2024.

R. Indraswari, W. Herulambang, and R. Rokhana, “Deteksi Penyakit Mata Pada Citra Fundus Menggunakan Convolutional Neural Network (CNN),” Techno.Com, vol. 21, no. 2, pp. 378–389, 2022, doi: 10.33633/tc.v21i2.6162.

Y. Pratama, E. Rasywir, F. Fachruddin, D. Kisbianty, and B. Irawan, “Eksperimen Layer Pooling menggunakan Standar Deviasi untuk Klasifikasi Dataset Citra Wajah dengan Metode CNN,” Build. Informatics, Technol. Sci., vol. 5, no. 1, pp. 200–210, 2023, doi: 10.47065/bits.v5i1.3604.

L. F. S. Scabini and O. M. Bruno, “Structure and performance of fully connected neural networks: Emerging complex network properties,” Phys. A Stat. Mech. its Appl., vol. 615, p. 128585, 2023, doi: 10.1016/j.physa.2023.128585.

E. Shavna Gracia, N. Anisa Sri Winarsih, and D. Nuswantoro, “Comparison of VGG16, MobileNetV2, InceptionV3, ResNet50, and Custom CNN Architectures for Furniture Image Classification,” Infotekmesin, vol. 16, no. 01, pp. 24–30, 2025, doi: 10.35970/infotekmesin.v16i1.2500.

J. J. Pangaribuan, R. Romindo, A. R. Mitra, O. P. Barus, and S. Prashanth, “American Sign Language Alphabet Recognition Using Convolutional Neural Network,” Proc. - 2024 2nd Int. Conf. Technol. Innov. Its Appl. ICTIIA 2024, pp. 1–6, 2024, doi: 10.1109/ICTIIA61827.2024.10761663.

C. Miller, T. Portlock, D. M. Nyaga, and J. M. O. Sullivan, “A review of model evaluation metrics for machine learning in genetics and genomics,” no. September, pp. 1–13, 2024, doi: 10.3389/fbinf.2024.1457619.




DOI: https://doi.org/10.24167/jbt.v6i1.14500

Refbacks

  • There are currently no refbacks.


e-ISSN: 2776-0332 | View My Stats