Identification of Customer Facial Expressions to Improve Service Experience Using Convolutional Neural Networks (CNN)

Icha Miranti Irzan; Sriani Sriani

doi:10.24167/sisforma.v12i2.14314

Identification of Customer Facial Expressions to Improve Service Experience Using Convolutional Neural Networks (CNN)

Icha Miranti Irzan, Sriani Sriani

Abstract

Service experience is an important factor that determines the success and sustainability of a business. Understanding customer facial expressions can help business owners identify emotional responses that arise during the service process. This study develops a model for identifying customer facial expressions using a Convolutional Neural Network (CNN) with the MobileNetV2 architecture to enhance the customer service experience at Toko Saudara. This study aims to develop a system model that can identify customer facial expressions and measure the accuracy of the model in classifying facial expressions based on the dataset used. The model was trained using a transfer learning approach with training parameters of 50 epochs, a batch size of 32, and a learning rate of 0.0001 using the Adam optimizer. The dataset consisted of 2000 augmented images from customer CCTV recordings with two main classes, namely satisfied and dissatisfied. The dataset was split into 80% training, 10% validation, and 10% testing portions. The model achieved a training accuracy of 95.94% with a loss value of 0.1377, and testing performance reached 98% accuracy, with precision, recall, and F1-score all at 98%. The resulting model is able to accurately identify customer facial expressions so that it can be used by businesses to understand customer emotional responses and improve service quality.

Keywords

Identification; Facial Expression; Service Experience; Convolutional Neural Network; MobileNetV2

Full Text:

PDF

References

A. Wardhana, Consumer Behavior In The Digital Era 4.0. Jawa Tengah: Eureka Media Aksara, 2024.

Admin, “Pengertian Pengalaman Pelanggan: Panduan Lengkap untuk Meningkatkan Kepuasan Pelanggan,” Bidang Usaha, 2024. https://bidangusaha.co.id/pengertian-pengalaman-pelanggan/

P. Indonesia, “Consumer Insights Survey 2023,” 2023. https://www.pwc.com/id/en/consumer-industrial-products-services/indonesia-gcis-2023-placemat.pdf (accessed Apr. 02, 2025).

D. P. Anggraini, D. E. Fitriani, F. Ulfah, and T. Agustin, “Klasifikasi Ekspresi Wajah Menggunakan Convolutional Neural Network (CNN) Dengan Perbandingan Dua Model Dimodifikasi,” Semin. Nas. Amikom Surakarta, pp. 160–171, 2024.

D. Prasetyawan and R. Gatra, “Model Convolutional Neural Network untuk Mengukur Kepuasan Pelanggan Berdasarkan Ekspresi Wajah,” J. Tek. Inform. dan Sist. Inf., vol. 8, no. 3, pp. 661–673, 2022, doi: 10.28932/jutisi.v8i3.5493.

F. Ramadhani, S. Rahardiantoro, and M. Masjkur, “Acne Severity Classification Study Using Convolutional Neural Network Algorithm with MobileNetV2 Architecture,” Indones. J. Stat. Its Appl., vol. 8, no. 2, pp. 112–128, 2024, doi: 10.29244/ijsa.v8i2p112-128.

O. Yuliardi, I. Zufria, and M. D. Irawan, “Sistem Pakar Mendiagnosis Penyakit pada Tanaman Jeruk Menggunakan Metode Dempster Shafer,” J. Compr. Sci., vol. 2, no. 1, pp. 389–397, 2023, doi: 10.59188/jcs.v2i1.224.

A. T. Akbar, S. Saifullah, and H. Prapcoyo, “Klasifikasi Ekspresi Wajah Menggunakan Covolutional Neural Network,” J. Teknol. Inf. dan Ilmu Komput., vol. 11, no. 6, pp. 1399–1412, 2024, doi: 10.25126/jtiik.2024118888.

Lina, A. M. Adhitya, Wasino, and D. Ajienegoro, “Identifikasi Emosi Wajah Pengguna Konferensi Video Menggunakan Convolutional Neural Network Dengan Arsitektur Vgg-16,” J. Teknol. Inf. dan Ilmu Komput., vol. 9(5), pp. 1047–1054, 2022.

M. Diqi, “Mengenal Deep Learning,” ResearchGate, 2023. https://www.researchgate.net/publication/367636088_MENGENAL_DEEP_LEARNING

M. N. Kholis, “Mengenal Convolutional Neural Network (CNN),” Medium, 2023. https://muchamadnurkholis.medium.com/mengenal-convolutional-neural-network-cnn-36969caa709 (accessed Mar. 13, 2025).

Supiyandi, W. E. Judistira, S. Nurliani, R. S. Darmono, and I. Putri, “Penerapan Deep Learning dalam Analisis Citra Gigi,” J. Pendidik. Dan Ilmu Sos., vol. 2, no. 4, pp. 117–128, 2024, doi: 10.54066/jupendis.v2i4.2165.

D. Alamsyah and D. Pratama, “Implementasi Convolutional Neural Networks (CNN) untuk Klasifikasi Ekspresi Citra Wajah pada FER-2013 Dataset,” J. Teknol. Inf., vol. 4, no. 2, pp. 350–355, 2020, doi: 10.36294/jurti.v4i2.1714.

M. S. A. M. Al-Gaashani, W. Xu, and E. Y. Obsie, “MobileNetV2-Based Deep Learning Architecture With Progressive Transfer Learning For Accurate Monkeypox Detection,” Appl. Soft Comput., vol. 169, 2025, doi: https://doi.org/10.1016/j.asoc.2024.112553.

A. Hadhiwibowo, S. R. Asri, and R. A. Dinata, “Penerapan Convolutional Neural Network dengan Arsitektur Mobilenetv2 Pada Aplikasi Penerjemah dan Pembelajaran Bahasa Isyarat,” TIN Terap. Inform. Nusant., vol. 4, no. 8, pp. 518–523, 2024, doi: 10.47065/tin.v4i8.4879.

DOI: https://doi.org/10.24167/sisforma.v12i2.14314