TensorFlow Based AI Training for Translating Indonesian Sign Language (BISINDO)

Akbar Sujiwa; Nailul Hasan; Nenni Mona  Aruan

doi:10.36456/best.vol7.no1.10312

Authors

Akbar Sujiwa Universitas Pembangunan Nasional "Veteran" Jawa Timur
Nailul Hasan Universitas Pembangunan Nasional "Veteran" Jawa Timur
Nenni Mona Aruan Universitas Pembangunan Nasional "Veteran" Jawa Timur

DOI:

https://doi.org/10.36456/best.vol7.no1.10312

Keywords:

BISINDO, Sign Language, TensorFlow, AI, Translating

Abstract

Communication is an important aspect of social life, including for deaf people who rely on sign language as their primary means of interaction. However, the limited understanding of sign language in society, especially BISINDO (Bahasa Isyarat Indonesia), is often a barrier to effective communication. This study aims to develop a simple sign language translator system using TensorFlow and Keras-based machine learning technology, as well as image processing support from OpenCV. The system is trained to recognize six basic signs in BISINDO, namely “Halo”, “Nama”, “Saya, “A”, “D”, and “I”, using hand images as training data. Each word is represented by 20 images with variations in horizontal position. The test results show that the system is able to recognize signs with fairly good accuracy under certain conditions, but still has difficulty when faced with variations in hand positions that are different from the training data. This study shows the potential of sign language recognition technology in supporting inclusive communication, as well as the importance of data enrichment and variation in viewpoints in the model training process so that the system can function more optimally in various real conditions.

References

Jecho Gustian Althanio, Rasianna BR. Saragih, and Yuliati, “Komunikasi Interpersonal Pada Penyandang Tunarungu Dalam Interaksi Sosial di Kota Bengkulu,” JURNAL KAGANGA, vol. 8, no. 2, pp. 126–135, Oct. 2024.

A. S. Nugraheni, A. P. Husain, and H. Unayah, “OPTIMALISASI PENGGUNAAN BAHASA ISYARAT DENGAN SIBI DAN BISINDO PADA MAHASISWA DIFABEL TUNARUNGU DI PRODI PGMI UIN SUNAN KALIJAGA,” Jurnal Holistika, vol. 5, no. 1, pp. 28–33, Feb. 2023, doi: 10.24853/holistika.5.1.28-33.

Nisria, Mustafa, and Hadis, “IMPLEMENTASI BISINDO DALAM BERKOMUNIKASI PADA SESAMA ANAK TUNARUNGU,” PINISI JOURNAL OF EDUCATION, vol. 2, no. 5, pp. 1–10, Sep. 2022.

J. L. J. Boobala A., C. Charan Kesava Reddy, C. A. Reddy, and C. Bala Venkata Sai Rohith, “Real-Time Sign Language and Audio Conversion Using AI,” 2024 International Conference on Communication, Control, and Intelligent Systems (CCIS), pp. 1–6, Dec. 2024, doi: 10.1109/CCIS63231.2024.10932061.

M. Abadi et al., “TensorFlow: A system for large-scale machine learning,” Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, pp. 265–283, May 2016, Accessed: May 11, 2025. [Online]. Available: https://arxiv.org/pdf/1605.08695

F. Ertam, “Data classification with deep learning using tensorflow,” 2nd International Conference on Computer Science and Engineering, UBMK 2017, pp. 755–758, Oct. 2017, doi: 10.1109/UBMK.2017.8093521.

M. Amsaprabhaa, H. S. Sree, Jayashre, K. Muthamizhvalavan, N. Gummaraju, and S. Padmajaa, “American Sign Language Real Time Detection Using TensorFlow and Keras in Python,” 2024 3rd International Conference for Innovation in Technology, INOCON 2024, 2024, doi: 10.1109/INOCON60754.2024.10511469.

D. H. Noronha, K. Gibson, B. Salehpour, and S. J. E. Wilton, “LeFlow: Automatic Compilation of TensorFlow Machine Learning Applications to FPGAS,” Proceedings - 2018 International Conference on Field-Programmable Technology, FPT 2018, pp. 396–399, Dec. 2018, doi: 10.1109/FPT.2018.00082.

A. O. Hashi, S. Z. M. Hashim, and A. B. Asamah, “A Systematic Review of Hand Gesture Recognition: An Update From 2018 to 2024,” IEEE Access, 2024, doi: 10.1109/ACCESS.2024.3421992.

E. Krsak and T. Kello, “Improving teoretic train driving time with AI and TensorFlow,” ISIA 2020 - Proceedings, 4th International Symposium on Informatics and its Applications, Dec. 2020, doi: 10.1109/ISIA51297.2020.9416531.

W. F. Bastari, A. Sujiwa, and R. Setyobudi, “Penerapan Internet Of Things Pada Aplikasi Alat Deteksi Dan Monitoring Tekanan Darah,” Seminar Nasional Hasil Riset dan Pengabdian, vol. 5, pp. 609–621, Aug. 2023, Accessed: May 16, 2025. [Online]. Available: https://snhrp.unipasby.ac.id/prosiding/index.php/snhrp/article/view/601

A. S. Karche, A. V. Kamble, K. A. Maru, S. S. Kedari, and D. D. Sarpate, “American Sign Language Recognition Application,” 2025 International Conference on Emerging Smart Computing and Informatics (ESCI), pp. 1–6, Mar. 2025, doi: 10.1109/ESCI63694.2025.10988218.

P. Venkadesh, G. Jelin Taric, P. Mary mariyal, S. V. Divya, S. Danumithra, and K. Anuranjani, “SignChatAI-Generative AI for Deaf-and-Mute Community using Indian Sign Language(ISL),” International Conference on Computing and Intelligent Reality Technologies, Proceedings of ICCIRT 2024, pp. 84–88, 2024, doi: 10.1109/ICCIRT59484.2024.10921817.

G. Serrano and D. Kwak, “Real-time Sign Language Recognition Using Computer Vision and AI,” Proceedings - 2023 International Conference on Computational Science and Computational Intelligence, CSCI 2023, pp. 1214–1220, 2023, doi: 10.1109/CSCI62032.2023.00198.

I. S. M. Dissanayake, P. J. Wickramanayake, M. A. S. Mudunkotuwa, and P. W. N. Fernando, “Utalk: Sri Lankan sign language converter mobile app using image processing and machine learning,” ICAC 2020 - 2nd International Conference on Advancements in Computing, Proceedings, pp. 31–36, Dec. 2020, doi: 10.1109/ICAC51239.2020.9357300.

A. Sujiwa and R. R. Dianto, “INFUSION MONITORING SYSTEM FOR PATIENTS BASED ON THE INTERNET OF THINGS (IOT) WITH ANDROID NOTIFICATION SYSTEM,” BEST : Journal of Applied Electrical, Science, & Technology, vol. 4, no. 2, pp. 41–46, Nov. 2022, doi: 10.36456/BEST.VOL4.NO2.6172.

R. R. Milinda, Sara, B. Ramsiej, R. Yasmin, V. D. Ambeth Kumar, and K. Sekar, “Sign Language Detection in Real-Time Applications,” International Conference on Computing and Intelligent Reality Technologies, Proceedings of ICCIRT 2024, pp. 1–4, 2024, doi: 10.1109/ICCIRT59484.2024.10921810.

C. Mattmann, Machine Learning with TensorFlow, Second Edition. Manning, 2020. [Online]. Available: http://ieeexplore.ieee.org/document/10280532

D. Pietro Pau, T. A. Naramo, and M. D. Randriatsimiovalaza, “Coding Mel Spectrogram using Keras and Tensorflow for Home Appliances Tiny Classification,” Digest of Technical Papers - IEEE International Conference on Consumer Electronics, vol. 2023-January, 2023, doi: 10.1109/ICCE56470.2023.10043378.

A. Kumari Sirivarshitha, K. Sravani, K. S. Priya, and V. Bhavani, “An approach for Face Detection and Face Recognition using OpenCV and Face Recognition Libraries in Python,” 2023 9th International Conference on Advanced Computing and Communication Systems, ICACCS 2023, pp. 1274–1278, 2023, doi: 10.1109/ICACCS57279.2023.10113066.

“GitHub - Kazuhito00/hand-gesture-recognition-using-mediapipe: MediaPipe(Python版)を用いて手の姿勢推定を行い、検出したキーポイントを用いて、簡易なMLPでハンドサインとフィンガージェスチャーを認識するサンプルプログラムです。（Estimate hand pose using MediaPipe(Python version). This is a sample program that recognizes hand signs and finger gestures with a simple MLP using the detected key points.）.” Accessed: May 19, 2025. [Online]. Available: https://github.com/Kazuhito00/hand-gesture-recognition-using-mediapipe