Assistive e-learning technologies for special needs education with mobile sign language applications and deep learning-based recognition
Keywords:
Deep learning, Sign language recognition, Assistive learning, Special needs education, Mobile learning, Inclusive educationAbstract
This study explores the integration of assistive e-learning technologies designed to enhance educational accessibility for Deaf and hard-of-hearing learners. Specifically, it examines three innovative systems: (1) a mobile sign language application based on the Kitabah method for teaching Quranic literacy, (2) a deep learning-based sign language recognition (SLR) system using ResNet-18 for recognizing Arabic Hijaiyyah letters, and (3) a mobile sign language dictionary aimed at improving communication and literacy for Deaf learners. The Kitabah-based mobile application integrates Arabic script with sign language gestures, providing an interactive learning experience. A deep learning model (ResNet-18) is employed for sign language recognition (SLR), achieving 98% accuracy. The mobile sign language dictionary offers gesture-based learning content with usability testing yielding a System Usability Scale (SUS) score of 78.06, indicating effective user interaction. Results from user testing indicate significant improvements in learning outcomes: 85% of participants reported increased comfort with digital learning tools, 82.5% rated the system effective for learning sign language, and 90% reported improved recognition of sign words. The study demonstrates the potential of assistive e-learning technologies to create adaptive and accessible learning environments for students with special needs. This research contributes to the development of inclusive, AI-powered e-learning solutions that enhance the learning experience for Deaf and hard-of-hearing learners.
References
1. I. Dzulkifli, “Quranic Education for Deaf Students in Malaysia; Implementation and Challenges,” Nadwa: Jurnal Pendidikan Islam, 2022, doi: 10.21580/nw.2022.16.1.10823.
2. A. Rakhmadi, A. Yudhana, and Sunardi, “Integrating Social Responsibility in the Development of Sign Language Recognition Technology for Quranic Recitation,” in 2025 International Conference on Information Technology and Computing (ICITCOM), IEEE, Aug. 2025, pp. 30–35. doi: 10.1109/ICITCOM66635.2025.11265077.
3. M. B. Sasongko and N. Usman, “Analysis of The Quran Isyarat Learning Management at The Magelang Deaf Education Foundation,” Al Ulya, vol. 10, no. 01, pp. 20–40, 2025, doi: 10.32665/alulya.v10i1.3559.
4. M. D. Meitantya, C. A. Sari, E. H. Rachmawanto, and R. R. Ali, “VGG-16 Architecture on CNN for American Sign Language Classification,” Jurnal Teknik Informatika (Jutif), vol. 5, no. 4, pp. 1165–1171, Jul. 2024, doi: 10.52436/1.jutif.2024.5.4.2160.
5. U. Fadlilah et al., “Android Application Prototype of Basic BISINDO Introduction and Practice for General People,” Sep. 2022.
6. U. Fadlilah, D. Wismoyohadi, A. K. Mahamad, and B. Handaga, “Bisindo information system as potential daily sign language learning,” in AIP Conference Proceedings, American Institute of Physics Inc., Jun. 2019. doi: 10.1063/1.5112492.
7. A. Fajrah, M. A. Hairul, and M. Reissyaf, “Implementation of the Kitabah method in learning the Qur’an for deaf students in TPQ. Adzan As-Sami’ Gorontalo,” in Proceeding of International Conference on Islamic and Interdisciplinary Studies (ICIIS 2023), 2023. [Online]. Available: https://jurnal.uindatokarama.ac.id/index.php/iciis/issue/archive
8. B. Pamungkas, R. Wahab, and S. Suwarjo, “Teaching of the Quran and Hadiths Using Sign Language to Islamic Boarding School Students with Hearing Impairment,” International Journal of Learning, Teaching and Educational Research, vol. 22, no. 5, pp. 227–242, May 2023, doi: 10.26803/ijlter.22.5.11.
9. C. Singh, D. Sharma, A. P. Dubey, and N. Tyagi, “Sign Language Detection Using CNN-YOLOv8l,” in 2023 International Conference on Advances in Computation, Communication and Information Technology, ICAICCIT 2023, Institute of Electrical and Electronics Engineers Inc., 2023, pp. 12–17. doi: 10.1109/ICAICCIT60255.2023.10465792.
10. B. Y. Al-Khuraym and M. M. Ben Ismail, “Arabic Sign Language Recognition using Lightweight CNN-based Architecture,” International Journal of Advanced Computer Science and Applications, vol. 13, no. 4, pp. 319–328, 2022, doi: 10.14569/IJACSA.2022.0130438.
11. A. Rakhmadi, N. D. Wulandari, S. Rosad, and M. Syukri, “A Mobile Sign Language Dictionary: Enhancing Communication for the Deaf and Mute,” Mestro: Jurnal Teknik Mesin dan Elektro, vol. 7, no. 1, pp. 73–79, Jun. 2025, doi: 10.47685/mestro.v7i1.617.
12. A. Alnuaim, M. Zakariah, W. A. Hatamleh, H. Tarazi, V. Tripathi, and E. T. Amoatey, “Human-Computer Interaction with Hand Gesture Recognition Using ResNet and MobileNet,” Comput. Intell. Neurosci., vol. 2022, 2022, doi: 10.1155/2022/8777355.
13. N. Adaloglou et al., “A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition,” IEEE Trans. Multimedia, vol. 24, pp. 1750–1762, 2022, doi: 10.1109/TMM.2021.3070438.
14. I. Papastratis, C. Chatzikonstantinou, D. Konstantinidis, K. Dimitropoulos, and P. Daras, “Artificial intelligence technologies for sign language,” Sensors, vol. 21, no. 17, Sep. 2021, doi: 10.3390/s21175843.
15. M. Beri, R. Chauhan, K. S. Gill, and H. S. Pokhariyal, “A Comprehensive Review of Smart Glove Technologies for Enhancing Visual Impairment Assistance,” in Proceedings of International Conference on Emerging Technologies and Innovation for Sustainability, EmergIN 2024, Institute of Electrical and Electronics Engineers Inc., 2024, pp. 68–72. doi: 10.1109/EmergIN63207.2024.10961564.
16. X. He, Y. Lin, Z. Hu, X. Xu, R. Xu, and W. Xiang, “AI Chinese sign language recognition interactive system based on audio-visual integration,” in 2023 IEEE International Conference on Electrical, Automation and Computer Engineering, ICEACE 2023, Institute of Electrical and Electronics Engineers Inc., 2023, pp. 962–968. doi: 10.1109/ICEACE60673.2023.10442295.
17. U. Fadlilah et al., “Android Application Prototype of Basic BISINDO Introduction and Practice for General People,” in Proceeding of International Conference on Special Edcation in South East Asia Region (ICSAR 2022), Sep. 2022.
18. U. Fadlilah, A. K. Mahamad, and B. Handaga, “The Development of Android for Indonesian Sign Language Using Tensorflow Lite and CNN: An Initial Study,” in Journal of Physics: Conference Series, IOP Publishing Ltd, Apr. 2021. doi: 10.1088/1742-6596/1858/1/012085.
19. R. N. I. A. Razak and N. Senan, “Mobile Learning for Manually Coded Malay Sign Language Using Augmented Reality,” Journal of Soft Computing and Data Mining, vol. 3, no. 1, pp. 86–94, Jun. 2022, doi: 10.30880/jscdm.2022.03.01.009.
20. A. Rakhmadi, A. Yudhana, Sunardi, and S. Riyadi, “Enhancing Arabic Sign Recognition with ResNet: Towards Inclusive Quranic Technologies,” in 2025 12th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE), IEEE, Aug. 2025, pp. 1–7. doi: 10.1109/ICITACEE66165.2025.11232873.
21. I. Z. Saiful Bahri et al., “Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP,” Information (Switzerland), vol. 14, no. 6, Jun. 2023, doi: 10.3390/info14060319.
22. V. Kandukuri, S. R. Gundedi, V. Kamble, and V. Satpute, “Deaf and Mute Sign Language Translator on Static Alphabets Gestures using MobileNet,” in 2023 2nd International Conference on Paradigm Shifts in Communications Embedded Systems, Machine Learning and Signal Processing, PCEMS 2023, Institute of Electrical and Electronics Engineers Inc., 2023. doi: 10.1109/PCEMS58491.2023.10136074.
23. M. Al-Barham et al., “RGB Arabic Alphabets Sign Language Dataset,” Jan. 2023, [Online]. Available: http://arxiv.org/abs/2301.11932
24. Y. Ding, S. Huang, and R. Peng, “Data Augmentation and Deep Learning Modeling Methods on Edge-Device-Based Sign Language Recognition,” in Proceedings - 2020 5th International Conference on Information Science, Computer Technology and Transportation, ISCTT 2020, Institute of Electrical and Electronics Engineers Inc., Nov. 2020, pp. 490–497. doi: 10.1109/ISCTT51595.2020.00093.
25. S. A. Nivishna Shree, D. Sasikala, and S. A. Nehaa Shree, “Performance Analysis of Sign Language Recognition using Deep Neural Network Architecture,” in Proceedings - 4th International Conference on Smart Technologies, Communication and Robotics 2025, STCR 2025, Institute of Electrical and Electronics Engineers Inc., 2025. doi: 10.1109/STCR62650.2025.11020537.
26. S. C. Ke, Abd. K. Mahamad, S. Saon, U. Fadlilah, and B. Handaga, “Malaysian Sign Language Translator for Mobile Application,” in Proceedings of the 11th International Conference on Robotics, Vision, Signal Processing and Power Applications, N. M. Mahyuddin, N. R. Mat Noor, and H. A. Mat Sakim, Eds., Singapore: Springer Singapore, 2022, pp. 909–914.
27. P. Vlachogianni and N. Tselios, “Perceived usability evaluation of educational technology using the System Usability Scale (SUS): A systematic review,” Journal of Research on Technology in Education, vol. 54, no. 3, pp. 392–409, 2022, doi: 10.1080/15391523.2020.1867938.
28. W. H. Cheah, N. Mat Jusoh, M. M. T. Aung, A. Ab Ghani, and H. Mohd Amin Rebuan, “Mobile Technology in Medicine: Development and Validation of an Adapted System Usability Scale (SUS) Questionnaire and Modified Technology Acceptance Model (TAM) to Evaluate User Experience and Acceptability of a Mobile Application in MRI Safety Screening,” Indian Journal of Radiology and Imaging, vol. 33, no. 1, pp. 36–45, Jan. 2023, doi: 10.1055/s-0042-1758198.
29. D. Castilla et al., “Psychometric Properties of the Spanish Full and Short Forms of the System Usability Scale (SUS): Detecting the Effect of Negatively Worded Items,” Int. J. Hum. Comput. Interact., vol. 40, no. 15, pp. 4145–4151, 2024, doi: 10.1080/10447318.2023.2209840.
30. T. Alisya, M. L. Hamzah, E. Saputra, T. K. Ahsyar, and Syaifullah, “Evaluation of User Experience on ShopeePay Digital Wallet Using System Usability Scale (SUS) and User Experience Questionnaire (UEQ) Methods,” in 2023 3rd International Conference on Emerging Smart Technologies and Applications, eSmarTA 2023, Institute of Electrical and Electronics Engineers Inc., 2023. doi: 10.1109/eSmarTA59349.2023.10293705.
31. M. Hyzy et al., “System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis,” JMIR Mhealth Uhealth, vol. 10, no. 8, Aug. 2022, doi: 10.2196/37290.
Downloads
Published
Conference Proceedings Volume
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.