Word prediction using Dynamic Skip Connections along with ARABERT and LSTM in Arabic Language

Authors

Ahad Almalki, Information Technology Department, Faculty of Computing and Technology, King Abdulaziz University, Jeddah, Saudi ArabiaFollow
Faris Kateb, Information Technology Department, Faculty of Computing and Technology, King Abdulaziz University, Jeddah, Saudi ArabiaFollow
Rayan Mosli, Information Technology Department, Faculty of Computing and Technology, King Abdulaziz University, Jeddah, Saudi ArabiaFollow

Abstract

Natural Language Generation (NLG) plays a crucial role in modern digital tools, including chatbots, virtual support, content suggestions, and tailored marketing, making bots more responsive and reducing the need for human staff. While there's much research on NLG for languages like English, languages like Arabic, Urdu, and Chinese still face challenges. This study examines Arabic NLG's unique aspects, dialects, and word variations. With around 420 million Arabic speakers globally, it's crucial to advance NLG for this language. We compared three models: Long Short-Term Memory (LSTM), a mix of Bidirectional Encoder Representations from Transformers (BERT) and LSTM, and a version that adds Dynamic Skip Connection (DSC). Our aim is to find the best model for predicting Arabic words with the least mistakes. In our experiment, we found that adding DSC wasn't beneficial. However, combining BERT and LSTM with an attention mechanism reduced loss and favorable perplexity values.

Publication Date

2024

Received Date

23-Oct-2023

Revised Date

2-Jan-2024

Accepted Date

28-Feb-2024

Recommended Citation

Almalki, Ahad; Kateb, Faris; and Mosli, Rayan (2024) "Word prediction using Dynamic Skip Connections along with ARABERT and LSTM in Arabic Language," ASEAN Journal on Science and Technology for Development: Vol. 41: No. 1, Article 11.
DOI: https://doi.org/10.61931/2224-9028.1569
Available at: https://ajstd.ubd.edu.bn/journal/vol41/iss1/11