Alnefaie, S.S.M., Atwell, E. and Alsalka, M.A. (2023) Is GPT-4 a Good Islamic Expert for Answering Quran Questions? In: Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023). 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), 20-21 Oct 2023, Taipei City, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP) , pp. 124-133.
Abstract
In this research, we investigated GPT-4 as a question-answering model for the Holy Quran. As a first step, we built the Quran question–answer pair (QUQA) dataset, comprising 2,189 questions, and made it freely available via our repository. This dataset was then used to benchmark the performance of the current Generative Pre-trained Transformer 4 (GPT-4) model from the OpenAI research laboratory. The results show that GPT-4 did not do well with this dataset, with a 0.23 partial Average Precision (pAP) score, 0.26 F1@1 score, and 0.19 Exact Match (EM) score. Therefore, further improvement is needed for Classical Arabic responses generated by GPT model.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2023 ACL. This is an open access article under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited. |
Keywords: | GPT-4, large language model, and Quran question–answer pair (QUQA) dataset |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 19 Dec 2023 10:29 |
Last Modified: | 19 Dec 2023 10:29 |
Published Version: | https://aclanthology.org/2023.rocling-1.15/ |
Status: | Published |
Publisher: | The Association for Computational Linguistics and Chinese Language Processing (ACLCLP) |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:206760 |