Cohn, A.G. orcid.org/0000-0002-7652-8907 and Blackwell, R.E. orcid.org/0000-0002-0554-8062 (2024) Evaluating the Ability of Large Language Models to Reason About Cardinal Directions. In: Adams, B., Griffin, A.L., Scheider, S. and McKenzie, G., (eds.) Leibniz International Proceedings in Informatics, LIPIcs. 16th International Conference on Spatial Information Theory (COSIT 2024), 17-20 Sep 2024, Québec City, Canada. Schloss Dagstuhl – Leibniz-Zentrum für Informatik , 28:1-28:9. ISBN 978-3-95977-330-0
Abstract
We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-created with ChatGPT, focuses largely on recall of world knowledge about CDs; the second is generated from a set of templates, comprehensively testing an LLM’s ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first, second or third person. Even with a temperature setting of zero, Our experiments show that although LLMs are able to perform well in the simpler dataset, in the second more complex dataset no LLM is able to reliably determine the correct CD, even with a temperature setting of zero.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Editors: |
|
Copyright, Publisher and Additional Information: | © Anthony G Cohn and Robert E Blackwell. This is an open access article under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited. |
Keywords: | Large Language Models, Spatial Reasoning, Cardinal Directions |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 10 Mar 2025 11:34 |
Last Modified: | 10 Mar 2025 11:34 |
Status: | Published |
Publisher: | Schloss Dagstuhl – Leibniz-Zentrum für Informatik |
Identification Number: | 10.4230/LIPIcs.COSIT.2024.28 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:224231 |