Assessing Performance in Extracting Topological, Direction and Distance Spatial Relations from Reddit using LLMs

This paper provides an initial exploration of the capabilities of large language models (LLMs) to extract spatial relations from unstructured social media text. The approach examines the performance of GPT-4o and Gemini 1.5-Pro using a diverse set of spatial relation terms, and seeks to determine whether certain spatial relation types (topological, distance and direction) are more challenging to extract. To evaluate, GPT-4o and Gemini 1.5-Pro output is compared to manually labeled spatial relation triplets from Reddit place descriptions. The findings demonstrate challenges in extracting spatial relations for LLMs, with the highest model only achieving 0.48 precision. However, performance varied across spatial relation types, as direction relations were extracted with higher precision (0.75) compared to distance relations (0.62) and topological relations (0.35).