Huang, X.A., Malfa, E.L., Marro, S. et al. (3 more authors) (2024) A Notion of Complexity for Theory of Mind via Discrete World Models. [Preprint - arXiv CoRR]
Abstract
Theory of Mind (ToM) can be used to assess the capabilities of Large Language Models (LLMs) in complex scenarios where social reasoning is required. While the research community has proposed many ToM benchmarks, their hardness varies greatly, and their complexity is not well defined. This work proposes a framework to measure the complexity of ToM tasks. We quantify a problem's complexity as the number of states necessary to solve it correctly. Our complexity measure also accounts for spurious states of a ToM problem designed to make it apparently harder. We use our method to assess the complexity of five widely adopted ToM benchmarks. On top of this framework, we design a prompting technique that augments the information available to a model with a description of how the environment changes with the agents' interactions. We name this technique Discrete World Models (DWM) and show how it elicits superior performance on ToM tasks.
Metadata
Item Type: | Preprint |
---|---|
Authors/Creators: |
|
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) > Artificial Intelligence |
Funding Information: | Funder Grant number Alan Turing Institute Not Known Foreign Commonwealth and Development Office Not Known ESRC (Economic and Social Research Council) ES/W003473/1 |
Depositing User: | Symplectic Publications |
Date Deposited: | 14 Aug 2024 11:10 |
Last Modified: | 14 Aug 2024 11:10 |
Identification Number: | 10.48550/arXiv.2406.11911 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:216123 |