Lin, F., La Malfa, E., Hofmann, V. et al. (3 more authors) (2024) Graph-enhanced large language models in asynchronous plan reasoning. In: Proceedings of Machine Learning Research. ICML'24: Proceedings of the 41st International Conference on Machine Learning, 21-27 Jul 2024, Vienna, Austria. ACM , pp. 30108-30134.
Abstract
Planning is a fundamental property of human intelligence. Reasoning about asynchronous plans is challenging since it requires sequential and parallel planning to optimize time costs. Can large language models (LLMs) succeed at this task? Here, we present the first large-scale study investigating this question. We find that a representative set of closed and open-source LLMs, including GPT-4 and LLaMA-2, behave poorly when not supplied with illustrations about the task-solving process in our benchmark AsyncHow. We propose a novel technique called Plan Like a Graph (PLaG) that combines graphs with natural language prompts and achieves state-of-the-art results. We show that although PLaG can boost model performance, LLMs still suffer from drastic degradation when task complexity increases, highlighting the limits of utilizing LLMs for simulating digital devices. We see our study as an exciting step towards using LLMs as efficient autonomous agents. Our code and data are available at https://github.com/fangru-lin/graph-llm-asynchow-plan.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | This is an author produced version of a conference paper accepted for publication in ICML, made available under the terms of the Creative Commons Attribution License (CC-BY), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited. |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) |
Funding Information: | Funder Grant number ESRC (Economic and Social Research Council) ES/W003473/1 Alan Turing Institute Not Known Foreign Commonwealth and Development Office Not Known Alan Turing Institute Not Known |
Depositing User: | Symplectic Publications |
Date Deposited: | 10 Mar 2025 16:14 |
Last Modified: | 10 Mar 2025 16:14 |
Status: | Published |
Publisher: | ACM |
Identification Number: | 10.5555/3692070.3693283 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:224227 |