Whelan, M.T., Jimenez-Rodriguez, A., Prescott, T.J. orcid.org/0000-0003-4927-5390 et al. (1 more author) (2023) A robotic model of hippocampal reverse replay for reinforcement learning. Bioinspiration & Biomimetics, 18 (1). 015007. ISSN 1748-3182
Abstract
Hippocampal reverse replay, a phenomenon in which recently active hippocampal cells reactivate in the reverse order, is thought to contribute to learning, particularly reinforcement learning (RL), in animals. Here, we present a novel computational model which exploits reverse replay to improve stability and performance on a homing task. The model takes inspiration from the hippocampal-striatal network, and learning occurs via a three-factor RL rule. To augment this model with hippocampal reverse replay, we derived a policy gradient learning rule that associates place-cell activity with responses in cells representing actions and a supervised learning rule of the same form, interpreting the replay activity as a 'target' frequency. We evaluated the model using a simulated robot spatial navigation task inspired by the Morris water maze. Results suggest that reverse replay can improve performance stability over multiple trials. Our model exploits reverse reply as an additional source for propagating information about desirable synaptic changes, reducing the requirements for long-time scales in eligibility traces combined with low learning rates. We conclude that reverse replay can positively contribute to RL, although less stable learning is possible in its absence. Analogously, we postulate that reverse replay may enhance RL in the mammalian hippocampal-striatal system rather than provide its core mechanism.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2022 The Authors. This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
Keywords: | computational neuroscience; hippocampal reply; reinforcement learning; robotics; Animals; Robotics; Robotic Surgical Procedures; Hippocampus; Reinforcement, Psychology; Spatial Navigation; Mammals |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number EUROPEAN COMMISSION - HORIZON 2020 785907 EUROPEAN COMMISSION - HORIZON 2020 945539 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 13 Dec 2022 11:47 |
Last Modified: | 13 Dec 2022 11:47 |
Status: | Published |
Publisher: | IOP Publishing |
Refereed: | Yes |
Identification Number: | 10.1088/1748-3190/ac9ffc |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:194352 |