A Mechanistic Theory Unifying Value and Schema in the Prefrontal Cortex
TL;DR
A mechanistic model of PFC that unifies value and structure through behavioural flexibility is developed and it is demonstrated that recurrent neural networks meta-trained with either reinforcement learning or supervised learning recapitulate this exact model on various value-based tasks.
Abstract
Animals exhibit behavioural flexibility by making reward-optimal decisions in unfamiliar situations. Computing value from minimal reward feedback requires leveraging known structure in the world. The Prefrontal Cortex (PFC) plays a pivotal role in both value computation and learning world structure, however mechanistic explanations for each—within the frameworks of Reinforcement Learning and Cognitive Maps—are largely distinct. We develop a mechanistic model of PFC that unifies value and structure through behavioural flexibility. In this model, value acts as a control signal that gates structured working memory representations of the current context in order to ultimately control behaviour. This is a significant departure from the conventional perspective of value, in which it is used for action readout. Critically, this model generalises to novel contexts without the need for additional synaptic plasticity. We demonstrate that recurrent neural networks (RNNs) meta-trained with either reinforcement learning or supervised learning recapitulate this exact model on various value-based tasks. These results not only provide a parsimonious explanation of both the value and schema frameworks of PFC function, but also propose a new mechanistic interpretation of the role of PFC in flexible decision making.