A fundamental question in neuroscience is how the brain creates an internal model of the world to guide actions using sequences of ambiguous sensory information. This is naturally formulated as a reinforcement learning problem under partial observations, where an agent must estimate relevant latent variables in the world from its evidence, anticipate possible future states, and choose actions that optimize total expected reward. This problem can be solved by control theory, which allows us to find the optimal actions for a given system dynamics and objective function. However, animals often appear to behave suboptimally. Why? We hypothesize that animals have their own flawed internal model of the world, and choose actions with the highest expected subjective reward according to that flawed model. We describe this behavior as rational but not optimal. The problem of Inverse Rational Control (IRC) aims to identify which internal model would best explain an agent’s actions. Our contribution here generalizes past work on Inverse Rational Control which solved this problem for discrete control in partially observable Markov decision processes. Here we accommodate continuous nonlinear dynamics and continuous actions, and impute sensory observations corrupted by unknown noise that is private to the animal. We first build an optimal Bayesian agent that learns an optimal policy generalized over the entire model space of dynamics and subjective rewards using deep reinforcement learning. Crucially, this allows us to compute a likelihood over models for experimentally observable action trajectories acquired from a suboptimal agent. We then find the model parameters that maximize the likelihood using gradient ascent. Our method successfully recovers the true model of rational agents. This approach provides a foundation for interpreting the behavioral and neural dynamics of animal brains during complex tasks.
|Original language||English (US)|
|Journal||Advances in Neural Information Processing Systems|
|State||Published - 2020|
|Event||34th Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online|
Duration: Dec 6 2020 → Dec 12 2020
Bibliographical noteFunding Information:
The authors thank Dora Angelaki, James Bridgewater, Kaushik Lakshminarasimhan, Baptiste Caziot, Zhengwei Wu, Rajkumar Raju, and Yizhou Chen for useful discussions. MK, SD, and XP were supported in part by an award from the McNair Foundation. SD and XP were supported in part by the Simons Collaboration on the Global Brain award 324143 and NSF 1450923 BRAIN 43092. MK and XP were supported in part by NSF CAREER Award IOS-1552868. MK was supported in part by National Research Foundation of Korea grant NRF-2020R1F1A1069182. PS and XP were supported in part by BRAIN Initiative grant NIH 5U01NS094368.
© 2020 Neural information processing systems foundation. All rights reserved.