Inverse rational control with partially observable continuous nonlinear dynamics

Minhae Kwon, Saurabh Daptardar, Paul Schrater, Xaq Pitkow

Research output: Contribution to journalConference articlepeer-review

1 Scopus citations

Abstract

A fundamental question in neuroscience is how the brain creates an internal model of the world to guide actions using sequences of ambiguous sensory information. This is naturally formulated as a reinforcement learning problem under partial observations, where an agent must estimate relevant latent variables in the world from its evidence, anticipate possible future states, and choose actions that optimize total expected reward. This problem can be solved by control theory, which allows us to find the optimal actions for a given system dynamics and objective function. However, animals often appear to behave suboptimally. Why? We hypothesize that animals have their own flawed internal model of the world, and choose actions with the highest expected subjective reward according to that flawed model. We describe this behavior as rational but not optimal. The problem of Inverse Rational Control (IRC) aims to identify which internal model would best explain an agent’s actions. Our contribution here generalizes past work on Inverse Rational Control which solved this problem for discrete control in partially observable Markov decision processes. Here we accommodate continuous nonlinear dynamics and continuous actions, and impute sensory observations corrupted by unknown noise that is private to the animal. We first build an optimal Bayesian agent that learns an optimal policy generalized over the entire model space of dynamics and subjective rewards using deep reinforcement learning. Crucially, this allows us to compute a likelihood over models for experimentally observable action trajectories acquired from a suboptimal agent. We then find the model parameters that maximize the likelihood using gradient ascent. Our method successfully recovers the true model of rational agents. This approach provides a foundation for interpreting the behavioral and neural dynamics of animal brains during complex tasks.

Original languageEnglish (US)
JournalAdvances in Neural Information Processing Systems
Volume2020-December
StatePublished - 2020
Event34th Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online
Duration: Dec 6 2020Dec 12 2020

Bibliographical note

Funding Information:
The authors thank Dora Angelaki, James Bridgewater, Kaushik Lakshminarasimhan, Baptiste Caziot, Zhengwei Wu, Rajkumar Raju, and Yizhou Chen for useful discussions. MK, SD, and XP were supported in part by an award from the McNair Foundation. SD and XP were supported in part by the Simons Collaboration on the Global Brain award 324143 and NSF 1450923 BRAIN 43092. MK and XP were supported in part by NSF CAREER Award IOS-1552868. MK was supported in part by National Research Foundation of Korea grant NRF-2020R1F1A1069182. PS and XP were supported in part by BRAIN Initiative grant NIH 5U01NS094368.

Publisher Copyright:
© 2020 Neural information processing systems foundation. All rights reserved.

Fingerprint

Dive into the research topics of 'Inverse rational control with partially observable continuous nonlinear dynamics'. Together they form a unique fingerprint.

Cite this