Two sides of the same coin: Beneficial and detrimental consequences of range adaptation in human reinforcement learning

Sophie Bavard; Aldo Rustichini; Stefano Palminteri

doi:10.1126/sciadv.abe0340

Two sides of the same coin: Beneficial and detrimental consequences of range adaptation in human reinforcement learning

Sophie Bavard, Aldo Rustichini, Stefano Palminteri

Economics (Twin Cities)

Research output: Contribution to journal › Article › peer-review

5 Scopus citations

Abstract

Evidence suggests that economic values are rescaled as a function of the range of the available options. Although locally adaptive, range adaptation has been shown to lead to suboptimal choices, particularly notable in reinforcement learning (RL) situations when options are extrapolated from their original context to a new one. Range adaptation can be seen as the result of an adaptive coding process aiming at increasing the signal-to-noise ratio. However, this hypothesis leads to a counterintuitive prediction: Decreasing task difficulty should increase range adaptation and, consequently, extrapolation errors. Here, we tested the paradoxical relation between range adaptation and performance in a large sample of participants performing variants of an RL task, where we manipulated task difficulty. Results confirmed that range adaptation induces systematic extrapolation errors and is stronger when decreasing task difficulty. Last, we propose a range-adapting model and show that it is able to parsimoniously capture all the behavioral results.

Original language	English (US)
Article number	eabe0340
Journal	Science Advances
Volume	7
Issue number	14
DOIs	https://doi.org/10.1126/sciadv.abe0340
State	Published - Apr 2021

Bibliographical note

Funding Information:
S.P. is supported by an ATIP-Avenir grant (R16069JS), the Programme Emergence(s) de la Ville de Paris, the Fondation Fyssen, the Fondation Schlumberger pour l’Education et la Recherche, the FrontCog grant (ANR-17-EURE-0017) and the Institut de Recherche en Santé Publique (IRESP, grant number: 20II138-00). S.B. is supported by MILDECA (Mission Interministerielle de Lutte contre les Drogues et les Conduites Addictives) and the EHESS (Ecole des Hautes Etudes en Sciences Sociales). A.R. thanks the US Army for financial support (contract W911NF2010242). The funding agencies did not influence the content of the manuscript. Author contributions: S.B. and S.P. designed the experiments. S.B. ran the experiments. S.B. and S.P. analyzed the data. S.B., A.R., and S.P. interpreted the results. S.B. and S.P. wrote the manuscript. A.R. edited the manuscript. Competing interests: The authors declare that they have no financial or other competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials and are available from Github repository (https://github.com/hrl-team/range). All custom scripts have been made available from Github repository (https://github.com/hrl-team/range). Additional modified scripts can be accessed upon request.

Publisher Copyright:
Copyright © 2021 The Authors, some rights reserved.

PubMed: MeSH publication types

Journal Article
Research Support, Non-U.S. Gov't

Publisher link

10.1126/sciadv.abe0340

Find It

Find It at U of M Libraries

Cite this

@article{1c04170f03db4bc1b5ca16aba173ae01,

title = "Two sides of the same coin: Beneficial and detrimental consequences of range adaptation in human reinforcement learning",

abstract = "Evidence suggests that economic values are rescaled as a function of the range of the available options. Although locally adaptive, range adaptation has been shown to lead to suboptimal choices, particularly notable in reinforcement learning (RL) situations when options are extrapolated from their original context to a new one. Range adaptation can be seen as the result of an adaptive coding process aiming at increasing the signal-to-noise ratio. However, this hypothesis leads to a counterintuitive prediction: Decreasing task difficulty should increase range adaptation and, consequently, extrapolation errors. Here, we tested the paradoxical relation between range adaptation and performance in a large sample of participants performing variants of an RL task, where we manipulated task difficulty. Results confirmed that range adaptation induces systematic extrapolation errors and is stronger when decreasing task difficulty. Last, we propose a range-adapting model and show that it is able to parsimoniously capture all the behavioral results.",

author = "Sophie Bavard and Aldo Rustichini and Stefano Palminteri",

note = "Funding Information: S.P. is supported by an ATIP-Avenir grant (R16069JS), the Programme Emergence(s) de la Ville de Paris, the Fondation Fyssen, the Fondation Schlumberger pour l{\textquoteright}Education et la Recherche, the FrontCog grant (ANR-17-EURE-0017) and the Institut de Recherche en Sant{\'e} Publique (IRESP, grant number: 20II138-00). S.B. is supported by MILDECA (Mission Interministerielle de Lutte contre les Drogues et les Conduites Addictives) and the EHESS (Ecole des Hautes Etudes en Sciences Sociales). A.R. thanks the US Army for financial support (contract W911NF2010242). The funding agencies did not influence the content of the manuscript. Author contributions: S.B. and S.P. designed the experiments. S.B. ran the experiments. S.B. and S.P. analyzed the data. S.B., A.R., and S.P. interpreted the results. S.B. and S.P. wrote the manuscript. A.R. edited the manuscript. Competing interests: The authors declare that they have no financial or other competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials and are available from Github repository (https://github.com/hrl-team/range). All custom scripts have been made available from Github repository (https://github.com/hrl-team/range). Additional modified scripts can be accessed upon request. Publisher Copyright: Copyright {\textcopyright} 2021 The Authors, some rights reserved.",

year = "2021",

month = apr,

doi = "10.1126/sciadv.abe0340",

language = "English (US)",

volume = "7",

journal = "Science advances",

issn = "2375-2548",

publisher = "American Association for the Advancement of Science",

number = "14",

}

TY - JOUR

T1 - Two sides of the same coin

T2 - Beneficial and detrimental consequences of range adaptation in human reinforcement learning

AU - Bavard, Sophie

AU - Rustichini, Aldo

AU - Palminteri, Stefano

N1 - Funding Information: S.P. is supported by an ATIP-Avenir grant (R16069JS), the Programme Emergence(s) de la Ville de Paris, the Fondation Fyssen, the Fondation Schlumberger pour l’Education et la Recherche, the FrontCog grant (ANR-17-EURE-0017) and the Institut de Recherche en Santé Publique (IRESP, grant number: 20II138-00). S.B. is supported by MILDECA (Mission Interministerielle de Lutte contre les Drogues et les Conduites Addictives) and the EHESS (Ecole des Hautes Etudes en Sciences Sociales). A.R. thanks the US Army for financial support (contract W911NF2010242). The funding agencies did not influence the content of the manuscript. Author contributions: S.B. and S.P. designed the experiments. S.B. ran the experiments. S.B. and S.P. analyzed the data. S.B., A.R., and S.P. interpreted the results. S.B. and S.P. wrote the manuscript. A.R. edited the manuscript. Competing interests: The authors declare that they have no financial or other competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials and are available from Github repository (https://github.com/hrl-team/range). All custom scripts have been made available from Github repository (https://github.com/hrl-team/range). Additional modified scripts can be accessed upon request. Publisher Copyright: Copyright © 2021 The Authors, some rights reserved.

PY - 2021/4

Y1 - 2021/4

N2 - Evidence suggests that economic values are rescaled as a function of the range of the available options. Although locally adaptive, range adaptation has been shown to lead to suboptimal choices, particularly notable in reinforcement learning (RL) situations when options are extrapolated from their original context to a new one. Range adaptation can be seen as the result of an adaptive coding process aiming at increasing the signal-to-noise ratio. However, this hypothesis leads to a counterintuitive prediction: Decreasing task difficulty should increase range adaptation and, consequently, extrapolation errors. Here, we tested the paradoxical relation between range adaptation and performance in a large sample of participants performing variants of an RL task, where we manipulated task difficulty. Results confirmed that range adaptation induces systematic extrapolation errors and is stronger when decreasing task difficulty. Last, we propose a range-adapting model and show that it is able to parsimoniously capture all the behavioral results.

AB - Evidence suggests that economic values are rescaled as a function of the range of the available options. Although locally adaptive, range adaptation has been shown to lead to suboptimal choices, particularly notable in reinforcement learning (RL) situations when options are extrapolated from their original context to a new one. Range adaptation can be seen as the result of an adaptive coding process aiming at increasing the signal-to-noise ratio. However, this hypothesis leads to a counterintuitive prediction: Decreasing task difficulty should increase range adaptation and, consequently, extrapolation errors. Here, we tested the paradoxical relation between range adaptation and performance in a large sample of participants performing variants of an RL task, where we manipulated task difficulty. Results confirmed that range adaptation induces systematic extrapolation errors and is stronger when decreasing task difficulty. Last, we propose a range-adapting model and show that it is able to parsimoniously capture all the behavioral results.

UR - http://www.scopus.com/inward/record.url?scp=85103754658&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85103754658&partnerID=8YFLogxK

U2 - 10.1126/sciadv.abe0340

DO - 10.1126/sciadv.abe0340

M3 - Article

C2 - 33811071

AN - SCOPUS:85103754658

VL - 7

JO - Science advances

JF - Science advances

SN - 2375-2548

IS - 14

M1 - eabe0340

ER -

Two sides of the same coin: Beneficial and detrimental consequences of range adaptation in human reinforcement learning

Abstract

Bibliographical note

PubMed: MeSH publication types

Publisher link

Find It

Other files and links

Fingerprint

Cite this