Transferring and generalizing deep-learning-based neural encoding models across subjects

Haiguang Wen, Junxing Shi, Wei Chen, Zhongming Liu

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a target subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the target subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while a deep residual neural network driven by image recognition was used to model visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish both subject-specific and population-wide predictive models of cortical representations of high-dimensional and hierarchical visual features.

Original languageEnglish (US)
Pages (from-to)152-163
Number of pages12
JournalNeuroImage
Volume176
DOIs
StatePublished - Aug 1 2018

Fingerprint

Learning
Population
Brain Mapping
Magnetic Resonance Imaging
Brain
Recognition (Psychology)
Transfer (Psychology)

Keywords

  • Bayesian inference
  • Deep learning
  • Incremental learning
  • Natural vision
  • Neural encoding

Cite this

Transferring and generalizing deep-learning-based neural encoding models across subjects. / Wen, Haiguang; Shi, Junxing; Chen, Wei; Liu, Zhongming.

In: NeuroImage, Vol. 176, 01.08.2018, p. 152-163.

Research output: Contribution to journalArticle

Wen, Haiguang ; Shi, Junxing ; Chen, Wei ; Liu, Zhongming. / Transferring and generalizing deep-learning-based neural encoding models across subjects. In: NeuroImage. 2018 ; Vol. 176. pp. 152-163.
@article{d9c2e97bc7e247bb8e82b30637e0d337,
title = "Transferring and generalizing deep-learning-based neural encoding models across subjects",
abstract = "Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a target subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the target subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while a deep residual neural network driven by image recognition was used to model visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish both subject-specific and population-wide predictive models of cortical representations of high-dimensional and hierarchical visual features.",
keywords = "Bayesian inference, Deep learning, Incremental learning, Natural vision, Neural encoding",
author = "Haiguang Wen and Junxing Shi and Wei Chen and Zhongming Liu",
year = "2018",
month = "8",
day = "1",
doi = "10.1016/j.neuroimage.2018.04.053",
language = "English (US)",
volume = "176",
pages = "152--163",
journal = "NeuroImage",
issn = "1053-8119",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Transferring and generalizing deep-learning-based neural encoding models across subjects

AU - Wen, Haiguang

AU - Shi, Junxing

AU - Chen, Wei

AU - Liu, Zhongming

PY - 2018/8/1

Y1 - 2018/8/1

N2 - Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a target subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the target subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while a deep residual neural network driven by image recognition was used to model visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish both subject-specific and population-wide predictive models of cortical representations of high-dimensional and hierarchical visual features.

AB - Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a target subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the target subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while a deep residual neural network driven by image recognition was used to model visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish both subject-specific and population-wide predictive models of cortical representations of high-dimensional and hierarchical visual features.

KW - Bayesian inference

KW - Deep learning

KW - Incremental learning

KW - Natural vision

KW - Neural encoding

UR - http://www.scopus.com/inward/record.url?scp=85046431235&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85046431235&partnerID=8YFLogxK

U2 - 10.1016/j.neuroimage.2018.04.053

DO - 10.1016/j.neuroimage.2018.04.053

M3 - Article

C2 - 29705690

AN - SCOPUS:85046431235

VL - 176

SP - 152

EP - 163

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

ER -