Online Learning for Multimodal Data Fusion with Application to Object Recognition

Shahin Shahrampour, Mohammad Noshad, Jie Ding, Vahid Tarokh

Research output: Contribution to journalArticlepeer-review

10 Scopus citations


We consider online multimodal data fusion, where the goal is to combine information from multiple modes to identify an element in a large dictionary. We address this problem in the context of object recognition by focusing on tactile sensing as one of the modes. Using a tactile glove with seven sensors, various individuals grasp different objects to obtain 7-D time series, where each component represents the pressure sequence applied to one sensor. The pressure data of all objects is stored in a dictionary as a reference. The objective is to match a streaming vector time series from grasping an unknown object to a dictionary object. We propose an algorithm that may start with prior knowledge provided by other modes. Receiving pressure data sequentially, the algorithm uses a dissimilarity metric to modify the prior and form a probability distribution over the dictionary. When the dictionary objects are dissimilar in shape, we empirically show that our algorithm recognize the unknown object even with a uniform prior. If there exists a similar object to the unknown object in the dictionary, our algorithm needs the prior from other modes to detect the unknown object. Notably, our algorithm maintains a similar performance to standard offline classification techniques, such as support vector machine, with a significantly lower computational time.

Original languageEnglish (US)
Article number8039518
Pages (from-to)1259-1263
Number of pages5
JournalIEEE Transactions on Circuits and Systems II: Express Briefs
Issue number9
StatePublished - Sep 2018

Bibliographical note

Funding Information:
Manuscript received August 4, 2017; accepted September 13, 2017. Date of publication September 18, 2017; date of current version August 28, 2018. This work was supported by DARPA under Grant N66001-15-C-4028 and Grant W911NF-14-1-0508. This brief was recommended by Associate Editor L.-P. Chau. (Corresponding author: Shahin Shahrampour.) S. Shahrampour, J. Ding, and V. Tarokh are with the John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138 USA (e-mail:;;

Publisher Copyright:
© 2004-2012 IEEE.


  • Online learning
  • mirror descent
  • object recognition
  • tactile sensing


Dive into the research topics of 'Online Learning for Multimodal Data Fusion with Application to Object Recognition'. Together they form a unique fingerprint.

Cite this