Online Learning for Multimodal Data Fusion with Application to Object Recognition

Shahin Shahrampour, Mohammad Noshad, Jie Ding, Vahid Tarokh

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


We consider online multimodal data fusion, where the goal is to combine information from multiple modes to identify an element in a large dictionary. We address this problem in the context of object recognition by focusing on tactile sensing as one of the modes. Using a tactile glove with seven sensors, various individuals grasp different objects to obtain 7-D time series, where each component represents the pressure sequence applied to one sensor. The pressure data of all objects is stored in a dictionary as a reference. The objective is to match a streaming vector time series from grasping an unknown object to a dictionary object. We propose an algorithm that may start with prior knowledge provided by other modes. Receiving pressure data sequentially, the algorithm uses a dissimilarity metric to modify the prior and form a probability distribution over the dictionary. When the dictionary objects are dissimilar in shape, we empirically show that our algorithm recognize the unknown object even with a uniform prior. If there exists a similar object to the unknown object in the dictionary, our algorithm needs the prior from other modes to detect the unknown object. Notably, our algorithm maintains a similar performance to standard offline classification techniques, such as support vector machine, with a significantly lower computational time.

Original languageEnglish (US)
Article number8039518
Pages (from-to)1259-1263
Number of pages5
JournalIEEE Transactions on Circuits and Systems II: Express Briefs
Issue number9
StatePublished - Sep 2018
Externally publishedYes


  • Online learning
  • mirror descent
  • object recognition
  • tactile sensing

Fingerprint Dive into the research topics of 'Online Learning for Multimodal Data Fusion with Application to Object Recognition'. Together they form a unique fingerprint.

Cite this