Ensembles of Informative Representations for Self-Supervised Learning

The requirement of large-size labeled training datasets often prohibits the deployment of supervised learning models in several applications with high acquisition costs and privacy concerns. To alleviate the burden of obtaining labels, self-supervised learning aims to identify informative data representations using auxiliary tasks that do not require external labels. The representations serve as refined inputs for the main learning task aimed at improving sample efficiency. Nonetheless, selecting individual auxiliary tasks and combining the corresponding extracted representations constitutes a nontrivial design problem. Agnostic of the approach for extracting individual representations per auxiliary task, this paper develops a weighted ensemble approach for obtaining a unified representation. The weights signify the relative dominance of individual representations in informing predictions for the main task. The representation ensemble is further augmented with the input data to improve accuracy and avoid information loss concerns. Numerical tests on real datasets showcase the merits of the advocated approach.