We present an application of machine learning to the semi-automatic synthesis of robust servo-trackers for underwater robotics. In particular, we investigate an approach based on the use of Boosting for robust visual tracking of color objects in an underwater environment. To this end, we use AdaBoost, the most common variant of the Boosting algorithm, to select a number of low-complexity but moderately accurate color feature trackers and we combine their outputs. The novelty of our approach lies in the design of this family of weak trackers, which enhances a straightforward color segmentation tracker in multiple ways. From a large and diverse family of possible filters, we select a small subset that optimizes the performance of our trackers. The tracking process applies these trackers on the input video frames, and the final tracker output is chosen based on the weights of the final array of trackers. By using computationally inexpensive, but somewhat accurate trackers as members of the ensemble, the sy tem is able to run at quasi real-time, and thus, is deployable on-board our underwater robot. We present quantitative cross-validation results of our spatio-chromatic visual tracker, and conclude by pointing out some difficulties faced and subsequent shortcomings in the experiments we performed, along with directions of future research in the area of ensemble trackingin real-time.