Accurate tracking is a difficult task in most computer vision applications. Errors in target localization and tracking result not only from the general uncontrolled nature of the environment, but also from inaccurate modeling of the target motion. This work presents a novel solution for the robust estimation of target trajectories obtained from real-world scenes such as traffic intersections. The main contribution of this work is a deterministic sampling approach applied to the filtering step of the switching Kalman filter/smoother. The unscented transform is used to obtain a fixed set of samples of the state distribution in the filtering step. Results demonstrating the improved accuracy and robustness of the proposed method, namely, deterministic sampling or unscented transform-based switching Kalman filter (DS-SKS or UKS) and the standard switching Kalman filter/smoother (SKS) are presented.