The goal of this project is to develop a passive vision-based sensing system capable of monitoring an intersection by observing the vehicle and pedestrian flow, and predicting situations that might give rise to accidents. A single camera mounted at an arbitrary position looking at an intersection is used. However, for extended applications multiple cameras will be needed. Some of the key elements are camera calibration, motion tracking, vehicle classification, and predicting collisions. In this paper, we focus on motion tracking. Motion segmentation is performed using an adaptive background model that models each pixel as a mixture of Gaussians. The method used is similar to the Stauffer method for motion segmentation. Tracking of objects is performed by computing the overlap between oriented bounding boxes. The oriented boxes are computed by vector quantization of blobs in the scene. The principal angles computed during vector quantization along with other cues of the object are used for classification of detected entities into vehicles and pedestrians.