The magnetoresistance of ultrathin insulating films of Bi has been studied with magnetic fields applied parallel and perpendicular to the plane of the sample. Deep in the strongly localized regime, the magnetoresistance is negative and independent of field orientation. As film thicknesses increase, the magnetoresistance becomes positive, and a difference between values measured in perpendicular and parallel fields appears, which is a linear function of the magnetic field and is positive. This is not consistent with the quantum interference picture. We suggest that it is due to vortices present on the insulating side of the superconductor-insulator transition.