This paper proposes a vision based alternative to the passive arm pose estimation for underwater remotely operated vehicle (ROV) performing manipulation tasks. The proposed approach attaches a fixed landmark on an underwater fixture and uses the camera images of the landmark object points to infer the pose of the ROV. A gradient descent least squares algorithm on the SO(3) manifold is proposed for accurately and efficiently estimating the pose. The algorithm has been implemented on a low-cost single board computer. Numerical comparison with other existing algorithms as well as in-air and underwater experiments show the efficacy of the algorithm. Positional accuracy of the order of 1-2.5mm while the landmark is approximately 1m away has been demonstrated.