The gearbox is one of the most common and important components in the drivetrains. Thus, the online monitoring of the dynamic behavior of geared system is crucial for the optimization, diagnosis and prognosis of the drivetrains. The conventional online monitoring system for gearboxes is to use the vibration sensor mounted on the gear housing. However, in the measured housing vibration signal, the dynamic response of the monitored geared pair is usually distorted, which is caused by the complex transfer path of the vibration. Therefore, to advance the art of online monitoring of gearboxes, this work proposes to employ the fiber Bragg grating as the strain sensor to mount near the gear mesh region. The experimental assessment of the feasibility of the fiber Bragg grating based online monitoring system is conducted in a laboratory fixed-axis spur gearbox. To validate and analyze the measurement from the fiber Bragg grating system, a gear mesh model is developed using the finite element method. The comparison between the measurement and theoretical simulation show the proposed fiber Bragg grating based online monitoring system is capable to capture the variation of the root strain during the gear mesh process. This result proves the proposed technique has a promising potential in developing a commercial online monitoring system to measure the subtle dynamic behavior of gearboxes.