While the associations of common variants in the FTO gene with obesity have been widely replicated in adults, there is conflicting evidence regarding their effects in infancy. We hypothesize that the genetic influences on growth traits vary during infancy, and that conflicting results may stem from variation in the ages at which FTO associations have been examined. Using longitudinal weight and length data at 0, 1, 3, 6, 9, 12, 18, 24, 30, and 36 months in 917 (444 females, 473 males) family members from the Fels Longitudinal Study, we used a variance components-based approach (SOLAR) to: (i) examine differences in heritability (gene-by-age interaction) in weight, length, relative weight (BMI and ponderal index (PI)) and instantaneous weight and length velocities over the course of infancy, and (ii) test whether a common FTO variant (rs9939609) was associated with infant growth at three ages (maximum trait heritability, birth and 36 months). All heritabilities at birth (of 39-74%) were significant (P 3.9 × 1010), but changed with age (gene-by-age interaction, P<0.05). Weight, relative weight, and weight velocity reached maximum heritabilities (of 76-89%) at 6-9 months, while length and length velocity reached maximum heritabilities (of 96-99%) at 18-30 months. We found no association of rs9939609 with growth status or velocity measured at any age (P<0.11). This study for the first time demonstrates the fluctuation of genetic influences on infant growth, but further work is required to determine which gene variants explain the strong additive genetic effects observed.