Background Smokers have lower risk of obesity, which some consider a “beneficial” side effect of smoking. However, some studies suggest that smoking is simultaneously associated with higher central adiposity and, more specifically, ectopic adipose deposition. Little is known about the association of smoking with intermuscular adipose tissue (IMAT), an ectopic adipose depot associated with cardiovascular disease (CVD) risk and a key determinant of muscle quality and function. We tested the hypothesis that smokers have higher abdominal IMAT and lower lean muscle quality than never smokers. Methods and findings We measured abdominal muscle total, lean, and adipose volumes (in cubic centimeters) and attenuation (in Hounsfield units [HU]) along with subcutaneous (SAT) and visceral adipose tissue (VAT) volumes using computed tomography (CT) in 3,020 middle-aged Coronary Artery Risk Development in Young Adults (CARDIA) participants (age 42–58, 56.3% women, 52.6% white race) at the year 25 (Y25) visit. The longitudinal CARDIA study was initiated in 1985 with the recruitment of young adult participants (aged 18–30 years) equally balanced by female and male sex and black and white race at 4 field centers located in Birmingham, AL, Chicago, IL, Minneapolis, MN, and Oakland, CA. Multivariable linear models included potential confounders such as physical activity and dietary habits along with traditional CVD risk factors. Current smokers had lower BMI than never smokers. Nevertheless, in the fully adjusted multivariable model with potential confounders, including BMI and CVD risk factors, adjusted mean (95% CI) IMAT volume was 2.66 (2.55–2.76) cm3 in current smokers (n = 524), 2.36 (2.29–2.43) cm3 in former smokers (n = 944), and 2.23 (2.18–2.29) cm3 in never smokers (n = 1,552) (p = 0.007 for comparison of former versus never smoker, and p < 0.001 for comparison of current smoker versus never and former smoker). Moreover, compared to participants who never smoked throughout life (41.6 [41.3–41.9] HU), current smokers (40.4 [39.9–40.9] HU) and former smokers (40.8 [40.5–41.2] HU) had lower lean muscle attenuation suggesting lower muscle quality in the fully adjusted model (p < 0.001 for comparison of never smokers with either of the other two strata). Among participants who had ever smoked, pack-years of smoking exposure were directly associated with IMAT volume (β [95% CI]: 0.017 [0.010–0.025]) (p < 0.001). Despite having less SAT, current smokers also had higher VAT/SAT ratio than never smokers. These findings must be viewed with caution as residual confounding and/or reverse causation may contribute to these associations. Conclusions We found that, compared to those who never smoked, current and former smokers had abdominal muscle composition that was higher in adipose tissue volume, a finding consistent with higher CVD risk and age-related physical deconditioning. These findings challenge the belief that smoking-associated weight loss or maintenance confers a health benefit.