Introduction The relationship between low income and worse health outcomes is evident, yet its association with cognitive outcomes is less explored. Most studies have measured income at one time and none have examined how sustained exposure to low income influences cognition in a relatively young cohort. This study examined the effect of sustained poverty and perceived financial difficulty on cognitive function in midlife. Methods Income data were collected six times between 1985 and 2010 for 3,383 adults from the Coronary Artery Risk Development in Young Adults prospective cohort study. Sustained poverty was defined by the percentage of time participants’ household income was <200% of the federal poverty level—“never” in poverty, “0< to <1/3,” “≥1/3 to <100%” or “all-time.” In 2010, at a mean age of 50 years, participants underwent a cognitive battery. Data were analyzed in 2015. Results In demographic-adjusted linear regression models, individuals with all-time poverty performed significantly worse than individuals never in poverty: 0.92 points worse on verbal memory (z-score, −0.28; 95% CI=−0.43, −0.13), 11.60 points worse on processing speed (z-score, −0.72; 95% CI=−0.85, −0.58), and 3.50 points worse on executive function (z-score, −0.32; 95% CI=−0.47, −0.17). Similar results were observed with perceived financial difficulty. Findings were robust when restricted to highly educated participants, suggesting little evidence for reverse causation. Conclusions Cumulative exposure to low income over 2 decades was strongly associated with worse cognitive function of a relatively young cohort. Poverty and perceived hardship may be important contributors to premature aging among disadvantaged populations.