Intermediate wheatgrass [IWG; Thinopyrum intermedium (Host) Barkworth & D.R. Dewey subsp. intermedium] is being developed as a new perennial grain crop that has a large allohexaploid genome similar to that of wheat (Triticum aestivum L.). Breeding for increased seed weight is one of the primary goals for improving grain yield of IWG. As a new crop, however, the genetic architecture of seed weight and size has not been characterized, and selective breeding of IWG may be more intricate than wheat because of its self-incompatible mating system and perennial growth habit. Here, seed weight, seed area size, seed width, and seed length were evaluated across multiple years, in a heterogeneous breeding population comprised of 1126 genets and two clonally replicated biparental populations comprised of 172 and 265 genets. Among 10,171 DNA markers discovered using genotyping-by-sequencing (GBS) in the breeding population, 4731 markers were present in a consensus genetic map previously constructed using seven full-sib populations. Thirty-three quantitative trait loci (QTL) associated with seed weight and size were identified using association mapping (AM), of which 23 were verified using linkage mapping in the biparental populations. About 37.6% of seed weight variation in the breeding population was explained by 15 QTL, 12 of which also contributed to either seed length or seed width. When performing either phenotypic selection or genomic selection for seed weight, we observed the frequency of favorable QTL alleles were increased to >46%. Thus, by combining AM and genomic selection, we can effectively select the favorable QTL alleles for seed weight and size in IWG breeding populations.