The circumplex structure derived from similarity ratings of affect words is assumed to be a conceptual representation of affect anchored in semantic knowledge. Recently, it been suggested that this structure is not based on semantic knowledge at all, but may instead reflect a type of episodic knowledge: The degree to which emotions covary in everyday life. In two experience-sampling studies, we compared the semantic and the episodic hypotheses by comparing participants' similarity ratings to the observed covariations in their own affective experience computed from their momentary reports. In Study 2, participants also provided estimates of the degree to which their emotions covaried. Evidence from both studies indicate that similarity judgements are related both to semantic and episodic information, indicating that a pure episodic account of similarity ratings, and the mental representation of affect that they reflect, is untenable.