Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions

London Lowmanstone, Ruyuan Wan, Risako Owan, Jaehyung Kim, Dongyeop Kang

Research output: Contribution to journalConference articlepeer-review


Annotating data via crowdsourcing is time-consuming and expensive. Due to these costs, dataset creators often have each annotator label only a small subset of the data. This leads to sparse datasets with examples that are marked by few annotators. The downside of this process is that if an annotator doesn't get to label a particular example, their perspective on it is missed. This is especially concerning for subjective NLP datasets where there is no single correct label: people may have different valid opinions. Thus, we propose using imputation methods to generate the opinions of all annotators for all examples, creating a dataset that does not leave out any annotator's view. We then train and prompt models, using data from the imputed dataset, to make predictions about the distribution of responses and individual annotations. In our analysis of the results, we found that the choice of imputation method significantly impacts soft label changes and distribution. While the imputation introduces noise in the prediction of the original dataset, it has shown potential in enhancing shots for prompts, particularly for low-response-rate annotators. We have made all of our code and data publicly available.

Original languageEnglish (US)
JournalCEUR Workshop Proceedings
StatePublished - 2023
Event2nd Workshop on Perspectivist Approaches to NLP, NLPerspectives 2023 - Krakow, Poland
Duration: Sep 30 2023 → …

Bibliographical note

Publisher Copyright:
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).


  • LeWiDi
  • NLPerspectives
  • annotation
  • content filtering
  • imputation
  • large language models
  • matrix factorization
  • natural language processing


Dive into the research topics of 'Annotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions'. Together they form a unique fingerprint.

Cite this