Impact of different cover letter content and incentives on non-response bias in a sample of Veterans applying for Department of Veterans Affairs disability benefits: a randomized, 3X2X2 factorial trial

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Background: Non-random non-response bias in surveys requires time-consuming, complicated, post-survey analyses. Our goal was to see if modifying cover letter information would prevent non-random non-response bias altogether. Our secondary goal tested whether larger incentives would reduce non-response bias. Methods: A mailed, survey of 480 male and 480 female, nationally representative, Operations Enduring Freedom, Iraqi Freedom, or New Dawn (OEF/OIF/OND) Veterans applying for Department of Veterans Affairs (VA) disability benefits for posttraumatic stress disorder (PTSD). Cover letters conveyed different information about the survey’s topics (combat, unwanted sexual attention, or lifetime and military experiences), how Veterans’ names had been selected (list of OEF/OIF/OND Veterans or list of Veterans applying for disability benefits), and what incentive Veterans would receive ($20 or $40). The main outcome, non-response bias, measured differences between survey respondents’ and sampling frame’s characteristics on 8 administrative variables, including Veterans’ receipt of VA disability benefits and exposure to combat or military sexual trauma. Analysis was intention to treat. We used ANOVA for factorial block-design, logistic, mixed-models to assess bias and multiple imputation and expectation-maximization algorithms to assess potential missing mechanisms (missing completely at random, missing at random, or not random) of two self-reported variables: combat and military sexual assault. Results: Regardless of intervention, men with any VA disability benefits, women with PTSD disability benefits, and women with combat exposure were over-represented among respondents. Interventions explained 0.0 to 31.2% of men’s variance and 0.6 to 30.5% of women’s variance in combat non-response bias and 10.2 to 43.0% of men’s variance and 0.4 to 31.9% of women’s variance in military sexual trauma non-response bias. Non-random assumptions showed that men’s self-reported combat exposure was overestimated by 19.0 to 28.8 percentage points and their self-reported military sexual assault exposure was underestimated by 14.2 to 28.4 percentage points compared to random missingness assumptions. Women’s self-reported combat exposure was overestimated by 8.6 to 10.6 percentage points and military sexual assault exposure, by 1.2 to 6.9 percentage points. Conclusions: Our interventions reduced bias in some characteristics, leaving others unaffected or exacerbated. Regardless of topic, researchers are urged to present estimates that include all three assumptions of missingness.

Original languageEnglish (US)
Article number61
JournalBMC Medical Research Methodology
Issue number1
StatePublished - Dec 2022

Bibliographical note

Funding Information:
We thank Andrea Cutting for data management and Derek Vang for study coordination. Views expressed are solely those of the authors and do not reflect the opinion, views, policies, or position of the Department of Veterans Affairs.

Funding Information:
The Center for Care Delivery Outcomes Research is a VA Health Services Research and Development (HSR&D) Service Center of Innovation (Center grant #HFP 98–001). This work was supported by the VA HSR&D Service (grant number IIR-14-004). The funder had no role in data analysis, manuscript preparation, or decision to publish.

Publisher Copyright:
© 2022, The Author(s).


  • Combat
  • Factorial design
  • Leverage salience theory
  • Mailed survey
  • Military sexual trauma
  • Non-response Bias
  • Randomized trial
  • Sexual assault

PubMed: MeSH publication types

  • Journal Article
  • Randomized Controlled Trial
  • Research Support, U.S. Gov't, Non-P.H.S.


Dive into the research topics of 'Impact of different cover letter content and incentives on non-response bias in a sample of Veterans applying for Department of Veterans Affairs disability benefits: a randomized, 3X2X2 factorial trial'. Together they form a unique fingerprint.

Cite this