Testing online surveys in Monitoring the Future

Changing the protocol from mail to web in a long-standing, longitudinal study is a big deal. Will data quality be the same? Will response rates differ? Will these differ by race, gender, or other factors? Monitoring the Future (MTF) is exploring web-based surveys as they have become more standard generally. The current experiment by Megan Patrick, Mick Couper, and YSI researchers tests three new protocols, device preferences, data quality, and demographic differences.


Since the mid-70s MTF has used paper questionnaires, administered to adolescents in school and mailed to adults for follow-up, to examine substance use and related factors in the US.


The current study compared three new conditions to standard mailed paper surveys (MTF Control) in the first wave of follow-ups after high school, at ages 19 and 20. In the Mail Push condition, participants received selection and reminder letters and calls that mirrored MTF Control but also provided web login information. In the Web Push condition, participants received letters that referred only to the online survey; a later reminder letter included a paper questionnaire plus online login information. The Web Push + Email condition mirrored Condition 2 and also emailed selection, reminder, and login information (email addresses gathered at 12th grade baseline, usable for 64% in this study).


Overall response rates were lower in the Web Push condition, compared to MTF Control and Mail Push. Response rates in Mail Push and Web Push + Email did not differ significantly from MTF Control. Participants in Web Push + Email were most likely to respond online and via smartphone. Hispanics were more likely than whites to respond online, as were those with higher parent education and those currently in college. Blacks were more likely than whites to respond via smartphone, and young adults not attending college were more likely than full-time four-year college students to respond via smartphone. These results suggest that demographic groups that were less likely to respond at follow-up overall across conditions had higher odds of response to the web survey (Hispanic young adults) and using smartphones (black young adults).


Item nonresponse as an indicator of data quality differed by condition and mode. The two Web Push conditions had lower missing data rates than MTF Control, and the web response overall had lower missing data than the paper response. Total percent missing data did not differ by device type (desktop, laptop, smartphone) among web responders.


There were some differences in substance use variables. Web Push + Email respondents reported higher rates of alcohol use (53%) compared to MTF Control (47%) and higher rates of cigarette use (15%) compared to Web Push (11%). We found higher rates of cigarette use among smartphone responders compared to desktop, laptop, or tablet responders. The current study gave participants the option of response mode and therefore shows selection effects.


The Web Push + Email condition was the most cost-effective. However, more work needs to be done to understand differences in respondent incentive check cashing. Email may be important for encouraging response among people who may not receive or open the paper mailing. At the same time, not cashing the incentive check could detrimentally impact future participation.


The article A sequential mixed mode experiment in the U.S. national Monitoring the Future study was published in the Journal of Survey Statistics and Methodology.