American Time Use Survey Testing Mode Effects between Web & Telephone

From above crop female freelancer typing on laptop keyboard while sitting at table and working on remote project at home

Problem

The Bureau of Labor Statistics is developing a web mode for the American Time Use Survey and needs to know how web-collected data differs from data collected by phone interview.

The Bureau of Labor Statistics (BLS) conducts the American Time Use Survey (ATUS) to collect nationally representative data on how, where, and with whom Americans ages 15 and older spend their time. ATUS respondents complete a time diary, reporting their activities for a 24-hour period from 4am the previous day to 4am of the current day.

Since its inception in 2003, the ATUS has been an interviewer-administered CATI phone survey conducted with a live interviewer. To modernize the ATUS and offer an alternative mode of data collection, BLS has been developing a self-administered web diary tool so that the survey can become multi-mode. BLS is concerned about mode differences between data collected via phone versus web and how accurately and completely web respondents can report their activities without the aid of an interviewer.

Solution

NORC is testing the ATUS web diary prototype to identify mode effects and reporting challenges.

For several years, NORC has assisted BLS in refining and testing the ATUS web diary. In 2022, NORC used a within-subjects design to collect web and phone diaries for 45 research participants and then compared the diaries for each person to identify differences in reporting between modes. The web diaries were completed by the participant while the interviewer observed. The results showed that while respondents were able to successfully report time and activity data via the web diary, there were clear mode effects between web and phone.

Following the 2022 web diary test, BLS revised the prototype and, in 2023, contracted with NORC to conduct a pilot test of the updated web diary tool. Using the AmeriSpeak panel, NORC will collect 800 self-administered web diaries and conduct 60 debriefing interviews with pilot test participants. The goal of the pilot test is to understand how participants report activities using the web diary without the aid of an interviewer. NORC will analyze the pilot test data to provide insight into how participants interpret the diary reporting task and where underreporting/misreporting may be occurring.

Result

This project is currently in the process of collecting and analyzing data.

Results will be summarized in a final report and presentation to BLS. NORC will summarize activity patterns in the pilot test data, conduct comparisons to the 2021-2022 ATUS data collected by phone, and identify potential mode effects, respondent burden and usability challenges. Debriefing interview data will provide additional insight into how participants determined which activities to report and the quality and accuracy of the data they reported. Findings from this project will inform future ATUS web diary development work.