Overall, our investigation suggests that while AI textbots show promise in enhancing data quality, their effect on user experience, particularly for mobile users, warrants careful consideration and further optimization. We urge survey researchers to consider whether the gains in data quality are substantial enough to offset the risks and additional complexities incurred by generative AI.

Beyond these findings on specific outcomes, our evaluation yielded practical insights for integrating LLM textbots into surveys. Among them:

• Consider the user experience design (UXD) of the textbot integration, including the amount and placement of probing. Our results suggest that excessive probing early in the survey can increase respondent dropout. Therefore, our preliminary recommendation is to place question probes in the middle or end sections of a survey. Randomizing the placement of questions and the application of probing may help mitigate negative effects and estimate and control for the effects of the marginal probe on outcomes of interest to the researcher.
• Conducting a pilot study can help fine-tune the textbot LLM to each question by identifying cases where probing failed or should have been conducted. Researchers can then refine LLM prompts for individual questions as well as use exemplar responses to probes in the pilot sample to calibrate the LLM, either through few-shot learning where examples are inserted into future prompts or in supervised learning where a systematic dataset is used to re-train the model.
• Different types of probing strategies should be considered, depending on the specific goals of the study. For example, some research may benefit from probing for depth (asking respondents to explain why or provide details for existing examples) while other studies might optimize for depth (eliciting secondary response categories besides the “main” category cited by a respondent). Probing may also help with specificity to ensure information is collected in the categories of interest or with the desired level of granularity. Exploring sequences of prompts may also enhance the quality of responses.

The results of this study open several new avenues for future research. Beyond the outcomes we measured, impacts on response latency may provide further insights into how conversational AI may impact cognitive burden (e.g., increasing the time to respond) and survey experience (e.g., delays in the survey flow due to activation of the LLM). More rigorous tuning of the textbot LLM for each survey question could further enhance the outcomes observed in this evaluation.

While our study establishes a baseline performance based on “out-of-the-box” textbot capabilities, future studies could aim to determine the ceiling of AI capabilities through advanced techniques like prompt engineering and different training methods. Researchers must also better understand the costs of using textbots, given early signs of negative impacts on user experience, and find ways to mitigate them where possible. Lastly, beyond the questions explored in this study, large language models may be useful for eliciting more thoughtful answers on topics like income, race, and sexual orientation and gender identity (SOGI) through more open-ended, conversational approaches.