Managing survey change: resource tradeoffs and quality metrics

The School of Arts & Social Sciences welcomes Brad Edwards to City, University of London.

Abstract

Successful long-running surveys may repeat year after year, asking the same questions in the same way and creating useful trend lines. Change comes slowly but inevitably if these surveys hope to remain vital and relevant. The Medical Expenditure Panel Survey in the US offers a case study in managing such change, transforming to commercial off-the-shelf data collection software (Blaise) while preserving the structure, format, and delivery schedule for all public use files.

Overlap in design, programming, and testing phases must be carefully managed. Moreover, balancing resources between the new development in parallel with the ongoing demands of field and processing operations in the old system, with no accommodations in the schedule or loss of timeliness for data users, is a major management challenge. After launching the new software in the field, numerous quality metrics (derived from both data and paradata) are monitored to distinguish changes in the trend lines from changes (errors or error reduction) that may result from the transformation itself.

About the speaker

Brad Edwards is a Westat vice president with more than 30 years of experience designing and managing large, complex surveys. He is Westat’s project director for the Medical Expenditure Panel Survey’s household component. Research interests include multicultural issues, panel survey design, mobile devices, performance dashboards, and data collector training and management. He co-chaired the 2015 conference “Total Survey Error: Improving Quality in an Era of Big Data,” and co-edited Total Survey Error in Practice (2017). He was also a co-editor of Survey Methods in Multinational, Multiregional, and Multicultural Contexts (2010) and Hard-to-Survey Populations (2015), and has authored or co-authored many book chapters, journal articles, short courses, and presentations at AAPOR, ESRA, and other conferences.