Dec 12 -- The National Center for Health Statistics (NCHS) invites comments to OMB by January 20, 2023 regarding the proposed Collaborating Center for Questionnaire Design and Evaluation. [Comments due 30 days after NCHS submission to OMB on December 21, 2022.]
Section 306 of the Public Health Service (PHS) Act (42 U.S.C. 242k), as amended, authorizes that the Secretary of Health and Human Services (DHHS), acting through NCHS, shall undertake and support (by grant or contract) research, demonstrations, and evaluations respecting new or improved methods for obtaining current data to support statistical and epidemiological activities for the purpose of improving the effectiveness, efficiency, and quality of health services in the United States.
The Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) is the focal point within NCHS for questionnaire and survey development, pre-testing, and evaluation activities for CDC surveys such as the National Survey of Family Growth (NSFG), the Research and Development Survey (RANDS) (including RANDS COVID), and other federally sponsored surveys. The CCQDER is requesting three years of OMB Clearance for this Generic submission.
The CCQDER and other NCHS programs conduct cognitive interviews, focus groups, in-depth or ethnographic interviews, usability tests, field tests/pilot interviews, and experimental research in laboratory and field settings, both for applied questionnaire development and evaluation as well as more basic research on measurement errors and survey response. Various techniques to evaluate interviewer administered, self-administered, telephone, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self-Interviewing (CASI), Audio Computer-Assisted Self-Interviewing (ACASI), and web-based questionnaires are used.
The most common questionnaire evaluation method is the cognitive interview. These evaluations are conducted by the CCQDER. The interview structure consists of respondents first answering a draft survey question and then providing textual information to reveal the processes involved in answering the test question. Specifically, cognitive interview respondents are asked to describe how and why they answered the question as they did. Through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered. By conducting a comparative analysis of cognitive interviews, it is also possible to determine whether particular interpretive patterns occur within particular sub-groups of the population. Interviews are generally conducted in small rounds totaling 40-100 interviews; ideally, the questionnaire is re-worked between rounds, and revisions are tested iteratively until interviews yield relatively few new insights.
Cognitive interviewing is inexpensive and provides useful data on questionnaire performance while minimizing respondent burden. Cognitive interviewing offers a detailed depiction of meanings and processes used by respondents to answer questions—processes that ultimately produce the survey data. As such, the method offers an insight that can transform understanding of question validity and response error. Documented findings from these studies represent tangible evidence of how the question performs. Such documentation also serves CDC data users, allowing them to be critical users in their approach and application of the data.
In addition to cognitive interviewing, a number of other qualitative and quantitative methods are used to investigate and research measurement errors and the survey response process. These methods include conducting focus groups, usability tests, in-depth or ethnographic interviews, and the administration and analysis of questions in both representative and non-representative field tests. Focus groups are conducted by the CCQDER. They are group interviews whose primary purpose is to elicit the basic socio-cultural understandings and terminology that form the basis of questionnaire design. Each group typically consists of one moderator and four to 10 participants, depending on the research question. In-depth or ethnographic interviews are one-on-one interviews designed to elicit the understandings or terminology that are necessary for question design, as well as to gather detailed information that can contribute to the analysis of both qualitative and quantitative data. Usability tests are typically one-on-one interviews that are used to determine how a given survey or information collection tool functions in the field, and how the mode and layout of the instrument itself may contribute to survey response error and the survey response process.
In addition to these qualitative methods, NCHS also uses various tools to obtain quantitative data, which can be analyzed alone or analyzed alongside qualitative data to give a much fuller accounting of the survey response process. For instance, phone, internet, mail, and in-person follow-up interviews of previous NCHS survey respondents may be used to test the validity of survey questions and questionnaires and to obtain more detailed information that cannot be gathered on the original survey. Additionally, field or pilot tests may be conducted on both representative and non-representative samples, including those obtained from commercial survey and web panel vendors. Beyond looking at traditional measures of survey errors (such as item missing rates and non-response, and don't know rates), these pilot tests can be used to run experimental designs in order to capture how different questions function in a field setting. Similar methodology has been adopted by other federal agencies, as well as by academic and commercial survey organizations.
In 2022-2025 NCHS/CCQDER staff plans to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, NCHS/CCQDER will work collaboratively with survey researchers from universities and other federal agencies to define and examine several research areas, including, but not limited to: (1) differences between face-to-face, telephone, and virtual/video-over internet cognitive interviewing; (2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing; (3) reactions of both survey respondents and survey interviewers to the use of Computer Assisted Personal Interviewing (CAPI), Audio Computer-Assisted Self-Interview (ACASI), video-over internet/virtual; (4) social, cultural and linguistic factors in the question response process; and (5) recruitment and respondent participation at varying levels of incentive in an effort to establish empirical evidence regarding remuneration and coercion. Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, questionnaires that are of current interest (such as RANDS and NIOSH) may be evaluated using several of the techniques described above, or different versions of a survey question will be developed, and the variants then administered to separate groups of respondents in order to study the cognitive processes that account for the differences in responses obtained across different versions.
These studies will be conducted either by CCQDER staff, DHHS staff, or NCHS contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.
CCQDER:
https://www.cdc.gov/nchs/ccqder/Index.html
NCHS submission to OMB:
https://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=202212-0920-003 Click View Supporting Statement for technical documentation. Submit comments through this site.
FR notice inviting public comment:
https://www.federalregister.gov/d/2022-26887
For AEA members wishing to submit comments, "A Primer on How to Respond to Calls for Comment on Federal Data Collections" is available at
https://www.aeaweb.org/content/file?id=5806