Dec 14 -- The Bureau of Labor Statistics, Department of Labor, invites comments by February 12, 2024 regarding the planned revision of the Cognitive and Psychological Research data collection.
The Bureau of Labor Statistics' Behavioral Science Research Center (BSRC) conducts theoretical, applied, and evaluative research aimed at improving the quality of data collected and published by the Bureau. Since its creation in 1988, the BSRC has advanced the study of survey methods research, approaching issues of non-sampling error within a framework that draws heavily on the theories and methods of the cognitive, statistical, and social sciences. The BSRC research focuses primarily on the assessment of survey instrument design and survey administration, as well as on issues related to interviewer training, the interaction between interviewer and respondent in the interview process, and the usability of data-collection instruments by both interviewers and respondents. Improvements in these areas result in greater accuracy and response rates of BLS surveys, frequently reduce costs in training and survey administration, and further ensure the effectiveness of the Bureau's overall mission.
Office of Management and Budget clearance is being sought for “Cognitive and Psychological Research.” The purpose of this request for clearance by the BSRC is to conduct cognitive and psychological research designed to enhance the quality of the Bureau's data collection procedures and overall data management. The BLS is committed to producing the most accurate and complete data within the highest quality assurance guidelines. The BSRC was created to aid in this effort and it has demonstrated the effectiveness and value of its approach. Over the next few years, demand for BSRC consultation is expected to remain high as approaches are explored and tested to address increasing nonresponse in key Bureau surveys. Moreover, as the use of web-based surveys continues to grow, so too will the need for careful tests of instrument design and usability, human-computer interactions, and the impact of multiple modes on data quality. The BSRC is uniquely equipped with both the skills and facilities to accommodate these demands.
The extension of the accompanying clearance package reflects an attempt to accommodate the increasing interest by BLS program offices and other agencies in the methods used, and the results obtained, by the BSRC. This package reflects planned research and development activities for FY2024 through FY2026, and its approval will enable the continued productivity of a state-of-the-art, multi-disciplinary program of behavioral science research to improve BLS survey methodology.
The basic goal of BSRC is to improve, through interdisciplinary research, the quality of the data collected and published by the BLS. BLS is committed to producing the most accurate and complete data within the highest quality assurance guidelines. It is with this mission in mind that BSRC was created to aid in the effort of not only maintaining, but also improving the quality of the data collection process.
In 1988, Commissioner Janet Norwood established this laboratory to employ behavioral science to investigate all forms of oral and written communication used in the collection and processing of survey data. This exploration also includes all aspects of data collection such as mode, manuals, and interviewer training. BSRC performs a state-of-the-art service for many programs within BLS, DOL, and other agencies as requested, providing questionnaire redesign efforts, survey updates, and improvements in the overall quality of the data collection process. These efforts, in turn, increase data quality and reduce respondent burden. The techniques proposed here have been successfully applied to many BLS surveys.
The research techniques and methods BSRC uses in these studies will include both analyses of questionnaire construction and interview process, as well as survey technology. Within the structure of the questionnaire, BSRC anticipates conducting analyses in the following domains:
-- Question Analysis - Evaluation of individual questions, appraising question intent, assessment of semantic clarity, and an examination of relationships between questions.
-- Term Analysis - Evaluation of specific wording and phrases in terms of their psycholinguistic properties and an assessment of respondent interpretation of the meaning of these terms, at both the conscious and unconscious levels.
-- Instruction Analysis - Inspection of instructions for their semantic clarity, the degree to which they reflect the stated intention of investigators, ease of interpretation, and other considerations which may elicit unambiguous and appropriate answers or behaviors from respondents or interviewers.
-- Format Analysis - Review of questionnaires or subsets of questions for perceptual characteristics in order to facilitate better respondent comprehension and to promote more focused attention on the questionnaire or form.
Within the interview process, BSRC conducts several analyses to assess nonverbal communication, interpersonal dynamics, and symbolic interaction--the use of cultural symbols to make social statements. Staff members conduct research to evaluate the overall effectiveness of data collection procedural characteristics, including:
-- Interviewer Characteristics and Behavior Analysis - Study of the presentation of appearance, manner, relation to respondent population, etc., in order to enhance interpersonal skills of interviewers in general and to develop and improve procedures for training interviewers.
-- Respondent Characteristics and Behavior Analysis - Assessment of the social, cultural, ethnic, and demographic characteristics of the respondent and how they may bear upon interactions with the interviewer. Staff members also observe the behavior of respondents for cues concerning their reactions to the interview process. Because BLS collects data from different populations that change over time, the analysis of respondent characteristics needs frequent updating.
-- Mode Characteristics - Examination of the unique properties of interviewer and/or respondent behavior as a function of the media used to collect data; for example, self-administered surveys, personal interviews, telephone interviews, video interviews, and interviews utilizing computer assistive technologies (e.g., CAPI, CASI and CATI).
-- Usability Analysis - Evaluation of the effectiveness, efficiency, and satisfaction with which respondents complete tasks assigned to them, especially when using self-guided instruments (PAPI or CASI). This method also applies to data collectors’ interactions with CATI or CAPI instruments, and data user interactions with published information.
-- Data Collection Methodology Analysis – Assessment of alternative formats for collecting survey data (e.g., respondent provided records, administrative records). Staff will evaluate the validity and reliability of data collected through the alternative methodology as well as the level of respondent burden relative to current procedures.
BLS also uses a variety of methodologies, such as usability analysis, debriefings, and in-depth interviews, to better understand how to communicate more effectively with stakeholder and user communities through its website and other materials.
The BSRC’s research is expected to 1) improve the data collection instruments employed by BLS, 2) increase the accuracy of the economic data produced by BLS and on which economic policy decisions are based, 3) increase the ease of administering survey instruments for both respondents and interviewers, 4) increase response rates in panel surveys as a result of reduced respondent burden, 5) increase the ease of use of the BLS website and other BLS products, and 6) enhance the reputation of BLS through an enhanced user experience with BLS products and publications.
The application of cognitive and psychological theories and methods to survey data collection is widespread and well established. The consequences of failing to scientifically investigate the data collection process is to lag in the use of accepted practices, to apply archaic survey development techniques based on intuition and trial and error, and ultimately to incur a severe cost in data quality and in burden to respondents, interviewers, and data users alike. For example, without knowledge of what respondents can be expected to remember about the past and how to ask questions that effectively aid in the retrieval of the appropriate information, survey researchers cannot ensure that respondents will not take shortcuts to avoid careful thought in answering the questions or to be subject to undue burden. Likewise, without investigation of the interviewers’ roles and abilities in the data collection process, survey researchers cannot ensure that interviewers will ask their questions correctly with ease and fluency, or record the respondent’s answers correctly.
The research includes such methods as:
-- cognitive interviews, including the use of vignettes
-- focus groups
-- eye tracking
-- structured evaluation tasks such as card sorts
-- experiments involving the administration of forms to study respondents
-- usability tests of existing or proposed data collection and data dissemination systems (including survey respondent materials, software applications, and the public BLS website)
BSRC will use a variety of technologies to support their research. For some methods, such as cognitive interviews, usability tests, or focus groups, BSRC may make audio and/or video recordings of the sessions, with the respondent’s consent, storing the recordings in secure locations. These recordings allow for more detailed analysis of the sessions. They also allow team members to observe the sessions if they were not able to attend it live. BSRC may also use tools to automate the presentation of test materials to respondents.
BSRC will also use online tools to recruit respondents and administer unmoderated, self-administered methods. With these tools, BSRC can conduct studies easily and efficiently with a large number of respondents with specific characteristics of interest. These online studies can allow researchers to administer smaller tasks across large groups of respondents, reducing the burden for any one respondent. Finally, these self-administered online methods allow for experimentation of survey features such as question wording or format, in a way that is simply not possible in the laboratory or via video conferencing due to the resources required to obtain necessary sample sizes to detect statistical differences.
BSRC:
https://www.bls.gov/osmr/overview.htm#BSRC
Draft technical documentation available at:
https://www.dropbox.com/scl/fo/58qb1ovewblt8ab2laqpd/h?rlkey=u5hgi3lkbc06l8eyp1oaum6dl&dl=0
Surveys tested 2021-2023:
https://www.reginfo.gov/public/do/PRAICList?ref_nbr=202009-1220-006
FRN:
https://www.federalregister.gov/d/2023-27432