Researchers in the United States have conducted studies suggesting that reports of people ingesting household cleaning agents as a way of protecting against coronavirus disease 2019 (COVID-19) are skewed by "problematic" responses to surveys.
A series of studies analyzing almost 1,400 survey participants showed that 80 to 90% of the claims regarding ingestion of household cleaners are made by "problematic respondents."
Jonathan Robinson from Prime Research Solutions in Queens, New York and colleagues from Columbia University and Marymount Manhattan College say these respondents also make implausible claims about "eating concrete for its iron content" and "recently having had a fatal heart attack."
Once "inattentive, mischievous, and careless respondents" were eliminated from the analysis, the team found no evidence to support that people had actually been ingesting household cleaners.
"These results show that reported ingestion of household cleaners and other similar dangerous practices are an artifact of problematic respondent bias," says the team.
The researchers warn that not screening out problematic respondents during survey-based studies can result in rare events being severely overestimated and normal practices being underestimated.
This presents a challenge to all survey research and threatens the validity of public-health policy, they add.
The team warns that researchers conducting surveys should mitigate against this risk by rigorously checking for problematic respondents, especially when the aim of surveys is to measure rare events.
A pre-print version of the paper is available on the medRxiv* server, while the article undergoes peer review.
Comparisons of problematic and non-problematic respondents’ reports of cleaning and disinfection practices since April, 2020.
*Important notice: medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
"Problematic responders" bias survey results
Surveys are one of the most commonly used data sources in social science, political science, public health and medical research and are used to inform public policy, medical practice, and public opinion.
However, survey research can be negatively impacted by participants who are inattentive, respond randomly to survey questions, or are 'mischievous' – misrepresenting who they are and providing responses that are intentionally false or misleading.
"Inattentive and mischievous respondents (collectively referred to in this report as 'problematic respondents') can bias the results of surveys," writes Robinson and colleagues. "The impact that such respondents can have on public health research has rarely been systematically examined."
Claims that people started ingesting disinfectants
One recent study suggested that in response to the COVID-19 pandemic, people started to engage in dangerous practices such as ingesting bleach and household disinfectants at an alarming rate.
The online survey results suggested that during April 2020, 39% of Americans engaged in at least one dangerous cleaning practice, including 4% who reported drinking or gargling diluted bleach, 4% who reported drinking or gargling soapy water, and 4% who reported drinking or gargling a household disinfectant.
"Translating percentages into raw numbers, according to these results tens million American adults were ingesting bleach and twenty million American adults were ingesting at least one cleaning product," write Robinson and colleagues.
Now, the team has examined this survey data to establish whether the rate of reported dangerous cleaning practices was overinflated due to problematic respondents.
The researchers used a combination of different instruments proven to be effective at detecting known characteristics of problematic respondent bias such as inattentiveness, mischievous responding, and acquiescence (the tendency to answer "yes" when offered a yes/no response option).
Up to 90% of the claims were made by problematic responders
In a series of studies covering almost 1,400 respondents, the team found that 4% of the survey population reported engaging in each of the three cleanser ingestion activities queried – drinking or gargling household disinfectant, soap, or bleach
However, after categorizing participants into problematic and non-problematic groups based on inattention and implausible claims, the researchers showed that 80 to 90% of these reports were from problematic respondents.
Consistent with the notion that problematic respondents can create the impression that almost anything can occur in the population no matter how implausible, 3% of these respondents also claimed they had never used the internet, 5.8% reported having "suffered a fatal heart attack," and 7% reported "eating concrete for its high iron content."
The respondents reported such claims at a similar rate to their reporting of ingesting household cleaners.
"Once inattentive, mischievous, and careless responses are taken out of the analytic sample, we find no evidence that people intentionally ingest household cleansers for protection against COVID-19 infection," writes the team.
The association between dangerous cleaning practices and health outcomes also became non-significant once problematic respondents were eliminated from the analysis.
"These results strongly suggest that reports of ingesting household cleansers to confer protection against COVID-19 infection are an artifact of problematic respondent bias," says Robinson and the team.
"A matter of public health concern"
Survey data reporting such practices among tens of millions of people risks normalizing such practices and potentially inadvertently reinforcing them among highly vulnerable people, say the researchers.
"For this reason, presenting the results of surveys that are subject to problematic respondent bias is itself a matter of public health concern."
Robinson and colleagues say validated instruments must be developed that protect surveys against such bias.
"To mitigate against these threats, surveys should rigorously check for problematic respondents using a fit-for-purpose approach, particularly when the survey aims to measure rare events," they write.
"Using these techniques significantly increases the accuracy of measurement and prevents problematic responses from invalidating survey results," concludes the team.
*Important notice: medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
The association between handgrip strength and functional outcomes in long COVID-19