A randomized trial of nearly 1,300 parents found that while AI chatbot conversations can briefly boost HPV vaccination intent, established public health messaging remains more effective at delivering lasting influence, but neither approach increased actual vaccine uptake.
Study: Large Language Model Chatbot Conversations vs Public Health Materials and Parental HPV Vaccination Intentions. Image credit: TommyStockProject/Shutterstock.com
Brief chatbot conversations may promote short-term HPV vaccination intention among parents, but did not outperform well-designed public health messaging over a 45-day follow-up period, according to a new study published in JAMA Network Open.
Can chatbots improve parents’ vaccination decisions?
Human papillomavirus (HPV) vaccination has shown immense promise in preventing approximately 90% of HPV-related cancers, including cervical and oropharyngeal cancers. Despite high effectiveness, the uptake of this vaccine is significantly low worldwide.
Parental unwillingness is one of the main barriers against HPV vaccination, in addition to key structural challenges, including cost and accessibility. Public health agencies have made significant efforts to increase vaccine coverage, including evaluating artificial intelligence (AI)–based tools, such as large language model (LLM)–based chatbots.
Previous studies exploring chatbot-based vaccine promotion have produced mixed results. Studies comparing chatbot conversations with no intervention have reported short-term improvement in health intentions, while others have reported limited or no effects on vaccination intentions.
This randomized clinical trial, led by researchers at the University of Pennsylvania, aimed to compare the effectiveness of brief LLM chatbot conversations and high-quality government public health materials in improving parental intentions to vaccinate their children against HPV.
Parents compared chatbots with official guidance
The study included a total of 1297 parents from the US, Canada, and the UK who had at least one HPV vaccine–eligible child. The participants were randomly assigned to the no-intervention group (control), the government public health material group (minimum 3-minute exposure), and the LLM-based chatbot group (a 3-minute interaction with the chatbot).
Two chatbot variants were tested in the study, including a default LLM output style and a more conversational style. The primary aim was to examine self-reported parental willingness to vaccinate their children immediately after the intervention and again at 15 and 45 days to assess the durability of intention.
Public health materials produced the strongest effects
The assessment of immediate vaccination intent revealed increases across all intervention groups, with the public health material group showing the largest increase, followed by the default chatbot group and the conversational chatbot group.
The follow-up assessment at 15 days post-intervention revealed that both the conversational chatbot and public health materials increased HPV vaccination intent compared to the no-intervention control. However, no such difference was observed between the default chatbot group and the control group at this time point.
The follow-up assessment at 45 days post-intervention revealed that only public health materials moderately increased vaccination intent compared with the control group. However, none of the interventions increased self-reported HPV vaccination uptake at any time points.
Chatbots increased interest but not vaccination
The study reveals that the existing government public health materials and a 3-minute conversation with a default LLM-based chatbot produced similarly sized short-term improvements in parental willingness to vaccinate their children, although public health materials generated the largest overall effect. However, over the 45-day follow-up period, public health materials outperform chatbot conversations in sustaining parental vaccination intent.
Despite increased parental willingness, the study did not observe an increase in self-reported vaccination uptake in any of the intervention groups. The researchers believe that the 45-day post-intervention follow-up period may be too short for intent to translate into action. Furthermore, the researchers note that recruitment fell slightly short of the planned sample size, which may have reduced the statistical power to detect differences at later follow-up time points.
The absence of increased vaccine uptake despite higher parental willingness suggests that changing intentions alone may not be enough to drive action. Practical challenges such as healthcare access, scheduling requirements, and competing family priorities may have prevented parents from following through on vaccination decisions.
This finding is consistent with previous research showing that brief persuasive interventions often have limited long-term impact on health behaviors. While chatbot conversations may boost vaccination intent in the short term, they may be less effective at addressing the broader real-world factors that ultimately influence vaccination decisions.
The study also explored the associations between different messaging approaches and short-term changes in intention. Exploratory analyses suggested that conversational strategies emphasizing external validation and collective benefits were associated with greater parental willingness, whereas more personalized, empathy-focused interactions were associated with lower intent. However, the researchers cautioned that these findings should not be interpreted as causal, as conversational content was shaped by participants' responses.
Interestingly, participants rated the default chatbot as more empathetic than public health materials and spent more time engaging with it, yet these advantages did not translate into larger or more durable improvements in vaccination intent.
HPV vaccine chatbots failed to beat public guidance
Overall, the study findings indicate that LLM chatbots, at least in their current form, did not outperform established public health communication resources in promoting HPV vaccination and produced less durable effects than public health materials.
Based on the findings, the researchers suggest that chatbots may serve as a supplementary resource for addressing questions or reinforcing existing guidance, but should not be blindly used to replace established communication strategies.
The study relied on self-reported information on vaccination willingness and uptake, which may be subject to recall bias. Furthermore, study participants were predominantly White and female and were recruited through online platforms. These factors may restrict the generalizability of the findings to other demographic groups and to those with limited digital access or lower digital literacy.
Download your PDF copy by clicking here.
Experimental vaccine triggers broadly neutralizing HIV antibodies in macaques