Skip to main content

The choice of message and messenger to drive behavior change that averts the health impacts of wildfires: an online randomized controlled experiment

Abstract

Background

To reduce the negative health effects from wildfire smoke exposure, effective risk and health communication strategies are vital. We estimated the behavioral effects from changes in message framing and messenger in public health messages about wildfire smoke on Facebook.

Methods

During September and October 2021, we conducted a preregistered online randomized controlled experiment in Facebook. Adult Facebook users (n = 1,838,100), living in nine wildfire-prone Western U.S. states, were randomly assigned to see one of two ad versions (narrative frame vs. informational frame) from one of two messengers (government vs. academic). We estimated the effects of narrative framing, the messenger, and their interactions on ad click-through rates, a measure of recipient information-seeking behavior.

Results

Narrative frame increased click-through rates by 25.3% (95% CI = 22.2, 28.4%), with larger estimated effects among males, recipients in areas with less frequent exposure to heavy wildfire smoke, and in areas where predominant political party affiliation of registered voters was Republican (although not statistically different from predominantly-Democrat areas). The estimated effect from an academic messenger compared to a government messenger was small and statistically nonsignificant (2.2%; 95% CI = − 0.3, 4.7%). The estimated interaction effect between the narrative framing and the academic messenger was also small and statistically nonsignificant (3.9%; 95% CI = − 1.1, 9.1%).

Conclusions

Traditional public service announcements rely heavily on communicating facts (informational framing). Shifting from a fact-focused, informational framing to a story-focused, narrative framing could lead to more effective health communication in areas at risk of wildfires and in public health contexts more broadly.

Trial registration

Date registered: August 19, 2021; Registration DOI: https://doi.org/10.17605/OSF.IO/JMWUF

Peer Review reports

Background

Increased temperatures and drought associated with climate change continue to affect the frequency, extent, and severity of wildfires globally [1]. In the Western U.S., for example, fire frequency and the average extent of annual burned land has increased over the last seventy years, with the average size of fires more than doubling [2]. In addition to generating deadly flames, wildfires also generate harmful smoke over large areas downwind from the fire. This widespread smoke has important public health implications for large populations of people who are not at direct risk from the fire’s flames [3].

To reduce the negative health effects from wildfire smoke, effective risk and exposure mitigation communication strategies are vital. Public health messaging strategies about wildfire smoke traditionally rely on informational messages, like public service announcements (PSAs), that are delivered by government messengers and use scientific facts to communicate the health risks associated with wildfire smoke. These conventional messages are typically presented in the form of expository or didactic texts that leverage logic and reasoning to induce behavioral change. However, this informational message framing may fail to effectively drive mitigation behavior because it traditionally uses an inaccessible language that is too broad to be useful or relevant at the individual level and too technical to understand when presented to the average person [4]. An additional challenge in motivating behavior change is a lack of public trust in government messengers [5, 6]. There is evidence that mistrust in messages conveyed by government actors may be a key factor preventing action to combat public health problems (e.g., Martin et al. [7]) and other social policy problems, such as climate change (e.g., Hoffman [8]). Recent studies suggested that experimental design can advance efforts to achieve effective messaging strategies to increase wildfire risk mitigation [9,10,11].

Here, in a pre-registered online randomized controlled experiment [12], we test how the behavioral effects of messaging strategies for wildfire health risks vary with different framings and the messenger types. We target digital messages to adult users of Facebook in wildfire-prone, U.S. communities. In a 2 × 2 design, we vary the message framing between a narrative (story-based) framing and an informational (fact-based) framing, and vary the messenger between an academic messenger and a government messenger.

In recent years, the use of narratives (story-based communication) has received considerable attention by scholars in the fields of science and health communications [13,14,15]. We define narratives in line with Kreuter et al., which is a definition frequently adopted in the public health communication literature: “Narrative is a representation of connected events and characters that has an identifiable structure, is bounded in space and time, and contains implicit or explicit messages about the topic being addressed” [16]. A nonnarrative, however, adopts “expository and didactic styles of communication that present propositions in the form of reasons and evidence supporting a claim” [16]. Both narrative and nonnarrative forms of communication can carry the same message.

To explain how narratives can be effective in persuading and engaging people, theoreticians have proposed a set of mechanisms, such as narrative transportation [17], identification [18], and emotion [19]. These mechanistic theories posit that narratives can capture an audience’s attention by transporting them into the narrative’s world and by triggering cognitive and emotional systems [14], ultimately synthesizing complex ideas into comprehendible frameworks that are central to the human experience. They can therefore be persuasive by drawing individuals into the events relayed in the narrative and by encouraging the recipient to relate to the characters within the narrative. By bringing attention to health issues through engaging plot lines and relatable characters, narratives are thought to effectively convey public health messages [20]. Additionally, when employed in risk communication, narratives have been shown to effectively induce mitigation behavior (see references in Byerly et al. [21]). Communicators may therefore use narratives in their risk mitigation-focused interventions. To our knowledge, however, these effects have not been tested in the wildfire smoke context and through evidence-based field experiments.

The effectiveness of narratives may, however, depend on who delivers them. A growing body of evidence, particularly from the field of behavioral economics, has reported that the messenger can have important implications for how audiences process a message [22,23,24,25]. Among commonly identified mechanisms of the messenger effect are the expertise and trustworthiness of the messenger, which determine the extent to which a messenger may be deemed credible and thus influence a message recipient’s attitudes or behaviors [26, 27]. Therefore, two messengers who provide the same information under identical framings can trigger different interpretations and subsequently different behavioral effects.

Despite the attention on the effects of narrative framing and messengers, experimental studies typically look at the effects of narrative framing [21, 28,29,30,31] separate from the effects of messengers [32, 33]. The individual effects of each intervention (i.e., message framing and messenger) may be modest, but the effects cumulatively add up. Yet, very few studies investigate the cumulative effect of multiple behavioral interventions in the same context, and even fewer do it in a clean 2 × 2 factorial design. Few studies explored the interaction between message framing and messenger as behavioral interventions (e.g., [26, 34]); however, they frequently suffer from several important limitations, including underpowered empirical designs, issues of multiple comparisons, and using self-reported outcome variables to evaluate treatment effects. None of those studies report on observed behavioral outcomes in naturally occurring settings.

Our experimental study uses real behavioral outcomes in a naturally occurring online setting to explore the effects of public health message frame, messenger, and their interaction within the context of wildfire smoke and through digital delivery and engagement. With a sample of over 1.8 million Facebook users, we estimated the effects of ads that manipulated message framing (Narrative vs. Informational) and messenger type (Government vs. Academic) on click-through rates (CTR: the rate at which Facebook users click on the ad after being exposed to it). When clicked, the ads transferred the user to a landing page, which provided links to outreach materials and educational resources aimed at improving public awareness about the health effects of wildfire smoke and the actions people can take to mitigate exposure. By conducting a “natural field experiment” [35], wherein subjects are exposed to treatments in a setting that is “real life” as opposed to a controlled laboratory environment, we enhanced the external and ecological validity of our study. Through our use of a natural online setting, respondents’ behavior is more likely to represent typical online behavior.

Observing real behavior associated with messaging and risk communication in natural hazards contexts can be notoriously challenging. Given these challenges, previous studies [9, 10] recommended that scholars observe information-seeking behavior as a “low-cost” and “meaningful” behavioral response. Therefore, our findings have implications for risk communicators who seek to use innovative messaging strategies for building trust and communicating effectively in public health contexts, like wildfire smoke safety.

Health risks of wildfire smoke

Wildfire smoke is a mix of toxic gasses and particles, such as benzene, carbon dioxide, carbon monoxide, and fine particulate matter (e.g., PM2.5) – those particles ≤ 2.5 μm in diameter – which are known to be dangerous for humans when exposed (both in- and outdoors) [36]. These particles can pass through the nose and throat and enter the lungs, and the smallest particles can even penetrate the blood circulatory system. Smoke inhalation and exposure can increase the risk of cardiovascular- and respiratory-related effects, such as asthma attacks, pneumonia, stroke and chronic heart and lung diseases [37].

Given the adverse effect of wildfire smoke on public health, minimizing exposures to mitigate these health effects is of paramount importance. Effective communication can influence smoke-prone communities to engage in risk-reduction behaviors, [38, 39] including information-seeking and planning [9, 10] for wildfire events ahead of time that will further mitigate the health risks of exposure.

Methods

We conducted an online message-framing experiment through Facebook ads. We used a 2 × 2 factorial design where Facebook users were randomly assigned to see one of two ad versions (Informational vs. Narrative messages) from one of two messenger types (Government vs. Academic sources). Randomization was achieved by employing the A/B split testing functionality embedded in Facebook ads. By using A/B testing, the sample was broken down into random, non-overlapping groups (see Orazi and Johnston [40] and Kohavi et al. [12] for more details on A/B split testing). Ad campaign delivery optimization was disabled to avoid disrupting the randomization. To prevent a subject from receiving ads multiple times, the ad exposure frequency was manually capped at one over the course of the experimental period. To make sure that roughly the same number of people were in each treatment arm, we equally divided the advertising budget across treatment arms.

The target audience included Facebook adult users (age ≥ 18 yrs), living in nine wildfire smoke-prone states in the Western U.S. including Arizona, California, Colorado, Idaho, Montana, Nevada, Oregon, Utah, and Washington. The target audience excluded users who opted out of receiving Facebook ads.

We measured the CTR for each treatment arm, which quantifies the number of clicks relative to the number of times the ad was shown. CTRs (i.e., response rates) are a commonly used digital marketing metric that represents users’ engagement and motivation [12, 40, 41]. To conduct a power analysis, we ran simulations based on realistic CTRs for Facebook newsfeed ads in the North American region. The median CTR across all industries in the Facebook newsfeed in 2020 was estimated to be 1.11% [42]. We simulated the data generating process such that the interaction effect of message framing and the messenger was half the size of their main effects on CTR. The power simulations imply that, with a sample size of 1.76 million users, we could detect main effects of 0.2 percentage point increase and an interaction effect of 0.1 percentage point increase in CTR with 80% power and a Type 1 error rate of 5%.

Data were collected for 14 consecutive days in September 2021 and October 2021. The advertisement budget and duration were pre-specified. The entire campaign reached over 1.8 million users.

Intervention materials

The informational message was a nonnarrative expository text that provided facts and reasons to support a claim about health risks associated with wildfire smoke exposure. The text was 87 words in length and was adapted from widely used PSAs about the same topic to replicate the status quo. We synthesized the text from PSAs delivered by the Centers for Disease Control and Prevention (CDC), U.S. Environmental Protection Agency (EPA), U.S. Department of Health and Human Services (HHS), and local government agencies in the study region (see Additional file 1 Fig. S1 for the screenshots of informational ads).

The narrative intervention was roughly the same length as the informational message (89 words) and embedded the same message, but in narrative form. To make the two framings comparable, the narrative intervention was deliberately framed to closely match the informational detail provided in the informational message frame. The narrative story featured Alex C., a generic character who experienced wildfire smoke inhalation impacts during the previous wildfire season. Several studies report that realism is central to narrative engagement [43,44,45]. Realistic narratives refer to a story-focused framing whose characters’ behaviors could reasonably occur in real world, are related to everyday experiences, portray actual events, and can fulfill the targeted audiences’ expectations. To construct a realistic digital narrative aligned with real-world stories of wildfire smoke exposure, we conducted a review of published stories via LexisNexis —an online tool with an advanced search engine enabling us to review contents published as news, blog posts, press article, etc. with the possibility of narrowing down the search to stories of those who suffered or experienced wildfire smoke. The text was adapted and synthesized from multiple sources and was based on the experience of real characters. We used a generic name for the story character to maintain confidentiality. To be inclusive, the chosen name was gender-, race-, and ethnicity-neutral with no age and location specifications (see Additional file 1 Fig. S1 for the screenshots of narrative ads). The perceived messenger on the ads was manipulated by varying the page extension between .EDU (for academic messenger) and .GOV (for government messenger).

We created a Facebook page named Wildfire Smoke to deliver the ads and to serve as a landing page for clickers (see Additional file 1 Fig. S2 for the screenshots of the landing page). This page provided visitors with publicly available information about wildfire smoke risk and mitigation actions that were scientifically sound (e.g., resources developed by CDC and EPA).

Statistical analysis

We contrast the CTRs across two (message frames: narrative vs informational) by two (messengers: academic vs government) treatment combinations and estimate the effects on the CTRs from the message frame, messenger, and their interactions. Additionally, we estimate conditional treatment effects moderated by gender, the predominant political party affiliation in recipients’ region, and regional records of heavy smoke exposure (all moderators were pre-registered). These subgroup analyses are motivated by persistent claims of gender differences in affective processing and orientation (e.g., see Friesdorf et al. [46]), the previous findings about political affiliation differences in narrative receptivity [21], and the geographic heterogeneity in the historic trends of smoke exposure, which may lead users to behave differently in response to public health messages and how such messages are framed. For the gender analysis, we use Facebook’s aggregate-level ad reports (i.e., CTR data) that are broken down into fixed “gender-age” bins (i.e., males/females 18–24, 25–34, 35–44, 45–54, 55–64 and 65+ yrs. old). For the two regional attributes (i.e., smoke exposure and predominant party affiliations), we use Facebook’s aggregate-level ad reports that are broken down into designated market areas (DMAs). A DMA is a region that includes multiple adjacent counties in the U.S. and is used by media (e.g., Facebook) to define television and radio markets.

To measure the dominant political party affiliation in a DMA, we first calculated the dominant political party affiliation in the counties that make up that DMA. If the number of registered voters for the Republican Party was greater than the number of registered voters for the Democratic Party, the county was considered majority-Republican (data was obtained for November 2020—the most recent presidential election). We acknowledge that measuring the moderating effect of individual-level party affiliation would also be interesting, but we have no way of matching our Facebook participants to public voting databases. Smoke exposure was also first calculated at the county level. We used the methods and the dataset described in Vargo's work [47]. This method utilizes data from the National Oceanic and Atmospheric Administration’s (NOAA) Office of Satellite and Product Operations Hazard Mapping System’s Smoke Product to estimate the potential daily burden of smoke related to wildland fires on communities across the United States. For each county, we estimated the daily exposure to wildfire smoke and assigned it to one of three categories: light, medium, or heavy, approximately corresponding to fine particulate matter (PM2.5) concentrations of 0–10, 10–21, and 22+ μg/m3, respectively (see Vargo for more details [47]; also see Haikerwal et al. [48]). Then, smoky days was calculated as the number of days in the past 28 months the county population experienced heavy exposure to wildfire smoke (28 months is the national average length-of-stay of a U.S. renter). Finally, we used the method described in Carney [49] and mapped counties to DMAs by determining which counties comprised each DMA and weighting the data by county population.

We specified our main and moderator analyses and preregistered our analysis plan on Open Science Foundation prior to examining the data and conducting the analysis herein (see https://osf.io/jmwuf). To reject the null hypotheses, we used two-tailed z-tests, and to control the false positive rate in multiple comparisons, we used the Bonferroni correction with family-wise error rate of (FWER ≤0.05) [50]. We conducted six different comparisons (two main effects, one interaction effect, and three moderator effects) using an alpha level of 0.05, and thus, employing Bonferroni correction would result in new alpha level to be 0.05 divided by 6. Additionally, to increase the precision of the estimation, we used logistic regression estimators that adjustsed for covariates including gender, age, and regional characteristics such as history of smoky days and predominant political party affiliation of the region. However, to protect the confidentiality of users, Facebook does not allow the generation of reports with all of these covariates combined. Instead, one could obtain ad reports including total observed clicks and non-clicks for each treatment arm broken down into subgroups, either by fixed gender-age bins or by regions (i.e., DMAs). Therefore, regression models could not include all covariates simultaneously (see Supplemental material).

Results

The ads were shown to 1,838,100 people. Table 1 shows the descriptive statistics of the full sample and each treatment arm. Overall, 26,390 unique users clicked on ads. Even though the frequency—the number of times an experimental condition was shown to an individual user—was manually capped at one, in some instances, Facebook showed the same ad to a user more than once. This yielded an average frequency of 1.014. This algorithmic error is endogenous to the platform and needs to be accounted for. We used the correction rule suggested by Orazi and Johnston [40]. For each treatment arm, we divided the total number of clicks by the frequency, rounding down. The corrected number of clicks and CTRs for each treatment arm is shown in Table 1.

Table 1 Descriptive statistics of the full sample and each treatment arm, Facebook users in Western U.S

The average CTR for informational ads was 1.26%. Recipients of the narrative framing had an average CTR that was 0.3 percentage points higher than recipients of the informational framing, or a 25.3% increase (95% CI = 22.2, 28.4%, p < 0.001). Users who received their message from the academic messenger had an average CTR that was 0.03 percentage points higher than users who received the message from the government messenger, or a 2.2% increase (95% CI = − 0.3, 4.7%, p = 0.076). The average effect of the narrative framing on CTRs was 3.9% higher when the ad came from academic messenger (95% CI = − 1.1, 9.1%, p = 0.13) (Fig. 1). This estimated interaction effect of academic messenger and narrative framing on CTRs was small and statistically nonsignificant. Our estimated treatment effects are robust to covariate adjustment using logistic regression (Table 2).

Fig. 1
figure 1

Estimated treatment effects of narrative framing, academic messenger and their interactions, conditional on gender and the predominant regional political party affiliation based upon voter registrations in the recipients’ area of residence. Error bars show 95% confidence intervals. NAR = narrative frame, INF = informational frame, EDU = academic messenger, GOV = government messenger, DEM = majority-Democrat regions, REP = majority-Republican regions

Table 2 Estimated treatment effect of framing on click-through behavior

In our moderator analyses, we estimated that the narrative framing, as compared to the informational framing, increased response rates among females by 20% (95% CI = 14.1, 26.1%), while this increase was 27.7% (95% CI = 21.5, 34.3%) among males (Fig. 1). The difference in subgroup treatment effects is − 6.4% (95% CI = − 11, − 1.7%, p < 0.01). This difference is statistically significant using the Bonferroni correction to control for multiple comparisons in our moderator analysis.

Furthermore, we estimated that the narrative framing increased CTRs among individuals living in majority-Republican areas by 28.9% (95% CI = 23.9, 34%), while this effect was smaller in majority-Democrat areas: 21.9% (95% CI = 15.7, 28.4%) (Fig. 1). The difference in the subgroup treatment effects is 5.4% (95% CI = − 0.1, 11.4%, p = 0.056), but it is not statistically significant using the Bonferroni correction to control for multiple comparisons in our moderator analysis. In addition, for ads from the academic messenger, as compared to the government messenger, we estimated a small, statistically nonsignificant difference of 2.2% (95% CI = − 3.2, 8.0%, p = 0.42) in CTRs among majority-Republican areas relative to majority-Democrat areas.

Finally, the effect of narrative framing on CTRs was moderated by the number of days in the past 28 months that an individual’s region was exposed to heavy wildfire smoke. We estimate that every one day increase in the number of smoky days leads to a decrease in the effect of the narrative framing of 0.3% (95% CI = − 0.4, − 0.1%, p < 0.001). This estimated effect implies that the effect of the narrative framing is larger among users who live in regions with less frequent exposure to heavy wildfire smoke (see Fig. 2).

Fig. 2
figure 2

The estimated click through rate (CTR) for informational and narrative framing given the number of smoky days in the targeted areas. A logistic regression estimator is used. The second vertical axis on the right shows the odds ratio of click through behavior for narrative ads relative to informational ads. Smoky days = number of days in the past 28 months a region was exposed to heavy wildfire smoke

Discussion

Effective communication of health information is critical to mitigate the adverse effects of environmental health hazards, such as wildfire smoke exposure. Yet, the informational messages about environmental health hazards that are typically used in PSAs conveyed by government messengers may not be the most effective way to communicate risk. These messages often adopt a fact-based, scientific language that is not easy to comprehend and is not relatable, particularly by the most vulnerable populations including people of low socio-economic status, outdoor workers, and older adults [37, 51, 52].

Our study provides causal evidence that shifting from a fact-focused, informational framing to a story-focused, narrative framing can affect peoples’ willingness to seek out information about the hazards of wildfire smoke and the ways they can mitigate the health effects from exposure to this smoke. In our experiment, the narrative frame, in comparison to the informational frame, increased the odds of information-seeking behavior as measured by the CTR, a commonly used digital marketing metric. Our study adds to the growing literature on the effectiveness of using narratives in public health interventions. However, we did not detect a large effect from changing the messenger type from a government messenger to an academic messenger: the estimated effect was positive, but small and statistically nonsignificant.

In our study, the CTRs were higher among all subgroups of recipients of narrative ads compared to recipients of informational ads, although the estimated treatment effect was moderated by recipient and regional attributes. Narrative ads were more effective in increasing CTRs among male users compared to female users. Like the results in Byerly et al. [21], our results thus contradict prior claims that narratives are more persuasive in female subjects and that women in general may report greater narrative transportation than men [17, 53] (also see references in Byerly et al. [21]). Moreover, we find evidence that narrative framing had larger, though statistically nonsignificant, effects on CTRs among Facebook recipients living in areas where the number of registered Republicans outweighed registered Democrats. These results about the moderating effect of regional party affiliations on message framing complement prior research that looks at such moderating effect from the political affiliation of the individuals themselves [21, 34]. The results, however, are only suggestive because the estimated effect is not statistically significant after employing the Bonferroni method to control for 5% FWER in our subgroup analyses. We conclude that further research is needed before making any claims about how the effect of narrative framing varies by regional party affiliations.

Importantly, our analysis implies that narrative framing was more effective in increasing CTRs among users who live in regions where heavy smoke events are less frequent. Climate change has made human communities more prone to wildfires, especially in places with no long history of exposure to flames and smoke, where the wildfires and smoke are new-fashioned phenomena. Our results, therefore, have important implications for effective communication of wildfire smoke heath impacts and mitigation behavior, especially in such regions where wildfire smoke is mostly unheard of, and the residents are less likely to be prepared for such events.

Even though click rates on Facebook ads are often low [42], these ads can reach very large populations at low cost, thereby having the potential to induce meaningful behavior change in a population. In our experiment, the narrative framing generated 3038 additional “engaged people” (people who click on ads) over the informational framing. If the narrative framing were scaled up to the entire Facebook population in these nine states of 38 million users (an estimate from the Facebook Ads Manager platform), one could generate about 114,000 more engaged users (people who click on ads to seek more information) than one could generate if the informational framing were used instead.

Our study is an important step towards a better understanding of how best to use digital media to communicate risks and mitigation actions, an understanding that could be applied across a wide array of domains. Digital media allows for low-cost and rapid dissemination of information. Social digital media, like Facebook, have become ubiquitous in American life. According to the Pew Research Center, as of 2021, Facebook is the largest social media platform used by Americans, with more than 200 million active users across demographic groups. Facebook and other online platforms could be used to deliver, for example, personalized information on air quality conditions and health risks. Using these online platforms to conduct online randomized controlled experiments to test the behavioral effects of health communication interventions and in a wide range of contexts, like wildfire smoke safety, can potentially enable accelerated public health improvements [12]. With a better understanding of how to harness these online platforms to enhance the communication of risks and to inform the adoption of individual or community level health-protective actions, practitioners could nudge people towards behaviors that improve public health across a broad range of communities.

Limitations

Our study has three main limitations. First, although we measured actual behavior in our randomized natural field experiment, the behavioral outcome was limited to information-seeking behavior measured by CTRs, as opposed to mitigation actions. Given the anonymized data on the Facebook platform, it was not possible to identify the subjects who observed the ads or follow-up with them about their attitudes toward, or engagement with, mitigation actions that may occur over long time periods. Yet, inducing information-seeking behavior is considered an important first step for individuals to pursue risk mitigation actions [9, 10].

Second, our design intentionally did not employ the use of named governmental or academic messengers. We avoided using specific institutions on the messages because we worried that recipient perceptions of particular institutions could confound our attempt to identify a generalizable treatment effect from changing the messenger from a government organization to an academic organization (e.g., an unknown federal agency versus an internationally known academic institution). Instead, we manipulated the landing page’s address presented on the ads between .GOV and .EDU extensions, which represented a government and an academic organization, respectively. However, these differences in the perceived messenger were potentially not salient to all ad recipients, particularly those who are not aware of domain extensions and their association with organization types. Future research can consider ways to signal differences in messenger types in a more salient way, as well as explore a wider range of institution categories, such as non-governmental organizations or health care providers.

Third, given the restrictions imposed by using Facebook ads manager, no data were made available with which we could run analyses of subgroup effects by party affiliation and smoke exposure at the county level. Our pre-analysis plan had specified doing the analysis at this level, but after learning more about the Facebook system, we determined that we could only do the analysis at a higher level of aggregation (i.e., DMAs.). This is the only deviation from our registered pre-analysis plan.

Conclusion

We conducted a natural field experiment on a social media platform to study how changes in the framing and messenger of public health risk messages affect behavior. Specifically, we contrasted the use of narrative vs information framing and the use of academic vs government messengers, and their interactions, on information-seeking behavior to avert the public health impacts of increased wildfire smoke exposure. Our study fills important gaps in the empirical literature that reports on the use of online randomized controlled experiments to assess the behavioral effects of digital message framing to communicate about public health risks.

Availability of data and materials

All data and code supporting the findings of this study are available and can be downloaded from OSF at https://osf.io/9ax6n/?view_only=55488a95825444aa830eadbd38fe3565.

References

  1. Burke M, Driscoll A, Heft-Neal S, Xue J, Burney J, Wara M. The changing risk and burden of wildfire in the United States. Proc Natl Acad Sci. 2021;118(2):e2011048118.

    Article  CAS  Google Scholar 

  2. Weber KT, Yadav R. Spatiotemporal trends in wildfires across the Western United States (1950–2019). Remote Sens. 2020;12(18):2959.

    Article  Google Scholar 

  3. Reisen F, Duran SM, Flannigan M, Elliott C, Rideout K. Wildfire smoke and public health risk. Int J Wildland Fire. 2015;24(8):1029.

    Article  CAS  Google Scholar 

  4. U.S. EPA. Wildfire Smoke and Health Risk Communication: Integrating Social and Natural Sciences to Improve Risk Communication and Management Strategies in Impacted Communities (Workshop Report, Mar 2017) [Internet]. Research Triangle Park, NC; 2017 Mar [cited 2021 Nov 29]. Available from: https://www.epa.gov/air-research/wildfire-smoke-and-health-risk-communication-workshop-and-report

  5. Errett NA, Roop HA, Pendergrast C, Kramer CB, Doubleday A, Tran KA, et al. Building a practice-based research agenda for wildfire smoke and health: a report of the 2018 Washington wildfire smoke risk communication stakeholder synthesis symposium. Int J Environ Res Public Health. 2019;16(13):2398.

    Article  Google Scholar 

  6. Winter G, Vogt CA, McCaffrey S. Examining social trust in fuels management strategies. J For. 2004;102(6):8–15.

    Google Scholar 

  7. Martin A, Gravelle TB, Baekkeskov E, Lewis J, Kashima Y. Enlisting the support of trusted sources to tackle policy problems: the case of antimicrobial resistance. PLoS One. 2019;14(3):e0212993.

    Article  CAS  Google Scholar 

  8. Hoffman AJ. How culture shapes the climate change debate: Stanford University Press; 2015.

    Google Scholar 

  9. Byerly H, Meldrum JR, Brenkert-Smith H, Champ P, Gomez J, Falk L, et al. Developing behavioral and evidence-based programs for wildfire risk mitigation. Fire. 2020;3(4):66.

    Article  Google Scholar 

  10. Meldrum JR, Brenkert-Smith H, Champ PA, Gomez J, Byerly H, Falk L, et al. Would you like to know more? The effect of personalized wildfire risk information and social comparisons on information-seeking behavior in the wildland–urban interface. Nat Hazards. 2021;106(3):2139–61.

    Article  Google Scholar 

  11. Brenkert-Smith H, Dickinson KL, Champ PA, Flores N. Social amplification of wildfire risk: the role of social interactions and information sources. Risk Anal. 2013;33(5):800–17.

    Article  Google Scholar 

  12. Kohavi R, Tang D, Xu Y, Hemkens LG, Ioannidis JPA. Online randomized controlled experiments at scale: lessons and extensions to medicine. Trials. 2020;21(1):150.

    Article  Google Scholar 

  13. Shen F, Sheer VC, Li R. Impact of narratives on persuasion in health communication: a Meta-analysis. J Advert. 2015;44(2):105.

    Article  Google Scholar 

  14. Dahlstrom MF. Using narratives and storytelling to communicate science with nonexpert audiences. Proc of the Nat Acad of Sci. 2014;111(Supplement_4).

  15. Yang Y, Hobbs JE. The power of stories: narratives and information framing effects in science communication. Am J Agric Econ. 2020;102(4):1271.

    Article  Google Scholar 

  16. Kreuter MW, Green MC, Cappella JN, Slater MD, Wise ME, Storey D, et al. Narrative communication in cancer prevention and control: a framework to guide research and application. Ann Behav Med. 2007;33(3):221–35.

    Article  Google Scholar 

  17. Green MC, Brock TC. The role of transportation in the persuasiveness of public narratives. J Pers Soc Psychol. 2000;79(5):701.

    Article  CAS  Google Scholar 

  18. de Graaf A, Hoeken H, Sanders J, Beentjes JWJ. Identification as a mechanism of narrative persuasion. Commun Res. 2012;39(6):802.

    Article  Google Scholar 

  19. Morris BS, Chrysochou P, Christensen JD, Orquin JL, Barraza J, Zak PJ, et al. Stories vs. facts: triggering emotion and action-taking on climate change. Clim Chang. 2019;154(1–2):19.

    Article  Google Scholar 

  20. Frank LB, Murphy ST, Chatterjee JS, Moran MB, Baezconde-Garbanati L. Telling stories, saving lives: creating narrative health messages. Health Commun. 2015;30(2):154–63.

    Article  Google Scholar 

  21. Byerly H, Ferraro PJ, Li T, Messer KD, Weigel C. A story induces greater environmental contributions than scientific information among liberals but not conservatives. One Earth. 2021;4(4):545.

    Article  Google Scholar 

  22. Arbuckle JG, Morton LW, Hobbs J. Understanding farmer perspectives on climate change adaptation and mitigation. Environ Behav. 2015;47(2):205–34.

    Article  Google Scholar 

  23. Whiting A, Kecinski M, Li T, Messer KD, Parker J. The importance of selecting the right messenger: A framed field experiment on recycled water products. Ecolog Eco. 2019;161:1–8.

    Article  Google Scholar 

  24. Maclean JC, Buckell J, Marti J. Information source and cigarettes: Experimental evidence on the messenger effect: National Bureau of Economic Research; 2019.

    Book  Google Scholar 

  25. Constantino SM, Pianta S, Rinscheid A, Frey R, Weber EU. The source is the message: the impact of institutional signals on climate change–related norm perceptions and behaviors. Clim Chang. 2021;166(3–4):35.

    Article  Google Scholar 

  26. Hafner R, Elmes D, Read D. Exploring the role of messenger effects and feedback frames in promoting uptake of energy-efficient technologies. Curr Psychol. 2019;38(6):1601.

    Article  Google Scholar 

  27. Pornpitakpan C. The persuasiveness of source credibility: a critical review of five decades’ evidence. J Appl Soc Psychol. 2004;34(2):243–81.

    Article  Google Scholar 

  28. Jiang LC. Effects of narrative persuasion in promoting influenza vaccination in Hong Kong: a randomized controlled trial. Pat Educ Couns. 2021;104(4):800.

    Article  Google Scholar 

  29. Ballard AM, Davis A, Hoffner CA. The impact of health narratives on persuasion in African American women: a systematic review and Meta-analysis. Health Commun. 2021;36(5):560.

    Article  Google Scholar 

  30. Adebayo AL, Davidson Mhonde R, DeNicola N, Maibach E. The effectiveness of narrative versus didactic information formats on pregnant Women’s knowledge, risk perception, self-efficacy, and information seeking related to climate change health risks. Int J Environ Res Public Health. 2020;17(19):6969.

    Article  Google Scholar 

  31. Murphy ST, Frank LB, Chatterjee JS, Moran MB, Zhao N, Amezola de Herrera P, et al. Comparing the relative efficacy of narrative vs nonnarrative health messages in reducing health disparities using a randomized trial. Am J Public Health. 2015;105(10):2117–23.

    Article  Google Scholar 

  32. Favero N, Jilke S, Wolfson JA, Xu C, Young MM. Messenger effects in COVID-19 communication: does the level of government matter? Health Pol OPEN. 2021;2:100027.

    Article  Google Scholar 

  33. Fishman J, Greenberg P, Bagga MB, Casarett D, Propert K. Comparing strategies for health information dissemination: messengers that can help or hinder. Am J Health Promot. 2018;32(4):932–8.

    Article  Google Scholar 

  34. Solnick RE, Chao G, Ross RD, Kraft-Todd GT, Kocher KE. Emergency physicians and personal narratives improve the perceived effectiveness of COVID-19 public health recommendations on social media: a randomized experiment. Acad Emerg Med. 2021;28(2):172.

    Article  Google Scholar 

  35. Harrison GW, List JA. Field experiments. J Econ Lit. 2004;42(4):1009–55.

    Article  Google Scholar 

  36. Doubleday A, Schulte J, Sheppard L, Kadlec M, Dhammapala R, Fox J, et al. Mortality associated with wildfire smoke exposure in Washington state, 2006–2017: a case-crossover study. Environ Health. 2020;19(1):1.

    Article  Google Scholar 

  37. Reid CE, Brauer M, Johnston FH, Jerrett M, Balmes JR, Elliott CT. Critical review of health impacts of wildfire smoke exposure. Environ Health Perspect. 2016;124(9):1334.

    Article  Google Scholar 

  38. Martin IM, Bender H, Raish C. What motivates individuals to protect themselves from risks: the case of wildland fires. Risk Anal. 2007;27(4):887–900.

    Article  Google Scholar 

  39. Remenick L. The role of communication in preparation for wildland fire: a literature review. Environ Commun. 2018;12(2):164–76.

    Article  Google Scholar 

  40. Orazi DC, Johnston AC. Running field experiments using Facebook split test. J Bus Res. 2020;118:189.

    Article  Google Scholar 

  41. Matz SC, Kosinski M, Nave G, Stillwell DJ. Psychological targeting as an effective approach to digital mass persuasion. Proc Natl Acad Sci. 2017;114(48):12714.

    Article  CAS  Google Scholar 

  42. AdStage. Q1 2020 Paid Search and Paid Social Benchmark Report [Internet]. 2020 [cited 2021 Nov 29]. Available from: https://www.adstage.io/resources/q1-2020-ppc-benchmark-report/?hsLang=en

  43. Yale RN. Measuring narrative believability: development and validation of the narrative believability scale (NBS-12). J Commun. 2013;63(3):578–99.

    Article  Google Scholar 

  44. Cho H, Shen L, Wilson K. Perceived realism. Commun Res. 2014;41(6):828–51.

    Article  Google Scholar 

  45. Busselle R, Bilandzic H. Fictionality and perceived realism in experiencing stories: a model of narrative comprehension and engagement. Commun Theo. 2008;18(2):255–80.

    Article  Google Scholar 

  46. Friesdorf R, Conway P, Gawronski B. Gender differences in responses to moral dilemmas. Per Soc Psychol Bull. 2015;41(5):696.

    Article  Google Scholar 

  47. Vargo JA. Time series of potential US wildland fire smoke exposures. Front Pub Health. 2020;21:8.

    Google Scholar 

  48. Haikerwal A, Akram M, del Monaco A, Smith K, Sim MR, Meyer M, et al. Impact of Fine Particulate Matter (PM 2.5 ) Exposure During Wildfires on Cardiovascular Health Outcomes. J Am Heart Assoc. 2015;4(7):e001653.

    Article  Google Scholar 

  49. Carney K. Current census demographics by DMA [Internet]. 2015 [cited 2021 Nov 29]. Available from: https://blog.cubitplanning.com/2015/05/current-census-demographics-by-dma/

  50. Wang R, Ware JH. Detecting moderator effects using subgroup analyses. Prev Sci. 2013;14(2):111.

    Article  Google Scholar 

  51. U.S. EPA. Integrated Science Assessment (ISA) for Particulate Matter (Final Report, Dec 2009) [Internet]. Washington, DC.; 2009 Dec [cited 2021 Nov 29]. Available from: https://cfpub.epa.gov/ncea/isa/recordisplay.cfm?deid=216546

  52. Rappold AG, Cascio WE, Kilaru VJ, Stone SL, Neas LM, Devlin RB, et al. Cardio-respiratory outcomes associated with exposure to wildfire smoke are modified by measures of community health. Environ Health. 2012;11(1):71.

    Article  Google Scholar 

  53. van Laer T, de Ruyter K, Visconti LM, Wetzels M. The extended transportation-imagery model: a Meta-analysis of the antecedents and consequences of consumers’ narrative transportation. J Consum Res. 2014;40(5):797.

    Article  Google Scholar 

Download references

Acknowledgements

Ferraro acknowledges support from the USDA-funded Center for Behavioral and Experimental Agri-Environmental Research (CBEAR; award #2019-67023-29854).

Funding

Not Applicable.

Author information

Authors and Affiliations

Authors

Contributions

P. A. and J. H. designed and conducted the study. P. A. analyzed the data and wrote the original draft. P. F. assisted with data analysis and writing of the article, and provided supervision, project administration, and resources. All authors reviewed the manuscript.

Corresponding author

Correspondence to Payam Aminpour.

Ethics declarations

Ethics approval and consent to participate

We obtained approval of Johns Hopkins University’s Institutional Review Board (HIRB00013613) and the Research Protections Office of the National Institute of Standards and Technology (EL-2021-0288).

Approvals were granted under the terms of FWA00005834 Federal-Wide Assurance of compliance with DHHS Regulations for Protection of Human Research Subjects and U.S. Department of Commerce Common Rule for the Protection of Human Subjects (45 CFR 46, Subpart A). Due to the Facebook Terms of Service, all subjects must consent to receive the ads or opt out otherwise. All data provided by Facebook were summary statistics. It was therefore impossible to identify, interact with, or obtain written consent from individuals. Anonymized aggregated data guaranteed the protection of participants’ privacy.

Consent for publication

Not Applicable.

Competing interests

The authors have no financial or non-financial competing interests to declare. Certain commercial equipment, instruments, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by the institutions authors affiliated with, nor is it intended to imply that the materials or equipment identified are necessarily the best available for the purpose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aminpour, P., Helgeson, J.F. & Ferraro, P.J. The choice of message and messenger to drive behavior change that averts the health impacts of wildfires: an online randomized controlled experiment. BMC Public Health 22, 2359 (2022). https://doi.org/10.1186/s12889-022-14801-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12889-022-14801-6