- Research article
- Open Access
An audit of the quality of online immunisation information available to Australian parents
BMC Public Health volume 17, Article number: 76 (2017)
The Internet is increasingly a source of health information for parents, who use the Internet alongside health care providers for immunisation information. Concerns have been raised about the reliability of online immunisation information, however to date there has been no audit of the quality or quantity of what is available to Australian parents. The objective of this study was to address this gap by simulating a general online search for immunisation information, and assessing the quality and quantity of the web sites returned by the search.
We used Google trends to identify the most common immunisation search terms used in Australia. The ten most common terms were entered into five search engines and the first ten non-commercial results from each search collated. A quality assessment tool was developed using the World Health Organization Global Advisory Committee on Vaccine Safety (GACVS) criteria for assessing the quality of vaccine safety web sites, and used to assess and score the quality of the sites.
Seven hundred web pages were identified, of which 514 were duplicates, leaving 186 pages from 115 web sites which were audited. Forty sites did not include human immunisation information, or presented personal opinion about individuals, and were not scored. Of the 75 sites quality scored, 65 (87%) were supportive of immunisation, while 10 (13%) were not supportive. The overall mean quality score was 57/100 (range 14/100 to 92/100). When stratified by pro and anti-vaccination stance, the average quality score for pro-vaccine sites was 61/100, while the average score for anti-vaccine sites was 30/100.
Pro-vaccine information could be divided into three content groups: generalist overview with little detail; well-articulated and understandable detail; and lengthy and highly technical explanations. The main area found to be lacking in pro-vaccine sites was lack of transparent authorship.
Our findings suggest a need for information which is easily found, transparently authored, well-referenced, and written in a way that is easily understood.
The internet has changed the way people find health information . Recent figures show that, of the 13.3 million Australians who access the Internet at home, research activities were among the most commonly reported undertakings . Increasingly, parents are using Internet-sourced information in addition to traditional consultations with health care providers to improve their understanding of children’s health-related issues . One study estimates 43% of Australian parents access child health information on the Internet .
Negative online messages are thought to undermine parents’ confidence in vaccines , leading government and health care providers to raise concerns about the reliability of health information available on the internet. However, no comprehensive assessment of the quantity or quality of online general immunisation information available to Australian parents has been conducted. A previous study by Wolfe and Sharp assessed the pro- or anti-vaccine stance of parental internet search results for immunisation information overseas  and found that the search terms and search engine used had a bearing on the amount of pro- or anti- vaccination material returned by the search, and the order in which they were presented. Another study by Sak et al. compared the quality of pro- and anti-vaccine web sites, but the sample was not intended to be representative of what a typical parent would see when searching . Finally, a previous unpublished study examined online information available in 2007 . This study used only three search terms (vaccine, vaccination and immunisation), and various iterations of one search engine (http://www.google.com, http://www.google.com.au, http://www.google.co.uk, http://www.google.ie, www.google.co.za and www.google.co.nz). The results were tabulated and assessed as “pro- or anti-vaccine”, and the intended audiences given (parents or health care workers). None of these studies sought to assess both the quality and quantity of the information given by a web search undertaken by a typical parent.
This study aims to address this gap in knowledge by investigating the quantity and quality of general immunisation information available to Australian parents who search for it online.
We identified the immunisation themed search terms most commonly used by Australians using Google Trends . We compared the relative frequencies with which the terms “vaccine”, “vaccination”, “immunisation”, “immunization” and “childhood vaccines” occurred in Australian internet searches between 2007 and 2014 in order to design a search strategy that closely simulated Australian parents’ online search behaviour. In Australia, the terms ‘vaccine’, ‘vaccination’ and ‘immunisation’ occurred relatively frequently. We used the ‘related search terms’ function to identify search terms that most commonly occurred alongside these three search terms. As the scope of this study was to examine the quantity and quality of general vaccine information, search terms relating to specific vaccines (for example, “flu vaccine” or “whooping cough vaccine”) were removed from this list. The reasons for this decision were twofold: Firstly, research shows that the majority of Australian parents are “unquestioning acceptors” (30–40%) or “cautious acceptors” (25–35%) of vaccines . We therefore modelled our search on the general information this majority would look for on line in the first instance. Secondly, while we acknowledge that some hesitant parents may have concerns about specific vaccines, the majority of concerns held by parents are generally applicable across all vaccines , and therefore would be covered by a more general search. Furthermore, time and resource constraints meant that we could not include all of the individual vaccines on the Australian childhood schedule in our search terms. The resulting list of search terms is presented in Table 1.
We identified the most commonly used search engines based on a previous unpublished study  and the Search Engine List (an on-line resource reporting this information) . The search engines used were: Google.com.au; bing.com; yahoo.com; Ask.com; and Lycos.com.au. When options of searching the entire internet or Australian sites only were offered by the respective search engines, the Australian option was selected.
In order to minimise ‘filter bubble’ effect (the automatic personalisation of the search results based on previous browsing history), the researcher cleared the browser history prior to conducting the searches and did not sign in where this option was offered . All searches were conducted on a single day, March 5 2014. The final list of terms in Table 1 were entered, one at a time, into each of the five search engines listed above. The top ten results of each search were copied into a separate document as a record of the search.
The top ten web pages returned for each search term/search engine combination were catalogued in a Microsoft Excel spread sheet and duplicate entries removed. Sponsored and advertised sites were not included, as research has shown people conducting a targeted search, such as for immunisation information, tend to avoid advertising material [14, 15]. Web sites that did not include information about childhood vaccination, such as news listings and personal web pages were also excluded.
The web pages were then used as “entry points” to the broader web site on which they were found, and the web site assessed as a whole. Wikipedia and publicly available Facebook pages were assessed individually, as each page is written and edited by different groups or individuals (See Fig. 1).
To help guide people in how to identify good quality immunisation information online, the World Health Organization (WHO) Global Advisory Committee on Vaccine Safety (GACVS) developed “Good information practices for vaccine safety web sites” to “assist readers of websites in identifying those sites providing information on vaccine safety that comply with good information practices” . This resource includes a checklist of attributes a reader can look for to aid in their assessment of the quality of the web site.
We developed a 43-item quality assessment tool based on the WHO GACVS checklist criteria for assessing the quality of vaccine safety web sites which is divided into six thematic domains: Mission of site; Disclosure of ownership; Transparency of sponsorship; Accountability to users; Quality of information; and Quantity of information (See Table 2). Items in the checklist were not universally applicable to all of the websites under consideration. Therefore, each item was scored as present (1), absent (0), or not applicable. The total number of not applicable items was subtracted from the denominator (43) to give an adjusted denominator, the highest possible score for that site. A score out of 100 was then calculated by expressing the total number of attributes present divided by the adjusted denominator as a percentage (see Table 2). The web pages included in the audit were divided into two groups and assessed independently by two researchers who met weekly to resolve uncertainty to ensure consistency was maintained. The final score was made by consensus between the two researchers (KW and MS).
Web sites which placed a lot of emphasis on the dangers of vaccines, expressed doubt about their necessity, or gave advice that vaccines should be avoided were classed as “not supportive of vaccines”. Web sites which placed emphasis on the positive aspects of vaccines, and recommended their use were classed as “supportive of vaccines”.
The author’s informed subjective observations on the health literacy accessibility of pro-vaccine sites were then discussed, with a focus on how well the information would meet the needs of parents according to their position on vaccination as defined by Leask et al. .
Quantity of information
Seven hundred web pages were returned by the combined searches, of which, 514 were duplicates, leaving a total of 186 pages from 115 web sites. (Refer to Fig. 1). Of these, 40 web sites were found not to include immunisation information, referred to veterinary vaccines, or presented personal opinion about people involved in immunisation, and were therefore excluded, leaving 65 web sites and ten individual pages included in the quality scoring exercise. The ten individual pages were from sites such as Wikipedia where individual pages have different authors, and therefore required scoring separately.
Some sites were represented more often than others in the search results, because a number of the pages returned originated from the same site. Table 3 lists the 11 web sites for which more than four pages were returned by our search. The most commonly occurring web site was the American Centres for Disease Control and Prevention, followed by the Australian Government Department of Health’s Immunise Australia website, and various other government and independent research bodies.
Pro- and anti-vaccination stance
Of the total 75 web sites quality assessed, 65 (87%) were supportive of immunisation, while 10 (13%) were not supportive. To gain perspective on what parents saw at the top of their search results, we analysed the first web page returned by each search. This analysis revealed that of these 70 “top hits” 64 (91%) were supportive of immunisation. The remaining 6 pages (8%) were not supportive of vaccination, most of which were published by the Australian Vaccination Sceptics Network (the main anti-vaccine lobby group in Australia). When these “top hits” were viewed by search engine, Yahoo returned the highest number of pages which were not supportive of vaccination across the search terms, with three hits (all from the Australian Vaccination Sceptics Network). Google followed with two hits (both also from the Australian Vaccination Sceptics Network), while ASK returned one hit for a book on Amazon which offered alternatives to vaccination. Neither Lycos nor Bing returned pages not supportive of vaccination as a first hit for any of the search terms.
Origin of web sites
Of the final 75 pages scored originating from 65 unique websites, 42 were hosted in Australia, 25 in the United States, six in the United Kingdom, and one each in Ireland and Switzerland.
Quality of information
The overall scores indicate that the quality of vaccine information the web sites contained was variable. The mean overall quality score across all 75 scored web pages was 57/100, with a maximum of 92/100 (www.historyofvaccines.org), and a minimum of 14/100 (www.vaccine-side-effects.com) (Table 4).
When stratified by pro and anti-vaccination stance, the average overall score for sites which were “supportive of vaccines” was 61/100, while the average score for sites “not supportive of vaccines” was 30/100.
When the total quality scores were disaggregated into the individual domain scores (refer to Table 4), it became evident that on average, the sites were clear in their mission, and the content generally matched the stated mission (domain 1). It can also be seen that, on average, the sites scored relatively well for accountability to users (domain 4), providing adequate contact details for the site owner, and where interactive content was available, the appropriate disclaimers and declarations were given. The three domains with lower average scores were disclosure of ownership (domain 2), transparency of sponsorship (domain 3) and quality of information (domain 5). At an individual level, many sites were not clear on ownership or funding sources. From a quality perspective, many sites contained correct information; however they often did not go further than general statements such as “Immunisation is a simple, safe and effective way of protecting your child against harmful diseases that can cause serious health problems and sometimes death” . Even when more detail was given, it was often not well referenced (if at all), and where references were available, the published papers themselves often required payment or subscription, and thus were not readily accessible to the general public. Similarly, the identity of the author of the information was rarely given, nor were the qualifications of the author(s) or editorial board, where applicable. When the domain 5 (quality of information) scores are stratified into internet domain types (.co/.com,.gov,.edu,.org and.net/.int/.info),.org scored the highest, followed by.info/.net/.int,.edu, and.gov, while.com internet domains scored the lowest on average in this regard (refer to Fig. 2).
Understandability of the information for the general public
The authors’ general observations were that the pro-vaccine information returned by this search can be qualitatively divided into three broad content groups; those that gave a generalist overview without much detail; those that provided well-articulated and understandable detail; and those that gave lengthy and highly technical explanations.
The majority of sites appeared to belong to the first group. While the information was almost always technically correct, there was rarely any explanatory depth to the content. These sites typically contained unsupported statements such as “immunisation is safe and effective”. Many government web sites contained this level of information, as reflected in the comparatively lower average domain 5, “quality of information” scores for.gov internet domain types reported above. Sometimes these sites gave links to external sites, some of which fell into the second group, and (usually) the third.
The second group of sites was relatively small, and usually took the form of “fact sheets” that were often difficult to locate on the site. These sites contained technical information that was well-communicated, and more often referenced other sources than the first group, however they were rarely attributable to an author.
The third group of sites was also relatively small. They were often highly technical and written at a level suitable for someone with higher education in science or medicine. The information contained on these sites was often well referenced, usually citing peer-reviewed scientific publications; however access to these publications was, at best, limited.
To our knowledge this is the first published audit of general immunisation information available to Australian parents. This information will help inform the improvement of current resources, and the development of future web-based information for Australian parents seeking immunisation information.
Despite recent calls to move away from it , many still ascribe to the “deficit model” of immunisation information communication; it is assumed that parents who are hesitant about or refuse vaccinations for their children must not know or understand the science supporting it. Further to this, there is often a generally held perception that anti-vaccination messages are rife on line, and that parents searching online for information will be “lead astray” by this misinformation. Our findings suggest that when an unbiased search for information is undertaken, the majority (87%) of web sites are supportive of vaccination, suggesting that the volume of “pro” versus “anti” immunisation information is more in favour of “pro” than often assumed. However, most people don’t clear their browser history to ensure their web search is unbiased: Filter bubble occurs when an algorithm is used by the search engine, such as Google’s Personalized Search function, to return hits based not only on the relevance of the web site, but on the sites the user (or someone using the same browser) has visited previously. This will result in the search returning web sites that confirm or agree with the user’s previously browsed pages, potentially filtering out information that disagrees with what the user is likely to read. One study found that search results are altered if the browser history has not been cleared prior to searching by an estimated 11.7% . This phenomenon would only fuel what is normal human confirmation bias (the tendency to only read what confirms one’s beliefs to begin with). Similarly, google searches include an “autocomplete” function, whereby google suggests completed search phrases based on what is typed initially. These suggestions are predicted using what is currently being typed, previous searches the user has made (if logged into their google account), and what other people are searching for . This may also have an effect on what people are accessing. Possible ways to combat this is to communicate to parents (particularly those wanting “balanced” or unbiased information) to clear the browser history before undertaking a search for immunisation information, and not to use any suggested search terms.
From a quality perspective, most pro-vaccine sites contained the correct information, but it was often not well referenced (if at all), did not disclose by whom it was written, and did not give the qualifications of the author(s) or editorial board, where applicable. The majority of these sites did not go on to explain the information to any great depth, and while a relatively smaller number did go into more detail, it was either difficult to locate and not well attributed, or was very “science-heavy” and difficult to read.
It is widely recognised that parental vaccine acceptance is not a dichotomous schema of “acceptors” and “rejecters”, but a spectrum of acceptance behaviours . Using the Vaccine Communication Framework devised by Leask and colleagues  and informed by Benin et al. , the majority of parents can be classified as “unquestioning acceptors”. These people often don’t require information beyond that supplied by the majority of sites which were technically correct, without much explanatory depth. Other parents, the ones who fall between “cautious acceptors” and “hesitant late or selective” acceptors do require more information . Our findings suggest that, while these parents are arguably the ones with the greatest information needs, their needs may not be being met by what is currently available to them; the information is either too basic, or too technical, with little readily available between the two extremes (refer to Fig. 3). Our findings suggest that greater focus is warranted on developing resources that are tailored to these parent’s information needs, and include well-communicated, well-referenced and transparently authored information which is easily identified in an online search.
Finally, the information was often not easily identified on the web sites, with the immunisation information often “buried” among other components of the site. Thus, while the quality of the information contained on a given site is important, it is equally important that the information be easily found. This leads to a broader consideration of the “usability” of the information; the way in which the internet is used to communicate vaccine information is evolving. While in the past the web functioned primarily as information provision, this role has developed into a platform that is much more interactive, where user participation dictates the type of information needed and how it is used . In addition to being easily found, immunisation information now needs to be more interactive.
This study has limitations. The internet is not static, and search results will vary with time, as content is added, removed, or updated. These searches were undertaken in 2014, therefore the content of some of the assessed sites may have been updated, or there may now be other information available. For example, the GACVS criteria used to develop the scoring system was updated in September 2015, after we had completed scoring the sites identified in our search. As previously mentioned, this search can only be taken as indicative of one undertaken with a cleared browser history and any personalisation of the search engine disabled. Parents searching for information while signed into their Google account for example, will be subject to filter bubble, and will therefore come across a greater volume of information aligned with their previous searches. Finally, vaccine-specific search terms were not included in this study. Excluding these terms may have resulted in some web sites not being included in the assessment, however it is likely that the general search we conducted may have identified at least some of the same sites a more specific search would have. Future studies could be considered to assess more specific search terms. Our search, however does provide a useful snapshot of what information is generally available.
Australian parents, when searching for immunisation information are largely finding technically correct information which isn’t well attributed, without explanatory depth. Conversely there are some sites which provide well-referenced highly technical information, which is challenging for the general public to understand when assessed against usual health literacy conventions. Furthermore, this information is often not easily identified in the initial stages of searching. Our findings suggest a need for information which is easily found, transparently authored, well-referenced, and written in such a way that highly technical information is conveyed in an easily understood format informed by an understanding of parents’ needs and cognitions associated with interpreting health information.
Global advisory committee of vaccine safety
World health organization
Witteman HO, Zikmund-Fisher BJ. The defining characteristics of Web 2.0 and their potential influence in the online vaccination debate. Vaccine. 2012;30(25):3734–40.
Australian Bureau of Statistics. 8146.0 - Household Use of Information Technology, Australia, 2010–11 Canberra: Australian Government; 2011 [cited 2014 January 8]. Available from: http://www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/8146.0Main+Features12010-11?OpenDocument.
Roche M, Skinner D. How Parents Search, Interpret, and Evaluate Genetic Information Obtained from the Internet. J Genet Couns. 2009;18(2):119–29.
Khoo K, Bolt P, Babl FE, Jury S, Goldman RD. Health information seeking by parents in the Internet age. J Paediatr Child Health. 2008;44(7–8):419–23.
Betch C, Renkewitz F, Betsch T, Ulshöfer C. The Influence of Vaccine-critical Websites on Perceiving Vaccination Risks. J Health Psychol. 2010;15(3):446–55.
Wolfe RM, Sharp LK. Vaccination or Immunization? The Impact of Search Terms on the Internet. J Health Commun. 2005;10(6):537–51.
Sak G, Diviani N, Allam A, Schulz PJ. Comparing the quality of pro- and anti-vaccination online information: a content analysis of vaccination-related webpages. BMC Public Health. 2016;16(1):38.
Anderson J. Development and Revision of Fact sheets for the National Centre for Immunisation Research and Surveillance of Vaccine Preventable Diseases: A project for submission of honours (MBBS). Sydney: University of Sydney; 2007.
Google. Google Trends 2016 [cited 2014]. Available from: https://www.google.com.au/trends/.
Leask J, Kinnersley P, Jackson C, Cheater F, Bedford H, Rowles G. Communicating with parents about vaccination: a framework for health professionals. BMC Pediatr. 2012;12(1):154.
Kennedy AM, LaVail K, Nowak G, Basket M, Landry S. Confidence about vaccines in the United States: understanding parents' perceptions. Health Aff (Millwood). 2011;30:1151–9
The Search Engine List 2010 [cited 2014 February 12]. Available from: http://www.thesearchenginelist.com/.
Hannak A, Sapiezynski P, Molav Kakhki A, Krishnamurthy B, Lazer d, Mislove A, et al. Measuring personalization of web search. International World Wide Web Conference; 13–17 May, 2013; Rio de Janeiro, Brazil: International World Wide Web Conference Committee; 2013.
Chang-Hoan C, Cheon HJ. WHY DO PEOPLE AVOID ADVERTISING ON THE INTERNET? Journal of Advertising. Winter. 2004;33(4):89–97. English.
Owens JW, Chaparro BS, Palmer EM. Text Advertising Blindness: The New Banner Blindness? J Usability Studies. 2011;6(3):172–97.
Global Advisory Committee on Vaccine Safety. Good information practices for vaccine safety web sites Geneva: World Health Organization 2014 [cited 2014 Feb 22]. Available from: http://www.who.int/vaccine_safety/good_vs_sites/en/.
Department of Human Services. Australian Childhood Immunisation Register Canberra: Australian Government; 2016 [cited 2016 10/3]. Available from: https://www.humanservices.gov.au/customer/services/medicare/australian-childhood-immunisation-register.
Leask J. Target the fence-sitters. Nature. 2011;473(7348):443–5.
Google. Search using autocomplete California: Google; 2016 [cited 2016 November 28]. Available from: https://support.google.com/websearch/answer/106230?hl=en.
Gust D, Brown C, Sheedy K, Hibbs B, Weaver D. G. N. Immunization Attitudes and Beliefs Among Parents: Beyond a Dichotomous Perspective. Am J Health Behav. 2005;29(1):81–92.
Benin AL, Wisler-Scher DJ, Colson E, Shapiro ED, Holmboe ES. Qualitative analysis of mothers' decision-making about vaccines for infants: the importance of trust. Pediatrics. 2006;117(5):1532–41.
This study was funded by the National Centre for Immunisation Research & Surveillance, Australia.
Availability of data and materials
The datasets during analysed during the current study are available from the corresponding author on reasonable request, with note that online content may have changed since the study was conducted. Search methods and scoring criteria / tools used to generate the scores on which conclusions were based are available in the manuscript.
KW conceived the study, ran the searches, developed the quality assessment tool and method, contributed to the quality assessment, analysed the data and drafted the manuscript. MS contributed to the quality assessment, data analysis and manuscript. NB contributed to the development of the quality assessment tool and method, and contributed to the data interpretation and manuscript. JL contributed to the development of the quality assessment tool and method, interpretation of findings and manuscript development. All authors read and approved the final manuscript.
KW receives funding from the National Centre for Immunisation Research and Surveillance, and works on projects for the Australian Government Department of Health and the New South Wales Department of Health. NB works on a vaccination communication research project funded by the Australian Government Department of Health and has received reimbursement for consultancy (unrelated to vaccination) from the World Health Organization. JL has received funding from New South Wales Health and the Australian Government Department of Health and has received reimbursement for consultancy from the World Health Organization. MS has no conflicts of interest to declare.
Consent for publication
Ethics approval and consent to participate
Not applicable to this study.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Wiley, K.E., Steffens, M., Berry, N. et al. An audit of the quality of online immunisation information available to Australian parents. BMC Public Health 17, 76 (2017). https://doi.org/10.1186/s12889-016-3933-9
- Health information
- Internet search
- Online information quality