An audit of the quality of online immunisation information available to Australian parents
© The Author(s). 2017
Received: 24 August 2016
Accepted: 12 December 2016
Published: 13 January 2017
The Internet is increasingly a source of health information for parents, who use the Internet alongside health care providers for immunisation information. Concerns have been raised about the reliability of online immunisation information, however to date there has been no audit of the quality or quantity of what is available to Australian parents. The objective of this study was to address this gap by simulating a general online search for immunisation information, and assessing the quality and quantity of the web sites returned by the search.
We used Google trends to identify the most common immunisation search terms used in Australia. The ten most common terms were entered into five search engines and the first ten non-commercial results from each search collated. A quality assessment tool was developed using the World Health Organization Global Advisory Committee on Vaccine Safety (GACVS) criteria for assessing the quality of vaccine safety web sites, and used to assess and score the quality of the sites.
Seven hundred web pages were identified, of which 514 were duplicates, leaving 186 pages from 115 web sites which were audited. Forty sites did not include human immunisation information, or presented personal opinion about individuals, and were not scored. Of the 75 sites quality scored, 65 (87%) were supportive of immunisation, while 10 (13%) were not supportive. The overall mean quality score was 57/100 (range 14/100 to 92/100). When stratified by pro and anti-vaccination stance, the average quality score for pro-vaccine sites was 61/100, while the average score for anti-vaccine sites was 30/100.
Pro-vaccine information could be divided into three content groups: generalist overview with little detail; well-articulated and understandable detail; and lengthy and highly technical explanations. The main area found to be lacking in pro-vaccine sites was lack of transparent authorship.
Our findings suggest a need for information which is easily found, transparently authored, well-referenced, and written in a way that is easily understood.
KeywordsImmunisation Health information Internet search Online information quality
The internet has changed the way people find health information . Recent figures show that, of the 13.3 million Australians who access the Internet at home, research activities were among the most commonly reported undertakings . Increasingly, parents are using Internet-sourced information in addition to traditional consultations with health care providers to improve their understanding of children’s health-related issues . One study estimates 43% of Australian parents access child health information on the Internet .
Negative online messages are thought to undermine parents’ confidence in vaccines , leading government and health care providers to raise concerns about the reliability of health information available on the internet. However, no comprehensive assessment of the quantity or quality of online general immunisation information available to Australian parents has been conducted. A previous study by Wolfe and Sharp assessed the pro- or anti-vaccine stance of parental internet search results for immunisation information overseas  and found that the search terms and search engine used had a bearing on the amount of pro- or anti- vaccination material returned by the search, and the order in which they were presented. Another study by Sak et al. compared the quality of pro- and anti-vaccine web sites, but the sample was not intended to be representative of what a typical parent would see when searching . Finally, a previous unpublished study examined online information available in 2007 . This study used only three search terms (vaccine, vaccination and immunisation), and various iterations of one search engine (http://www.google.com, http://www.google.com.au, http://www.google.co.uk, http://www.google.ie, www.google.co.za and www.google.co.nz). The results were tabulated and assessed as “pro- or anti-vaccine”, and the intended audiences given (parents or health care workers). None of these studies sought to assess both the quality and quantity of the information given by a web search undertaken by a typical parent.
This study aims to address this gap in knowledge by investigating the quantity and quality of general immunisation information available to Australian parents who search for it online.
List of search terms entered into search engines to assess what information is returned when Australian healthcare consumers search for generic immunisation information
Primary generic search term
List of related search terms returned by Google Trends
Final selected list of generic search terms examined
Vaccine side effects
Vaccine side effects
We identified the most commonly used search engines based on a previous unpublished study  and the Search Engine List (an on-line resource reporting this information) . The search engines used were: Google.com.au; bing.com; yahoo.com; Ask.com; and Lycos.com.au. When options of searching the entire internet or Australian sites only were offered by the respective search engines, the Australian option was selected.
In order to minimise ‘filter bubble’ effect (the automatic personalisation of the search results based on previous browsing history), the researcher cleared the browser history prior to conducting the searches and did not sign in where this option was offered . All searches were conducted on a single day, March 5 2014. The final list of terms in Table 1 were entered, one at a time, into each of the five search engines listed above. The top ten results of each search were copied into a separate document as a record of the search.
The top ten web pages returned for each search term/search engine combination were catalogued in a Microsoft Excel spread sheet and duplicate entries removed. Sponsored and advertised sites were not included, as research has shown people conducting a targeted search, such as for immunisation information, tend to avoid advertising material [14, 15]. Web sites that did not include information about childhood vaccination, such as news listings and personal web pages were also excluded.
To help guide people in how to identify good quality immunisation information online, the World Health Organization (WHO) Global Advisory Committee on Vaccine Safety (GACVS) developed “Good information practices for vaccine safety web sites” to “assist readers of websites in identifying those sites providing information on vaccine safety that comply with good information practices” . This resource includes a checklist of attributes a reader can look for to aid in their assessment of the quality of the web site.
Web site quality score criteria
WHO GAVCS Attribute
1. Mission of site
Purpose of the website stated.
Intended audience defined.
Site provides a mission statement.
The content of the site matches statement.
2. Disclosure of ownership/source
Individual or organization name on every page?
Physical address on every page?
Electronic address on every page?
Qualifications/credentials of organization or individual site owner specified
Type of organization made clear (e.g., government, non-profit, commercial)
Organization or individual's affiliations and alliances and disclosure of any relationship that might influence the content of the site.
Editorial Board, Advisory Board, or Board of Directors members listed with credentials.
3. Transparency of sponsorship
Disclosure of all sources of funding for organization/website (grants, sponsors, advertisers, fees, personal).
Disclosure of any relevant personal or financial associations that might be considered a potential conflict of interest.
If advertising is a source of funding, this should be clearly stated
A brief description of the owner's advertising policy is included.
Content intended to promote or sell a product or service should be clearly distinguished from the educational and scientific content.
4. Accountability to Users
Multiple methods of contacting the owner of the site (e-mail address, electronic form, mail, phone, fax) must be available from the home page.
Multiple methods of contacting the owner of the site (e-mail address, electronic form, mail, phone, fax) must be easily accessible from other pages of the site.
A site offering interactive exchanges (e.g., chat room, medical advice) provides information about the moderator or clinician's expertise and affiliations.
A site offering interactive exchanges (e.g., chat room, medical advice) provides information about the moderator or clinician's source of compensation
A site offering interactive exchanges (e.g., chat room, medical advice) provides a disclaimer that all posted information may not be accurate.
5. Quality of information
Authority of sources: Clear statement of source for all information, including 1author's name
Authority of sources: Clear statement on the sources of information, including Author's credentials and / or affiliations
Authority of sources: Clear statement on the sources of information, including Author's financial disclosure or potential conflict of interest
Description of any “seal of approval” or award the website has been granted.
Attribution: All information supported by citations to source resources with hypertext links if available.
The site should indicate whether information is based on scientific studies, expert consensus, professional opinion, or personal experience or opinion.
Accuracy: The information presented should be based on objective, scientific research.
The site should identify the evidence that supports a position including references to published studies and reference works.
Currency: The date that content was first developed.
The date of last update or modification should be clearly indicated on each piece. The date the whole site was updated or the copyright date is not adequate.
Site does not contain out-dated information
Review process: Statement of procedure used for selection of site content.
Includes a guarantee of the independence of the editorial process.
Includes the names and credentials of the Editorial Board.
Standards of writing/editing: Writing on the site should be professional, with proper grammar, spelling, and composition.
Completeness: Includes the comprehensiveness of a resource, including the breadth and depth of coverage.
6. Quantity of information
Includes the retrospective coverage (archived items).
Includes the balance of the information presented, such as admitting when an issue is controversial and including all reasonable sides in a fair way.
Provision of links to other resources: Offers hypertext links to other resources.
Indicate whether links to other sites are informational only or if such links imply endorsement.
Any links are carefully selected and their content is accurate and credible.
Any links are carefully selected and their content is current.
Web sites which placed a lot of emphasis on the dangers of vaccines, expressed doubt about their necessity, or gave advice that vaccines should be avoided were classed as “not supportive of vaccines”. Web sites which placed emphasis on the positive aspects of vaccines, and recommended their use were classed as “supportive of vaccines”.
The author’s informed subjective observations on the health literacy accessibility of pro-vaccine sites were then discussed, with a focus on how well the information would meet the needs of parents according to their position on vaccination as defined by Leask et al. .
Quantity of information
Seven hundred web pages were returned by the combined searches, of which, 514 were duplicates, leaving a total of 186 pages from 115 web sites. (Refer to Fig. 1). Of these, 40 web sites were found not to include immunisation information, referred to veterinary vaccines, or presented personal opinion about people involved in immunisation, and were therefore excluded, leaving 65 web sites and ten individual pages included in the quality scoring exercise. The ten individual pages were from sites such as Wikipedia where individual pages have different authors, and therefore required scoring separately.
Web Sites for which more than one page was identified by the search
Web Site Title
Web site address
Number of unique pages identified by the search
Centres for Disease Control and Prevention (USA)
Immunise Australia Program (Australian Federal Government Department of Health)
Australian Government Department of Health
National Centre for Immunisation Research and Surveillance (Australia)
New South Wales Health (NSW Government, Australia)
Pro- and anti-vaccination stance
Of the total 75 web sites quality assessed, 65 (87%) were supportive of immunisation, while 10 (13%) were not supportive. To gain perspective on what parents saw at the top of their search results, we analysed the first web page returned by each search. This analysis revealed that of these 70 “top hits” 64 (91%) were supportive of immunisation. The remaining 6 pages (8%) were not supportive of vaccination, most of which were published by the Australian Vaccination Sceptics Network (the main anti-vaccine lobby group in Australia). When these “top hits” were viewed by search engine, Yahoo returned the highest number of pages which were not supportive of vaccination across the search terms, with three hits (all from the Australian Vaccination Sceptics Network). Google followed with two hits (both also from the Australian Vaccination Sceptics Network), while ASK returned one hit for a book on Amazon which offered alternatives to vaccination. Neither Lycos nor Bing returned pages not supportive of vaccination as a first hit for any of the search terms.
Origin of web sites
Of the final 75 pages scored originating from 65 unique websites, 42 were hosted in Australia, 25 in the United States, six in the United Kingdom, and one each in Ireland and Switzerland.
Quality of information
Overall mean quality score, and mean domain scores of audited web sites on immunisation
Mean standardised quality score* of individual domains of scoring system
Overall standardised score** (n/100)
1. Mission of Site (n/100)
2. Disclosure of ownership (n/100)
3. Transparency of Sponsorship (n/100)
4. Accountability to users (n/100)
5. Quality of information (n/100)
6. Quantity of information (n/100)
When stratified by pro and anti-vaccination stance, the average overall score for sites which were “supportive of vaccines” was 61/100, while the average score for sites “not supportive of vaccines” was 30/100.
Understandability of the information for the general public
The authors’ general observations were that the pro-vaccine information returned by this search can be qualitatively divided into three broad content groups; those that gave a generalist overview without much detail; those that provided well-articulated and understandable detail; and those that gave lengthy and highly technical explanations.
The majority of sites appeared to belong to the first group. While the information was almost always technically correct, there was rarely any explanatory depth to the content. These sites typically contained unsupported statements such as “immunisation is safe and effective”. Many government web sites contained this level of information, as reflected in the comparatively lower average domain 5, “quality of information” scores for.gov internet domain types reported above. Sometimes these sites gave links to external sites, some of which fell into the second group, and (usually) the third.
The second group of sites was relatively small, and usually took the form of “fact sheets” that were often difficult to locate on the site. These sites contained technical information that was well-communicated, and more often referenced other sources than the first group, however they were rarely attributable to an author.
The third group of sites was also relatively small. They were often highly technical and written at a level suitable for someone with higher education in science or medicine. The information contained on these sites was often well referenced, usually citing peer-reviewed scientific publications; however access to these publications was, at best, limited.
To our knowledge this is the first published audit of general immunisation information available to Australian parents. This information will help inform the improvement of current resources, and the development of future web-based information for Australian parents seeking immunisation information.
Despite recent calls to move away from it , many still ascribe to the “deficit model” of immunisation information communication; it is assumed that parents who are hesitant about or refuse vaccinations for their children must not know or understand the science supporting it. Further to this, there is often a generally held perception that anti-vaccination messages are rife on line, and that parents searching online for information will be “lead astray” by this misinformation. Our findings suggest that when an unbiased search for information is undertaken, the majority (87%) of web sites are supportive of vaccination, suggesting that the volume of “pro” versus “anti” immunisation information is more in favour of “pro” than often assumed. However, most people don’t clear their browser history to ensure their web search is unbiased: Filter bubble occurs when an algorithm is used by the search engine, such as Google’s Personalized Search function, to return hits based not only on the relevance of the web site, but on the sites the user (or someone using the same browser) has visited previously. This will result in the search returning web sites that confirm or agree with the user’s previously browsed pages, potentially filtering out information that disagrees with what the user is likely to read. One study found that search results are altered if the browser history has not been cleared prior to searching by an estimated 11.7% . This phenomenon would only fuel what is normal human confirmation bias (the tendency to only read what confirms one’s beliefs to begin with). Similarly, google searches include an “autocomplete” function, whereby google suggests completed search phrases based on what is typed initially. These suggestions are predicted using what is currently being typed, previous searches the user has made (if logged into their google account), and what other people are searching for . This may also have an effect on what people are accessing. Possible ways to combat this is to communicate to parents (particularly those wanting “balanced” or unbiased information) to clear the browser history before undertaking a search for immunisation information, and not to use any suggested search terms.
From a quality perspective, most pro-vaccine sites contained the correct information, but it was often not well referenced (if at all), did not disclose by whom it was written, and did not give the qualifications of the author(s) or editorial board, where applicable. The majority of these sites did not go on to explain the information to any great depth, and while a relatively smaller number did go into more detail, it was either difficult to locate and not well attributed, or was very “science-heavy” and difficult to read.
Finally, the information was often not easily identified on the web sites, with the immunisation information often “buried” among other components of the site. Thus, while the quality of the information contained on a given site is important, it is equally important that the information be easily found. This leads to a broader consideration of the “usability” of the information; the way in which the internet is used to communicate vaccine information is evolving. While in the past the web functioned primarily as information provision, this role has developed into a platform that is much more interactive, where user participation dictates the type of information needed and how it is used . In addition to being easily found, immunisation information now needs to be more interactive.
This study has limitations. The internet is not static, and search results will vary with time, as content is added, removed, or updated. These searches were undertaken in 2014, therefore the content of some of the assessed sites may have been updated, or there may now be other information available. For example, the GACVS criteria used to develop the scoring system was updated in September 2015, after we had completed scoring the sites identified in our search. As previously mentioned, this search can only be taken as indicative of one undertaken with a cleared browser history and any personalisation of the search engine disabled. Parents searching for information while signed into their Google account for example, will be subject to filter bubble, and will therefore come across a greater volume of information aligned with their previous searches. Finally, vaccine-specific search terms were not included in this study. Excluding these terms may have resulted in some web sites not being included in the assessment, however it is likely that the general search we conducted may have identified at least some of the same sites a more specific search would have. Future studies could be considered to assess more specific search terms. Our search, however does provide a useful snapshot of what information is generally available.
Australian parents, when searching for immunisation information are largely finding technically correct information which isn’t well attributed, without explanatory depth. Conversely there are some sites which provide well-referenced highly technical information, which is challenging for the general public to understand when assessed against usual health literacy conventions. Furthermore, this information is often not easily identified in the initial stages of searching. Our findings suggest a need for information which is easily found, transparently authored, well-referenced, and written in such a way that highly technical information is conveyed in an easily understood format informed by an understanding of parents’ needs and cognitions associated with interpreting health information.
Global advisory committee of vaccine safety
World health organization
This study was funded by the National Centre for Immunisation Research & Surveillance, Australia.
Availability of data and materials
The datasets during analysed during the current study are available from the corresponding author on reasonable request, with note that online content may have changed since the study was conducted. Search methods and scoring criteria / tools used to generate the scores on which conclusions were based are available in the manuscript.
KW conceived the study, ran the searches, developed the quality assessment tool and method, contributed to the quality assessment, analysed the data and drafted the manuscript. MS contributed to the quality assessment, data analysis and manuscript. NB contributed to the development of the quality assessment tool and method, and contributed to the data interpretation and manuscript. JL contributed to the development of the quality assessment tool and method, interpretation of findings and manuscript development. All authors read and approved the final manuscript.
KW receives funding from the National Centre for Immunisation Research and Surveillance, and works on projects for the Australian Government Department of Health and the New South Wales Department of Health. NB works on a vaccination communication research project funded by the Australian Government Department of Health and has received reimbursement for consultancy (unrelated to vaccination) from the World Health Organization. JL has received funding from New South Wales Health and the Australian Government Department of Health and has received reimbursement for consultancy from the World Health Organization. MS has no conflicts of interest to declare.
Consent for publication
Ethics approval and consent to participate
Not applicable to this study.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Witteman HO, Zikmund-Fisher BJ. The defining characteristics of Web 2.0 and their potential influence in the online vaccination debate. Vaccine. 2012;30(25):3734–40.View ArticlePubMedGoogle Scholar
- Australian Bureau of Statistics. 8146.0 - Household Use of Information Technology, Australia, 2010–11 Canberra: Australian Government; 2011 [cited 2014 January 8]. Available from: http://www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/8146.0Main+Features12010-11?OpenDocument.
- Roche M, Skinner D. How Parents Search, Interpret, and Evaluate Genetic Information Obtained from the Internet. J Genet Couns. 2009;18(2):119–29.View ArticlePubMedGoogle Scholar
- Khoo K, Bolt P, Babl FE, Jury S, Goldman RD. Health information seeking by parents in the Internet age. J Paediatr Child Health. 2008;44(7–8):419–23.View ArticlePubMedGoogle Scholar
- Betch C, Renkewitz F, Betsch T, Ulshöfer C. The Influence of Vaccine-critical Websites on Perceiving Vaccination Risks. J Health Psychol. 2010;15(3):446–55.View ArticleGoogle Scholar
- Wolfe RM, Sharp LK. Vaccination or Immunization? The Impact of Search Terms on the Internet. J Health Commun. 2005;10(6):537–51.View ArticlePubMedGoogle Scholar
- Sak G, Diviani N, Allam A, Schulz PJ. Comparing the quality of pro- and anti-vaccination online information: a content analysis of vaccination-related webpages. BMC Public Health. 2016;16(1):38.View ArticlePubMedPubMed CentralGoogle Scholar
- Anderson J. Development and Revision of Fact sheets for the National Centre for Immunisation Research and Surveillance of Vaccine Preventable Diseases: A project for submission of honours (MBBS). Sydney: University of Sydney; 2007.
- Google. Google Trends 2016 [cited 2014]. Available from: https://www.google.com.au/trends/.
- Leask J, Kinnersley P, Jackson C, Cheater F, Bedford H, Rowles G. Communicating with parents about vaccination: a framework for health professionals. BMC Pediatr. 2012;12(1):154.View ArticlePubMedPubMed CentralGoogle Scholar
- Kennedy AM, LaVail K, Nowak G, Basket M, Landry S. Confidence about vaccines in the United States: understanding parents' perceptions. Health Aff (Millwood). 2011;30:1151–9
- The Search Engine List 2010 [cited 2014 February 12]. Available from: http://www.thesearchenginelist.com/.
- Hannak A, Sapiezynski P, Molav Kakhki A, Krishnamurthy B, Lazer d, Mislove A, et al. Measuring personalization of web search. International World Wide Web Conference; 13–17 May, 2013; Rio de Janeiro, Brazil: International World Wide Web Conference Committee; 2013.
- Chang-Hoan C, Cheon HJ. WHY DO PEOPLE AVOID ADVERTISING ON THE INTERNET? Journal of Advertising. Winter. 2004;33(4):89–97. English.Google Scholar
- Owens JW, Chaparro BS, Palmer EM. Text Advertising Blindness: The New Banner Blindness? J Usability Studies. 2011;6(3):172–97.Google Scholar
- Global Advisory Committee on Vaccine Safety. Good information practices for vaccine safety web sites Geneva: World Health Organization 2014 [cited 2014 Feb 22]. Available from: http://www.who.int/vaccine_safety/good_vs_sites/en/.
- Department of Human Services. Australian Childhood Immunisation Register Canberra: Australian Government; 2016 [cited 2016 10/3]. Available from: https://www.humanservices.gov.au/customer/services/medicare/australian-childhood-immunisation-register.
- Leask J. Target the fence-sitters. Nature. 2011;473(7348):443–5.View ArticlePubMedGoogle Scholar
- Google. Search using autocomplete California: Google; 2016 [cited 2016 November 28]. Available from: https://support.google.com/websearch/answer/106230?hl=en.
- Gust D, Brown C, Sheedy K, Hibbs B, Weaver D. G. N. Immunization Attitudes and Beliefs Among Parents: Beyond a Dichotomous Perspective. Am J Health Behav. 2005;29(1):81–92.View ArticlePubMedGoogle Scholar
- Benin AL, Wisler-Scher DJ, Colson E, Shapiro ED, Holmboe ES. Qualitative analysis of mothers' decision-making about vaccines for infants: the importance of trust. Pediatrics. 2006;117(5):1532–41.