Skip to main content

Development of a comprehensive list of criteria for evaluating consumer education materials on colorectal cancer screening



Appropriate patient information materials may support the consumer’s decision to attend or not to attend colorectal cancer (CRC) screening tests (fecal occult blood test and screening colonoscopy). The aim of this study was to develop a list of criteria to assess whether written health information materials on CRC screening provide balanced, unbiased, quantified, understandable, and evidence-based health information (EBHI) about CRC and CRC screening.


The list of criteria was developed based on recommendations and assessment tools for health information in the following steps: (1) Systematic literature search in 13 electronic databases (search period: 2000–2010) and completed by an Internet search (2) Extraction of identified criteria (3) Grouping of criteria into categories and domains (4) Compilation of a manual of adequate answers derived from systematic reviews and S3 guidelines (5) Review by external experts (6) Modification (7) Final discussion with external experts.


Thirty-one publications on health information tools and recommendations were identified. The final list of criteria includes a total of 230 single criteria in three generic domains (formal issues, presentation and understandability, and neutrality and balance) and one CRC-specific domain. A multi-dimensional rating approach was used whenever appropriate (e.g., rating for the presence, correctness, presentation and level of evidence of information). Free text input was allowed to ensure the transparency of assessment. The answer manual proved to be essential to the rating process. Quantitative analyses can be made depending on the level and dimensions of criteria.


This comprehensive list of criteria clearly has a wider range of evaluation than previous assessment tools. It is not intended as a final quality assessment tool, but as a first step toward thorough evaluation of specific information materials for their adherence to EBHI requirements. This criteria list may also be used to revise leaflets and to develop evidence-based health information on CRC screening. After adjustment for different procedure-specific criteria, the list of criteria can also be applied to other cancer screening procedures.

Peer Review reports


Since the 70s, cancer screening procedures have gained increasing significance for public health and are promoted in many countries. However, in recent years, it became clear that the communication of some procedures may have overemphasized their benefits and disregarded their risks [1]. Potential harms of cancer screening include adverse effects from the procedure itself, overdiagnosis, and false-positive results, including the mental stress and/or unnecessary diagnostic tests resulting there from [2]. For this reason and for ethical reasons that apply to all medical procedures with potential side effects, participants must give their informed consent before screening [3, 4]. In Germany, persons aged 50 and older who have statutory health insurance have free access to colorectal cancer (CRC) screening tests, including the fecal occult blood test (FOBT) and (since 2002) screening colonoscopy. As colonoscopy is an invasive and burdensome procedure with potentially lethal complications (very rare) [5, 6], special efforts are needed to ensure informed decision-making [3].

Like other health communication strategies, written health information materials support informed choices regarding whether or not to attend CRC screening and have certain tests performed [7, 8]. Appropriate information should meet evidence-based health information (EBHI) standards [9]. Accordingly, it must include balanced, unbiased, quantified, understandable, and evidence-based information about CRC and the potential benefits and harms of the screening procedures [10]. Numerous tools to evaluate the quality of health-related information are available. However, we found no tool or checklist that systematically evaluates health information on cancer screening procedures according to EBHI standards. The existing tools focus on criteria for the characterization of structural and process quality (e.g., DISCERN [11]), self-assessment of lay persons (e.g., Check-In [12]), or a reviewer’s subjective judgment of specific benefits and harms [13]. Collected information about structural and process quality is typically used as a surrogate marker for parameters of outcome quality. For example, editorial independence stands for balanced information or clarity, and layout for understandability. None of the generic tools, including the International Patient Decision Aid Standards Instrument (IPDASi) [14], directly evaluates reliability and understandability – two important features of EBHI. Thus, false or biased information may be rated as appropriate just because formal standards are met.

In Germany, many different players in healthcare provide information about CRC screening. These include governmental organizations, foundations, healthcare providers, and health insurance companies. Presumably, not all of the existing information meets EBHI standards, but rather depicts only the benefits without harms and/or strongly encourages participation in screening [15]. The German National Cancer Plan [16] was initiated by the Federal Ministry of Health in 2008 to develop and improve cancer screening and care of cancer patients. One aim is to enhance consumer information materials on the benefits and risks of screening procedures to support informed decision-making regarding whether to attend screening or not. In this context, the Federal Ministry of Health initiated a project on CRC screening to identify consumer education materials in conformity with EBHI standards. Initially, we developed a list of criteria that helps experts systematically assess whether the available flyers and brochures provide reliable, correct, understandable and unbiased information on CRC screening. The underlying concepts and methods of the development process as well as the resulting list of criteria and its strengths and weaknesses are presented in this article. Rating examples are provided to illustrate the application of this instrument.


Study design

The whole research project included steps 1) to identify consumer education materials on CRC 2) to develop an assessment tool for experts’ use 3) to assess the identified materials from experts’ view, and 4) to assess the materials from consumers’ view. The study protocol was approved by the ethics committee of Hannover Medical School (Application No. 1803–2013). In this article we focus on the development of the assessment tool.

The main goal was to produce a comprehensive list of criteria based on EBHI requirements for detailed assessment of the contents and correctness of health information on CRC screening and to make such assessments as objective as possible. Criteria were extracted from recommendations on EBHI and supplemented with criteria from previous health information assessment tools. The extracted criteria were sorted and categories and subcategories were defined. The list of criteria was developed in the following steps (Figure 1):

  1. (1)

    Systematic literature search to identify recommendations and assessment tools for health information

  2. (2)

    Extraction of the identified criteria

  3. (3)

    Grouping of the criteria into categories and a comprehensive list of criteria

  4. (4)

    Review of the list of criteria by external experts

  5. (5)

    Modification of the list of criteria

  6. (6)

    Creation of an answer manual

  7. (7)

    Discussion with external experts

Figure 1

Study design used to develop the list of criteria and answer manual.

These steps are described in detail in the following sections. As shown in Figure 1, we also rated German flyers and brochures on CRC screening with the developed list of criteria. The results are not shown, but rating examples from that part of the project will be used to illustrate how to apply the instrument.

Methodological considerations

The derived a priori methodological considerations for the list of criteria were as follows:

  1. (1)

    Criteria will not ask for aggregated information, if possible. For example, instead of asking for “any” adverse effects, it will ask in detail about single adverse effects like bleeding, pain, and perforation in order to prevent the reviewer from assessing combined information.

  2. (2)

    If possible, there will be no multi-level response options (e.g., Likert items), but rather “yes”, “no” or “unclear”. Rationale: The goal is not to obtain levels of agreement or disagreement, as with psychometric tools, but unambiguous statements.

  3. (3)

    The direction of the response options shall be adjusted in such a way that “yes” always corresponds to a rating of “adequate” or “appropriate”.

  4. (4)

    No numerical rating or sum score will be used (no use of scales yielding a summary score). Rationale: Summary scores imply an implicit weighting of the criteria that is not evidence-based but arbitrary.

  5. (5)

    Except for formal issues (specification of authors, publication date, etc.), each reviewer shall not only document whether specific information was reported but also whether it was correct, (e.g., whether the risk of bleeding was reported, and whether it was reported correctly).

  6. (6)

    Reporting about evidence levels or non-sufficient evidence will be recorded.

  7. (7)

    Each rating will be accompanied by the corresponding quotation from the information source to ensure the transparency of assessment.

  8. (8)

    A detailed, evidence-based answer manual will be developed and implemented in order to achieve consistent ratings.

  9. (9)

    Assessment of the information material will be carried out by two independent reviewers, and any discrepancies will be resolved by consensus with a third reviewer.

Systematic literature search

A comprehensive systematic literature search was carried out in 13 electronic databases, including EMBASE and Medline. The search included health information recommendations and assessment tools for information on the underlying disease. It was restricted to articles in English or German published from 1/2000 to 8/2010. The search strategy involved the following keyword combinations and their German translations: “criteria”, “quality”, “quality criteria”, “checklist”, “evaluation” or “assessment” combined with the terms ”decision support for patient-informed decision-making”, “patient information”, “shared decision/decision making”, “risk communication”, “health information”, “evidence-based patient information” and “information brochures”. This search yield a total of 3,097 documents that were stepwise selected on the level of title, abstract and full text. A manual search was carried out based on the references in the identified publications. Additionally, a web-based search was performed with the Google search engine using combinations of keywords similar to those used in the database research. The first 50 results of each of the 19 search terms were evaluated for appropriateness. The detailed strategy used for the database search and the web-based search is listed in the Appendix (see Additional file 1). Two independent reviewers screened and selected the articles.

Extraction and categorization of criteria

From the identified documents, two researchers (MD, BB) extracted and grouped criteria on formal issues, CRC, CRC screening procedures, neutrality and balance, while two others (GS, IK) extracted criteria on presentation and understandability. The resulting list of criteria was approved by the whole project team. These researchers also applied the list of criteria in step 3 of this project that is not within the scope of this article.

Answer manual

A manual providing the correct answers for each criterion was developed. The aim was to minimize the subjectivity of ratings and achieve clear and unambiguous assessments. Whenever possible, the correct answers were derived from selective literature searches focusing on evidence from systematic reviews, HTA reports or S3 guidelines. Evidence levels were assigned according to the Oxford Centre of Evidence-based Medicine [17].

Review by external experts

Twelve external experts in the field of (colorectal) cancer screening, including patient representatives and staff from governmental health-related institutions, cancer research institutes, providers of healthcare services, and statutory health insurances were asked to review the preliminary list of criteria. Six experts responded, providing feedback in a telephone interview with two researchers. Their comments were recorded instantly. Modifications proposed by the experts were discussed within the project team and, if approved, implemented into the list of criteria. The final list of criteria was discussed in a meeting with the experts.


Development of the list of criteria

Fifteen documents with recommendations and 16 with assessment tools for health information were identified. Among the recommendations, n = 2 referred to cancer screening [16, 18], n = 1 to screening [19], and n = 1 to orthopedic interventions [20]; n = 11 had no special focus [4, 9, 2129]. Among the assessment tools, n = 1 referred to colorectal cancer screening [15], n = 1 to diagnostic breast tests [30], n = 3 refer to mammography screening [3133], and n = 2 to patient decision aids [14, 34], and n = 7 had no special focus [11, 12, 3541]. Criteria for assessing health information were systematically extracted from these documents, and the single criteria were grouped into seven categories (Table 1): formal issues, information on CRC screening, information on screening colonoscopy, information on the fecal occult blood test, readability/comprehensibility, layout and neutrality and balance. These categories were further aggregated into four domains, one representing CRC-specific content issues and three describing generic issues applicable to different cancer screening procedures.

Table 1 Content structure of the list of criteria for evaluating consumer information materials on colorectal cancer (CRC) screening (n = 230 criteria*)
Table 2 Dimensions of the list of criteria (excerpt)

The preliminary list of criteria was modified in response to the experts’ reviews, mainly by including additional criteria (e.g. inability to drive after sedation, further risks in the preparation phase of colonoscopy, possibility of being unable to work on the day of examination, and the need to sign a consent form and give a blood sample before the examination).

Final list of criteria

The final list of criteria contains 230 criteria (Table 1). Most of the single criteria are rated multi-dimensionally: reporting: yes/no; correctness: yes/no/unclear; presentation: text, numbers, diagrams, tables and/or images; level of evidence: yes, no, lack of evidence indicated (Table 2). To enhance the rating transparency of each criterion, space for free text is provided for verbatim quotes or reported numbers, to document whether a number was presented as a natural frequency [42, 43], and to specify whether a denominator was included, etc.

Elements of the four domains are explained in detail below, including assessment examples, where appropriate.

Domain A: (CRC-specific) content issues

Domain A includes three categories (see Table 1). The subtopic “Information on the etiology and epidemiology of CRC” of the category “Information on CRC and CRC screening” is presented in Table 3 to elucidate the procedure for detection of epidemiological frequencies. It becomes clear that not all criteria have to be met for information material to qualify as being of high quality. Examples of how information in flyers and brochures of this category were assessed are shown below.

Table 3 Criteria for the aetiology and epidemiology of CRC (n = 20) (Domain A, Category: Information on CRC and CRC screening)

Example 1: “CRC is the second most common type of cancer in both men and women.

This statement would be rated as criterion 15 (Table 3):

–Reported? “Yes”

–Correct? “Yes”

–How presented: “Text” (not “Number”)

–Evidence level reported? “Not applicable”

–Quotes: Citation

Example 2: “22,000 people die each year from CRC.

This statement would be rated as criterion 8 (Table 3):

–Reported? “Yes”

–Correct? “No” (number is too low for Germany)

–How presented: “Number”

–Evidence level reported? “No”

–Quotes: “Denominator is lacking, outdated number”

Both examples show the importance of having a manual that provides the correct answers and numbers and, in the second case that defines what extent of deviation from the actual number is acceptable as “correct”. Therefore, the manual is a core part of the list of criteria.

The categories of the two screening procedures, fecal occult blood test and colonoscopy, are constructed similarly. They begin with information on the procedure itself and are supplemented by further criteria on colonoscopy preparation and sedation. Both procedures incorporate criteria on test characteristics (such as sensitivity, specificity, predictive value), on conduct in response to test results and, most importantly, on benefit and risks, including overdiagnosis.

Table 4 shows the criteria on the subtopics of benefits and risks of screening colonoscopy. Benefits include three relevant outcomes: CRC incidence, CRC mortality and all-cause mortality. Each outcome is divided into absolute and relative risk reduction and the number needed to screen. Risk criteria for screening colonoscopy are divided into risks during colonoscopy preparation (including colon cleansing), risks related to adverse effects of sedative drugs, risks of the procedure itself, and risks of overdiagnosis. The subject of overdiagnosis is included because it is known to occur in cancer screening to a varying extent depending on the type of cancer [4446]. Nevertheless, the extent of overdiagnosis or overtreatment of harmless polyps that would never turn into cancer in colorectal cancer screening is unknown and may be low as there are strong hints that colonoscopy will decrease CRC incidence like it is already shown for flexible sigmoidoscopy-based screening [47]. The rating procedure for benefits and risks is illustrated below.

Table 4 Criteria on benefits (n = 9) and risks (n = 23) of screening colonoscopy (Domain A, Category: Information on screening colonoscopy)

Example 3: “According to experts, more than three-quarters of CRC patients could be saved by early screening colonoscopy.

This statement would be rated as criterion 5, CRC mortality (Table 4: Benefits):

–Reported? “Yes”

Correct? “Yes”

–Presented as: “Number”

–Evidence level reported? “Yes”

–Quotes: “No natural frequency, denominator is lacking, no absolute risk reduction is given, evidence from level 3 (case-control) studies is falsely presented as experts’ evidence”.

Example 4: “80% of all CRCs can be prevented by screening colonoscopy.

This statement would be rated as criterion 2, CRC incidence (Table 4: Benefits):

–Reported? “Yes”

–Correct? “Yes”

–Presented as: “Number”

–Evidence level reported? “No”

–Quotes: “No natural frequency, denominator is lacking, no absolute risk reduction is given, evidence from level 3 (case–control) studies

Example 5: “… is a harmless drug preparation

This statement would be rated as criterion 1 (Table 4: Risks):

–Reported? “Yes”

Correct? “No”

–Presented as: “Text”

–Evidence level reported? “No”

–Quotes: Citation

Example 6: “… no pain

This statement would be rated as criterion 12 (Table 4: Risks):

–Reported? “Yes”

–Correct? “No”

–Presented as “Text”

–Evidence level reported? “No”

–Quotes: Citation

Domain B: formal issues (generic)

This domain assesses the formal characteristics of information materials, including information on the authors and editors, possible conflicts of interest, publication dates, aims and target groups. Due to the nature of this meta-information, only the presence or absence of these criteria is rated and not their correctness. As criteria in this domain are very are widely used, we do without describing them in detail.

Domain C: presentation and understandability (generic)

“Understandability (readability/comprehensibility)” assesses the language, sentences, content structure, and numbers of information materials, whereas “presentation (layout)” concerns the structure, font, visual elements and design of the materials (see Table 1). These criteria (e.g., “Sentences are of appropriate length”) require more detailed rating, such as that achieved by four response categories. Therefore, all criteria in this domain were rated on a four-point-scale (yes / mostly yes / mostly no / no). Furthermore, it makes no sense to rate the correctness of these criteria. For most of the assessments in this domain, it is essential to aggregate information: For example, when assessing the length of a sentence, the assessor must search the entire health information material for sentences that are too long. To ensure an unambiguous assessment, the manual should provide a definition of what is “too long” and what proportions of run-on long sentences should lead to which specific ratings. Table 5 provides a detailed list of criteria for sentences, numbers and visual elements followed by a rating example for this category.

Table 5 Criteria for sentences (n = 4), numerical data (n = 4) and visual elements (n = 9) (Domain C, sub topics from both categories)

Example 7: “every year 70,000 persons are newly diagnosed with colorectal cancer.”

Provided that this is the only number given in the health information, this statement would be rated as follows:

–Numerical data criterion 1: Natural frequencies are used? “Yes”

–Criterion 2: Reference parameters are given? “No”

–Criterion 3: Same denominators are used? “No”

–Criterion 4: Loss and gain framing is balanced? “No”

Usually, several numbers are stated in a text. In that case, an aggregated assessment is required.

Domain D: neutrality and balance (generic)

The last domain comprises seven criteria for assessment of neutral and balanced presentation:

1 “Is free of persuasive language”

2 “Is free of scare language”

3 “Is free of scary pictures or graphs”

4 “Is free of fear appeals”

5 “Is free of downplay or minimization”

6 “Is free of one-sided presentation of benefits without risks”

7 “Is free of unbalanced presentation of screening procedures”

The first five criteria are rated “no”, if any persuasive, scary or down-playing language is used to increase participation in screening. We initially defined these criteria as “Does not contain….”, but this phrase was abandoned because the possible double-negative reply might be confusing. The last two criteria combine benefits and risks and presentation of the procedures. To handle this aggregate information, careful operationalization within the manual is needed. Rating examples for this category are given below.

Example 8: “… should participate in bowel cancer screening.

This statement would be rated as criterion 1:

Met? “Yes”.

Example 9: “… is a wicked disease

This statement would be rated as criterion 2:

Met? “Yes”

Applications / practicability

For trained reviewers, the assessment takes about 15–30 minutes for flyers and 15–45 minutes for brochures. Documentation of the corresponding citations took up much of the time. Although this approach may be time-consuming, it may hasten consensus and, most importantly, ensures the transparency of quality assessment.

Inter-rater reliability was not evaluated because the final assessment was achieved by consensus in each case. Discrepant findings were mainly caused by overlooked aspects. Consensus usually took 5 to 15 minutes. Finally, data entry is very time-consuming due to the citations.

Possible methods of presenting rating results

Quantitative analyses can be made on the level of criteria and their dimensions. Until now, none were performed on the level of subtopics, categories or domains. A method to qualitatively sum up the single-criteria results is also lacking. To obtain an overview, the combined results of the two dimensions “Reported?” and “Correct?” can be visualized by means of a traffic light system, using green (correctly reported), yellow (reported but unclear), red (incorrectly reported), and white (not reported) marks. Figure 2 gives an example from the category “Risks and adverse effects of screening colonoscopy” as presented in brochures. This (traffic light) presentation provides a comprehensive overview of results for each criterion (rows), each type of education material (columns), and differences between materials as in benchmarking procedures. It clearly shows that information on risks is rather limited and sometimes false, and that three brochures contain no information on risks.

Figure 2

Presentation of exemplary rating results of risks and adverse effects of screening colonoscopy in 13 brochures (11 criteria).


High-quality patient information materials may help consumers make informed decisions for or against participation in CRC screening. For identification of appropriate information materials, we compiled a list of criteria via qualitative aggregation based on systematically identified recommendations and tools, and validated the list of criteria by a review process. This instrument is designed for use by persons with expertise in cancer screening. It explicitly is not a checklist for consumers to check the quality of health information. The final manual-based list of criteria contains 230 criteria in four domains. The criteria are rated on multiple dimensions (e.g., presence and correctness of information), if applicable. Free text entries (mainly verbatim quotes) were allowed to ensure rating transparency.

One main limitation of the list of criteria is the lack of a summary assessment. It may be tempting to use a numerical scoring system for simplicity, but there is no empirical evidence to support this. Consequently, as there is no justification for a scoring system that applies an arbitrary (one point per criterion, etc.) or explicit weighting system, we rejected the use of a numerical scoring system or scale. Scales for quantitative assessment of study quality were very popular until empirical evidence [48] and theoretical considerations [49] indicated that scales provide invalid results. Because of the lack of a summary assessment method, analyses with the proposed list of criteria are restricted to the level of single criteria. This is inconvenient due to the large number of criteria to be assessed. To give an overview of the results, we used a traffic light (status indicator) system. This system can indicate two dimensions (e.g., presence and correctness of information) simultaneously, and it provides detailed information on single and overall categories. Such a comparative overview is particularly useful for benchmarking purposes. The future aim is to develop a qualitative summary assessment based on ratings on the category level. A Delphi consensus process might be used to explore the importance of each criterion, as was done in International Patient Decision Aids Standard Instrument (IPDASi) development [14].

The list of criteria represents the maximum content of information material. Not all of the criteria are essential for high-quality information. The comprehensiveness and depth of information materials vary depending on the targets and target groups. Thus, it would be reasonable to differentially define essential criteria for short information materials like flyers and for more detailed materials like brochures. It would also be reasonable to select these mandatory, material-specific criteria in a Delphi procedure including experts and consumers. Obviously, expert and consumer opinion is needed to explore the importance of each criterion for further summary assessments and to develop specific assessment lists of criteria for short and more detailed information materials. This input could be used in further research to revise the list of criteria.

The rationale behind providing EBHI on cancer screening is to enable consumers to make informed choices for or against cancer screening. The proposed list of criteria examines whether health information materials meet EBHI standards. It cannot directly assess whether the information is suitable to support informed decision-making. There is evidence from two randomized controlled trials that decision aids on CRC screening via FOBT [8] and FOBT/colonoscopy [7] may effectively support informed choice. However, both studies compared an interactive decision booklet with an accompanying DVD [8] or interactive internet module [7] (intervention groups) against a standard governmental booklet (control group). The effect may have been mediated by the interactive components resulting in more intense study of the materials. The proposed list of criteria does not assess interactive components. Ultimately, if information materials are found to meet EBHI standards according to our list of criteria, it cannot be concluded that these materials promote informed choice. Such a claim would have to be verified in further studies.

The ethical goal of EBHI to enable as many of the target population to make an informed decision whether or not to participate in CRC screening [3] may be conflicting with the aim of achieving a high uptake [50]. There is inconclusive evidence on detailed information material, it may have a positive or no effect on participation [7, 10] or may even increase non-attendance [8]. Non-attendance based on an informed choice has to be accepted, while non-attendance arising from the EBHI itself and not from an informed choice is not desirable. EBHI especially may deter socioeconomically disadvantaged people and those with low health literacy from participating in screening [51] resulting in higher health inequalities. Further research is needed to explore tailored communication strategies for deprived target groups focusing on increasing knowledge and understanding to promote an informed choice-making.

To our knowledge, the proposed list of criteria is the first assessment tool designed to rate the correctness of consumer education materials on CRC screening. Many existing tools use structural quality as a surrogate for content quality, which might not always be correct. For example, it was shown that website origin does not predict content quality: the quality of university websites was not better than that of commercial websites [52]. Website certification programs like the HON (Health on the Net) code [53] and MedCERTAIN (MedPICS Certification and Rating of Trustworthy Health Information on the Net) were also established to ensure the provision of reliable information. The HON code requires health information website owners to abide by eight principles: to indicate the authors’ qualifications, information sources, funding sources and advertising policy and to maintain confidentiality, etc. However, there are concerns that these criteria might not be sufficient to identify trustworthy information. For example, the HON label failed to predict the good content quality of mental health-related websites in some cases [52]. In contrast, the DISCERN score was shown to be a content quality indicator of relatively high specificity [52, 54]. Other analyses of web-based information on depression found that content quality correlated with the DISCERN score and HON label [55]. However, the usual tools for the assessment of content quality do not check the correctness of information and might even rate false information as being of good quality.

As the proposed manual provides the correct answers to the criteria queries, it is an essential part of the rating method. Ideally, many CRC-specific content criteria should be explored by systematic reviews, especially if related to benefits and risks. This was not feasible in this project. Our research, which was mainly restricted to selectively searched evidence from S3 guidelines, systematic reviews and HTA reports, was still very time-consuming. It will be challenging to incorporate the latest evidence in the finished manual, as knowledge changes over time. It is also unclear how often the manual should be updated. Other problems can arise from different interpretations of the evidence. In breast cancer screening, for example, experts disagree on the actual numbers characterizing the benefits and risks in a British leaflet [56]. The National Cancer Institute (USA) took an interesting approach to providing key information, namely, by posting a one-page factsheet on lung cancer screening for doctors and patients providing numbers on benefits and risks derived from a randomized controlled trial [57]. In Europe, aggregated uniform evidence-based factsheets on screening procedures would complement the existing comprehensive guidelines [3] and would offer a thorough base of knowledge for different players who provide information on cancer screening.


The range of the proposed evaluation concept based on a list of 230 criteria and answer manual goes beyond that of previous instruments for quality assessment of health information in that it considers not only the presence, but also the correctness of health information. However, this comprehensive list of criteria is not intended as a final quality assessment tool, but rather as a first step toward thorough evaluation of specific information materials for adherence to EBHI standards made by persons with professional expertise in cancer screening. It may also be used to revise existing leaflets or to develop health information materials on colorectal cancer screening. Furthermore, the proposed list of criteria can be transferred to other cancer screening procedures after suitable modification of the procedure-specific criteria.



Colorectal cancer


Evidence-based health information.


  1. 1.

    Raffle A, Muir Gray JA: How screening started. Screening - Evidence and Practice. 2007, New York: Oxford University Press, 1-30. 1

    Google Scholar 

  2. 2.

    Gotzsche PC, Nielsen M: Screening for breast cancer with mammography. Cochrane Database Syst Rev. 2011, CD001877-

    Google Scholar 

  3. 3.

    Commission E: European Guidelines for Quality Assurance in Colorectal Cancer Screening and Diagnosis. 2010, Luxembourg: Publications Office of the European Union

    Google Scholar 

  4. 4.

    General Medical Counsil: Consent: Patients and Doctors Making Decisions Together.,

  5. 5.

    Patera N: Screening for Colorectal Cancer. Part 1: Screening-Tests and Program Design, HTA Project. HTA Project Report, No. 41a. Edited by: Ludwig Boltzmann Institut. 2010, Austria: The University of York Centre for Reviews and Dissemination

    Google Scholar 

  6. 6.

    Pox CP, Altenhofen L, Brenner H, Theilmeier A, Von SD, Schmiegel W: Efficacy of a nationwide screening colonoscopy program for colorectal cancer. Gastroenterology. 2012, 142: 1460-1467. 10.1053/j.gastro.2012.03.022.

    Article  PubMed  Google Scholar 

  7. 7.

    Steckelberg A, Hulfenhaus C, Haastert B, Muhlhauser I: Effect of evidence based risk information on “informed choice” in colorectal cancer screening: randomised controlled trial. BMJ. 2011, 342: d3193-10.1136/bmj.d3193.

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Smith SK, Trevena L, Simpson JM, Barratt A, Nutbeam D, McCaffery KJ: A decision aid to support informed choices about bowel cancer screening among adults with low education: randomised controlled trial. BMJ. 2010, 341: c5370-10.1136/bmj.c5370.

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Bunge M, Mühlhauser I, Steckelberg A: What constitutes evidence-based patient information? overview of discussed criteria. Patient Educ Couns. 2010, 78: 316-328. 10.1016/j.pec.2009.10.029.

    Article  PubMed  Google Scholar 

  10. 10.

    Austoker J, Giordano L, Hewitson P, Villain P: European guidelines for quality assurance in colorectal cancer screening and diagnosis. First Edition - Communication. Endoscopy. 2012, 44 (Suppl 3): SE164-SE185.

    PubMed  Google Scholar 

  11. 11.

    Charnock D, Shepperd S, Needham G, Gann R: DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999, 53: 105-111. 10.1136/jech.53.2.105.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Sänger S, Dierks M-L, Lang B, Englert G, Quadder B, Engelbrecht J, Huth A, Öllenschläge G, Rheinberger P: Check-in Instrument zur Qualitätsbewertung von Gedruckten und Elektronischen Gesundheitsinformationen.,

  13. 13.

    Kopke S, Berger B, Steckelberg A, Meyer G: Evaluation tools for patient information commonly used in Germany–a critical analysis. Z Arztl Fortbild Qualitatssich. 2005, 99: 353-357.

    PubMed  Google Scholar 

  14. 14.

    Elwyn G, O’Connor AM, Bennett C, Newcombe RG, Politi M, Durand MA, Drake E, Joseph-Williams N, Khangura S, Saarimaki A, Sivell S, Stiel M, Bernstein SJ, Col N, Coulter A, Eden K, Härter M, Rovner MH, Moumjid N, Stacey D, Thomson R, Whelan T, van der Weijden T, Edwards A: Assessing the quality of decision support technologies using the international patient decision Aid standards instrument (IPDASi). PLoS One. 2009, 4: e4705-10.1371/journal.pone.0004705.

    Article  PubMed  PubMed Central  Google Scholar 

  15. 15.

    Steckelberg A, Balgenorth A, Muhlhauser I: Analysis of German language consumer information brochures on screening for colorectal cancer. Z Arztl Fortbild Qualitatssich. 2001, 95: 535-538.

    CAS  PubMed  Google Scholar 

  16. 16.

    National Cancer Plan Germany: Aktueller Stand des Nationalen Krebsplans. Handlungsfelder, Ziele und Umsetzungsempfehlungen.,

  17. 17.

    Oxford Centre for Evidence-Based Medicine: Levels of Evidence.,

  18. 18.

    Deutsches Netzwerk Evidenzbasierte Medizin: Kriterien zur Erstellung von Patienteninformationen zu Krebsfrüherkennungsuntersuchungen.,

  19. 19.

    Meyerrose B, Sänger S: Muster-Gesundheitsaufklärung für Früherkennungsmaßnahmen.,

  20. 20.

    Badarudeen S, Sabharwal S: Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010, 468: 2572-2580. 10.1007/s11999-010-1380-y.

    Article  PubMed  PubMed Central  Google Scholar 

  21. 21.

    Ärztliches Zentrum für Qualität in der Medizin: Qualität von Informationen - So Erkennt man Informationsqualität.,

  22. 22.

    Medizin DNE: Die gute Praxis Gesundheitsinformation’. Z Evid Fortbild Qual Gesundhwes. 2010, 104: 66-68.

    Article  Google Scholar 

  23. 23.

    NHS: Written Information: General Guidance.,

  24. 24.

    Sänger S, Lang B, Klemperer D, Thomeczek C, Dierks M-L: Manual Patienteninformation. Empfehlungen zur Erstellung evidenzbasierter Patienteninformationen. Äzq Schriftenr Band. 2006, 25: 4-69.

    Google Scholar 

  25. 25.

    Schneider K, Walter U: Lernfördernde Gestaltung von Bild- und Textmaterialien für den Gesundheitsbereich. Frankfurt/M. 1992, Peter Lang. Internationaler Verlag der Wissenschaften: Berlin, Bern, New York, Paris, Wien

    Google Scholar 

  26. 26.

    Steckelberg A, Berger B, Köpke S, Heesen C, Mühlhauser I: Kriterien für evidenzbasierte Patienteninformationen. Z Arztl Fortbild Qualitatssich. 2005, 99: 343-351.

    PubMed  Google Scholar 

  27. 27.

    Clerehan R, Buchbinder R, Moodie J: A linguistic framework for assessing the quality of written patient information: its use in assessing methotrexate information for rheumatoid arthritis. Health Educ Res. 2005, 20: 334-344. 10.1093/her/cyg123.

    Article  PubMed  Google Scholar 

  28. 28.

    Shaddock J: Appraising the quality of consumer health information leaflets. Health Expect. 2002, 5: 84-87. 10.1046/j.1369-6513.2002.00171.x.

    Article  PubMed  Google Scholar 

  29. 29.

    Coulter A, Entwistle V, Gilbert D: Sharing decisions with patients: is the information good enough?. BMJ. 1999, 318: 318-322. 10.1136/bmj.318.7179.318.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  30. 30.

    Davey HM, Barratt AL, Butow PN, Houssami N: The impact of different criteria for selecting information to be provided to women undergoing diagnostic breast tests. Patient Educ Couns. 2008, 71: 86-94. 10.1016/j.pec.2007.12.001.

    Article  PubMed  Google Scholar 

  31. 31.

    Gummersbach E, Abholz H-H: Info-flyer “Mammographie-Screening” in Deutschland: Welche Informationen enthält er und welche sollte er enthalten?. Z Allgemeinmed. 2006, 82: 491-494. 10.1055/s-2006-942297.

    Article  Google Scholar 

  32. 32.

    Jorgensen KJ, Gotzsche PC: Presentation on websites of possible benefits and harms from screening for breast cancer: cross sectional study. BMJ. 2004, 328: 148-10.1136/bmj.328.7432.148.

    Article  PubMed  PubMed Central  Google Scholar 

  33. 33.

    Slaytor EK, Ward JE: How risks of breast cancer and benefits of screening are communicated to women: analysis of 58 pamphlets. BMJ. 1998, 317: 263-264. 10.1136/bmj.317.7153.263.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  34. 34.

    International Patient Decisions Aid Standards Collaboration: Criteria for Judging the Quality of Patient Decision Aids.,

  35. 35.

    Focus Patient: Erstes Österreichisches Qualitätssiegel für Gesundheitsinformationen.,

  36. 36.

    Tolsdorf M: Mit Broschüren Gezielt Informieren.,

  37. 37.

    Hoffmann T, Twork S, Kugler J: Patientenorientierung im Internet- qualitative Bewertung von Internetseiten für Multiple-Sklerose-Betroffene. Heilberufe. 2009, 61: 98-103. 10.1007/s00058-009-1621-1.

    Article  Google Scholar 

  38. 38.

    Muhlhauser I, Oser F: Does WIKIPEDIA provide evidence-based health care information? a content analysis. Z Evid Fortbild Qual Gesundhwes. 2008, 102: 441-448. 10.1016/j.zefq.2008.06.019.

    Article  PubMed  Google Scholar 

  39. 39.

    Charvet-Berard AI, Chopard P, Perneger TV: Measuring quality of patient information documents with an expanded EQIP scale. Patient Educ Couns. 2008, 70: 407-411. 10.1016/j.pec.2007.11.018.

    CAS  Article  PubMed  Google Scholar 

  40. 40.

    Moult B, Franck LS, Brady H: Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect: An Int J Publ Partic in Health Care and Health Policy. 2004, 7: 165-175.

    Article  Google Scholar 

  41. 41.

    White P, Smith H, Webley F, Frew A: A survey of the quality of information leaflets on hayfever available from general practices and community pharmacies. Clin Exp Allergy. 2004, 34: 1438-1443. 10.1111/j.1365-2222.2004.02055.x.

    CAS  Article  PubMed  Google Scholar 

  42. 42.

    Gigerenzer G, Edwards A: Simple tools for understanding risks: from innumeracy to insight. BMJ. 2003, 327: 741-744. 10.1136/bmj.327.7417.741.

    Article  PubMed  PubMed Central  Google Scholar 

  43. 43.

    Knapp P, Gardner PH, Carrigan N, Raynor DK, Woolf E: Perceived risk of medicine side effects in users of a patient information website: a study of the use of verbal descriptors, percentages and natural frequencies. Br J Health Psychol. 2009, 14: 579-594. 10.1348/135910708X375344.

    CAS  Article  PubMed  Google Scholar 

  44. 44.

    Welch HG, Black WC: Overdiagnosis in cancer. J Natl Cancer Inst. 2010, 102: 605-613. 10.1093/jnci/djq099.

    Article  PubMed  Google Scholar 

  45. 45.

    Jorgensen KJ, Zahl PH, Gotzsche PC: Overdiagnosis in organised mammography screening in denmark. A comparative study. BMC Womens Health. 2009, 9: 36-10.1186/1472-6874-9-36.

    Article  PubMed  PubMed Central  Google Scholar 

  46. 46.

    Elmore JG, Fletcher SW: Overdiagnosis in breast cancer screening: time to tackle an underappreciated harm. Ann Intern Med. 2012, 156: 536-537. 10.7326/0003-4819-156-7-201204030-00012.

    Article  PubMed  Google Scholar 

  47. 47.

    Elmunzer BJ, Hayward RA, Schoenfeld PS, Saini SD, Deshpande A, Waljee AK: Effect of flexible sigmoidoscopy-based screening on incidence and mortality of colorectal cancer: a systematic review and meta-analysis of randomized controlled trials. PLoS Med. 2012, 9: e1001352-10.1371/journal.pmed.1001352.

    Article  PubMed  PubMed Central  Google Scholar 

  48. 48.

    Jüni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282: 1054-1060. 10.1001/jama.282.11.1054.

    Article  PubMed  Google Scholar 

  49. 49.

    Greenland S, O’Rourke K: On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001, 2: 463-471. 10.1093/biostatistics/2.4.463.

    CAS  Article  PubMed  Google Scholar 

  50. 50.

    Raffle AE: Information about screening—is it to achieve high uptake or to ensure informed choice?. Health Expect. 2001, 4: 92-98. 10.1046/j.1369-6513.2001.00138.x.

    CAS  Article  PubMed  Google Scholar 

  51. 51.

    Marteau TM, Mann E, Prevost AT, Vasconcelos JC, Kellar I, Sanderson S, Parker M, Griffin S, Sutton S, Kinmonth AL: Impact of an informed choice invitation on uptake of screening for diabetes in primary care (DICISION): randomised trial. BMJ. 2010, 340: c2138-10.1136/bmj.c2138.

    Article  PubMed  PubMed Central  Google Scholar 

  52. 52.

    Khazaal Y, Chatton A, Zullino D, Khan R: HON label and DISCERN as content quality indicators of health-related websites. Psychiatr Q. 2012, 83: 15-27. 10.1007/s11126-011-9179-x.

    Article  PubMed  Google Scholar 

  53. 53.

    Boyer C, Baujard V, Geissbuhler A: Evolution of health web certification through the HONcode experience. Stud Health Technol Inform. 2011, 169: 53-57.

    PubMed  Google Scholar 

  54. 54.

    Khazaal Y, Chatton A, Cochand S, Zullino D: Quality of Web-based information on cocaine addiction. Patient Educ Couns. 2008, 72: 336-341. 10.1016/j.pec.2008.03.002.

    Article  PubMed  Google Scholar 

  55. 55.

    Zermatten A, Khazaal Y, Coquard O, Chatton A, Bondolfi G: Quality of Web-based information on depression. Depress Anxiety. 2010, 27: 852-858. 10.1002/da.20665.

    Article  PubMed  Google Scholar 

  56. 56.

    Hawkes N: Breast cancer screening is to be reviewed, cancer tsar announces. BMJ. 2011, 343: d6905-10.1136/bmj.d6905. doi:10.1136/bmj.d6905

    Article  PubMed  Google Scholar 

  57. 57.

    Woloshin S, Schwartz LM, Black WC, Kramer BS: Cancer screening campaigns — getting past uninformative persuasion. N Engl J Med. 2012, 367: 18-10.1056/NEJMicm1102441.

    Article  Google Scholar 

Pre-publication history

  1. The pre-publication history for this paper can be accessed here:

Download references


This article is based on the results of a research project on the promotion of informed decision-making, which is partially funded by the Federal Ministry of Health, Germany. We would like to thank the external experts Dr. Berndt Birkner, Dr. Regine Hagmann, Prof. David Klemperer, Prof. Jürgen Riemann, and Petra Uschold, who reviewed the list of criteria and those who took part in the final meeting.

We acknowledge support by Deutsche Forschungsgemeinschaft.

Author information



Corresponding author

Correspondence to Maren Dreier.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

BB performed the literature search. MD, BB, GS and IK selected the literature, extracted the criteria, developed the list of criteria, and tested the final list of criteria. UW, MLD, JT, and EMB critically revised the list of criteria and read and approved the final manuscript, especially UW and MLD. MD and BB drafted the manuscript. All authors participated in interpretation of the results and critically revised subsequent versions of the paper for important intellectual content. All authors read and approved the final manuscript.

Electronic supplementary material

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Dreier, M., Borutta, B., Seidel, G. et al. Development of a comprehensive list of criteria for evaluating consumer education materials on colorectal cancer screening. BMC Public Health 13, 843 (2013).

Download citation


  • Screening
  • Early diagnosis
  • Colorectal cancer
  • Bowel cancer
  • Informed decision-making
  • Informed choice
  • Evidence-based health information
  • Evidence-based patient information
  • Risk communication
  • Decision aids