Skip to main content

Table 2 Synopsis of the interviews transcribed, interviewer 1

From: Expert opinions on good practice in evaluation of health promotion and primary prevention measures related to children and adolescents in Germany

Respondent

1

2

3

4

5

6

7

8

A1. What does the term “evaluation” mean to you?

°Verify achievement of objectives

°Evaluation as scientific proof of effectiveness

°Evaluation as systematic approach for the verification of achievement of objectives of projects. Supporting function of evaluation

°Measure effects and find statements regarding the cost-effectiveness relation

°Evaluation is survey of data to assess measures

°Evaluation as social scientific-based statement of the value or benefit of an item

°Evaluation as the assessment of the effectiveness of a measure

°Systematic collection of data in compliance with scientific requirements

°Finally, an opinion regarding the project criteria should be made

°Evaluation can refer to processes or results

A2. What role does the evaluation of health promotion and prevention measures play in your daily work?

°Performs evaluations itself

(science)

°Sees evaluation as quality factor for programs that are offered or carried out itself

°Carries out evaluations itself

°Orders evaluations. Co-develops the concept

°Active in development of prevention offers and their evaluation

°Carries out evaluations of prevention projects itself

°Carry out evaluation projects themselves

°Works at research institute, which evaluates prevention measures itself

°Edits project proposals and final reports

°Passes recommendations for improvements

A3. Do you experience evaluations to be perceived in practice as meaningful or more disruptive? Please describe one example each of a meaningful evaluation or a less meaningful evaluation from your experience.

°Evaluations are useful if

°useful questions/ target criteria exist

°effects are relevant for intervention and interest in knowledge is present

°suggestions for further development are given

°comprehension of practitioners takes place

°practitioners understand instruments

°Useful as decision support for financing

°Is evaluation for cash useful at all? (cost savings desired. Cannot be verified in monetary terms)

°Disturbing regarding the detection of “soft factors” (if outcomes are not easily measurable)

°Less useful if there are no meaningful target criteria

°Dependent on questioning

°Can be experienced as an extra burden

°Disturbing because of great effort

°Too expensive if every single small project is evaluated

°Practitioners might feel uncomfortable with evaluations if they are examined carefully and their routines are broken

°Is possibly seen as disturbing by the prevention experts in the field, not so much on the part of the target groups

°Evaluation from supplier’s point of view is expensive and costly

°Abstract results from journals do not have a reference to the practitioner’s living environment °Institutions could see themselves as “cash cows”

°Can be experienced as unnecessary

°In practice also partly fear of the evaluation due to lacking knowledge

°Evaluation can also be experienced as a threat

°From a scientific point of view, evaluation offers chances

B. What according to your assessment is the practical importance of evaluation (in terms of measuring the costs and effects) of health promotion and prevention measures, especially in children and adolescents?

°Scientific view can give impulses

°Evaluation cannot clarify whether the program makes sense

°Contributes to further development of measures (based on ideas of practice), if desired effects are unpresentable

°Evaluation is important for the public, politicians, and investors

°Practical importance given if not only PR-effective factors are examined

°Practically relevant if a long-term follow-up exists

°Evaluation as basis for decision-making

°Costs as argument

°Soft variables possibly not helpful for political decisions

°Evaluation gives hints on what has an effect and what has no effect

°Important to deliver projects to be good practice

°Evaluation gives suggestions to improve preventive work/measures

°Especially regarding very innovative, cost-intensive projects, evaluation should take place

°Evaluation discloses new aspects

°Evaluation as enlargement of the people involved (new point of view)

°Evaluations open new perspectives for the people involved

°Evaluations address specialists and the structures they work in

°Target individuals can be involved by participatory forms

°No direct practical meaning as measures are also offered without evaluation. On a meta level, however, evaluations are a condition for implementation

°Evaluation to choose between different measures

°Evaluation important

°Cost analysis helps to convince the decision-maker and to continue a measure

Please describe, based on your experience, one example each for a useful evaluation or a less useful evaluation.

°Useful: Life-competence projects in the prevention of drug dependence/Tiger-Kidsa

°Safari-Kidsb

°“Münchner Modell der frühen Hilfen”c

°Absence time report, workplace health promotion

   

°Participative approach of movement promotion for underprivileged women

C1. What aspects of health promotion and prention measures (e.g., cost or effects) are particularly important to you, or when is a project successful for you? What would be an example of a particularly successful project in your opinion? (Why?)

°Achievement of objectives (to influence risk and protection factors) °Reaching the target group

°Transfer

°Acceptance of the performing person

°Qualified target group

°Sustainability

°Broad effectiveness

°Feasibility, participation, and sustainability

°Successful if people are addressed who benefit most from it

°Good effectiveness with appropriate cost employment

°View of the concerned individuals shall be considered

°Evaluation shall not only reach the middle class

°Participation (can reduce costs)

°Participative development of quality

°It is also a success if only a few are helped

°Comparability

°Textual consideration

°Connection between taking an action and the effects must be clearly produced

°Comparison with initial state must take place

°If subjective points of view of the people concerned are included in the evaluation and feedback is obtained from those people

°Considering also the benefit of the principal °If the measure is effective

°Primary effects

°Costs also important

°Broad effectiveness important

°Premise is that a program has an effect. Program should be realizable in terms of costs

°Validity is important: does the project produce what it expected to cause?

°Important if target group is achieved

°Successful project should have concrete questioning, should be able to be broadly used, or should address special target groups purposefully

°If targets are achieved

°If achievement of objectives can be attributed to the measure

°Has to be feasible under real conditions

°Sustainability regarding effects and transferability

°Must not be offered only once as a project

C2. Which parameters would make such success practically measurable?

°Protective factors as important as target figure

°Risk and protective factors with scales

°Representative studies

°There are hard and soft factors; however, in the purely preventive area, the soft factors are more important (growth, etc.)

°By observation of acceptance, availability

°Investigation of long-term stability of behavior °Sustainability of behavior and structures

°Reachability of participants

°Achievement of objectives

°Costs

°Dependence of object and operationalizability

°Can be soft or hard parameters

°Regarding validity, lifestyle parameters would be interesting or physical activity

°Measurements before and afterwards and longer follow-up important

°Protocol, what is implemented, when and where would be important

°Effectiveness depends on the individual case

°Sustainable implementation can be proven by documentation

°For the transfer, it is important to create transfer aids

C3. What has proven to be particularly easy to implement regarding the implementation of the evaluation of health promotion and prevention measures in your experience?

°Work closely together with those performing the task and introduce their expertise

°Address family/environment (sustainability)

°Use settings (existing structures)

°Adapt the survey tools to target groups

°Keep effort low

°Tight communication is most helpful for the project

°Stakeholder participation and transparency

°It is helpful to use existing structures rather than to approach individuals

°Plan evaluation from the beginning

°Minimal requirements that always work: intervention group, control group (not absolutely randomized)

°Consider useful indicators

C4. What obstacles are to be expected in the evaluation of health promotion and prevention measures?

°Time

°Staff

°Motivation

°Institutions in practice overstrained by evidence base °Motivation of participants as part of evaluation

°Great effort

°Understanding of the people involved

°No clear causality of the measure recognizable

°Cost involvement difficult due to privacy, especially when no manifest disease

°Costs that are influential as a result of prevention are not directly recognizable

°Consideration of costs only useful with long-term observation

°Effects only recognizable after a long time

°External evaluation of those possibly not familiar enough with the project

°Participation is target group dependent and cultural differences have to be noticed

°Too much effort is an obstacle

°Lacking acceptance by those concerned

°Barrier if evaluation is performed during current operations instead of already planned before

°Evaluation is felt as being irksome

°Wrong information °Refusal

°Those concerned do not gain any benefit

°Objective is adapted by principal

°Those concerned do not want to be examined precisely

°Evaluation abused as delay in decision-making

°Different interests, mistrust (scare sponsors off)

°Consent of parents for comprehension of children

°Recruitment of those who one wants to reach is difficult (prevention bias)

°Difficulties in determining whether effects also appear outside laboratory conditions (efficiency vs. efficacy)

°Long-term follow-up

°In children, effects appear time-lagged

°Effects of evaluations are never very strong

°Objectives are never reached at all

°Missing evaluation and quality culture

°Evaluation is felt as a threat

°Measures are complex

°Cost aspects

°Control group does not see any benefit from participation

  1. aTigerKids: Strauss A, Herbert B, Mitschek C, Duvinage K, Koletzko B. [TigerKids. Successful health promotion in preschool settings]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2011 Mar;54(3):322-9
  2. bSafari-Kids: https://www.dak.de/dak/leistungen/SAFARIKIDS-1079158.html
  3. cMünchner Modell der frühen Hilfen: https://www.muenchen.de/rathaus/Stadtverwaltung/Sozialreferat/Jugendamt/Familie/Fruehe-Hilfen.html