Respondent | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
---|---|---|---|---|---|---|---|---|
A1. What does the term “evaluation” mean to you? | °Verify achievement of objectives | °Evaluation as scientific proof of effectiveness | °Evaluation as systematic approach for the verification of achievement of objectives of projects. Supporting function of evaluation | °Measure effects and find statements regarding the cost-effectiveness relation | °Evaluation is survey of data to assess measures | °Evaluation as social scientific-based statement of the value or benefit of an item | °Evaluation as the assessment of the effectiveness of a measure | °Systematic collection of data in compliance with scientific requirements °Finally, an opinion regarding the project criteria should be made °Evaluation can refer to processes or results |
A2. What role does the evaluation of health promotion and prevention measures play in your daily work? | °Performs evaluations itself (science) | °Sees evaluation as quality factor for programs that are offered or carried out itself | °Carries out evaluations itself °Orders evaluations. Co-develops the concept | °Active in development of prevention offers and their evaluation | °Carries out evaluations of prevention projects itself | °Carry out evaluation projects themselves | °Works at research institute, which evaluates prevention measures itself | °Edits project proposals and final reports °Passes recommendations for improvements |
A3. Do you experience evaluations to be perceived in practice as meaningful or more disruptive? Please describe one example each of a meaningful evaluation or a less meaningful evaluation from your experience. | °Evaluations are useful if °useful questions/ target criteria exist °effects are relevant for intervention and interest in knowledge is present °suggestions for further development are given °comprehension of practitioners takes place °practitioners understand instruments | °Useful as decision support for financing °Is evaluation for cash useful at all? (cost savings desired. Cannot be verified in monetary terms) °Disturbing regarding the detection of “soft factors” (if outcomes are not easily measurable) °Less useful if there are no meaningful target criteria | °Dependent on questioning °Can be experienced as an extra burden | °Disturbing because of great effort | °Too expensive if every single small project is evaluated | °Practitioners might feel uncomfortable with evaluations if they are examined carefully and their routines are broken | °Is possibly seen as disturbing by the prevention experts in the field, not so much on the part of the target groups °Evaluation from supplier’s point of view is expensive and costly °Abstract results from journals do not have a reference to the practitioner’s living environment °Institutions could see themselves as “cash cows” | °Can be experienced as unnecessary °In practice also partly fear of the evaluation due to lacking knowledge °Evaluation can also be experienced as a threat °From a scientific point of view, evaluation offers chances |
B. What according to your assessment is the practical importance of evaluation (in terms of measuring the costs and effects) of health promotion and prevention measures, especially in children and adolescents? | °Scientific view can give impulses °Evaluation cannot clarify whether the program makes sense °Contributes to further development of measures (based on ideas of practice), if desired effects are unpresentable °Evaluation is important for the public, politicians, and investors °Practical importance given if not only PR-effective factors are examined | °Practically relevant if a long-term follow-up exists | °Evaluation as basis for decision-making °Costs as argument °Soft variables possibly not helpful for political decisions | °Evaluation gives hints on what has an effect and what has no effect | °Important to deliver projects to be good practice °Evaluation gives suggestions to improve preventive work/measures °Especially regarding very innovative, cost-intensive projects, evaluation should take place °Evaluation discloses new aspects °Evaluation as enlargement of the people involved (new point of view) | °Evaluations open new perspectives for the people involved °Evaluations address specialists and the structures they work in °Target individuals can be involved by participatory forms | °No direct practical meaning as measures are also offered without evaluation. On a meta level, however, evaluations are a condition for implementation °Evaluation to choose between different measures | °Evaluation important °Cost analysis helps to convince the decision-maker and to continue a measure |
Please describe, based on your experience, one example each for a useful evaluation or a less useful evaluation. | °Useful: Life-competence projects in the prevention of drug dependence/Tiger-Kidsa | °Safari-Kidsb | °“Münchner Modell der frühen Hilfen”c | °Absence time report, workplace health promotion | °Participative approach of movement promotion for underprivileged women | |||
C1. What aspects of health promotion and prention measures (e.g., cost or effects) are particularly important to you, or when is a project successful for you? What would be an example of a particularly successful project in your opinion? (Why?) | °Achievement of objectives (to influence risk and protection factors) °Reaching the target group °Transfer °Acceptance of the performing person °Qualified target group °Sustainability °Broad effectiveness | °Feasibility, participation, and sustainability | °Successful if people are addressed who benefit most from it | °Good effectiveness with appropriate cost employment °View of the concerned individuals shall be considered °Evaluation shall not only reach the middle class | °Participation (can reduce costs) °Participative development of quality °It is also a success if only a few are helped °Comparability °Textual consideration | °Connection between taking an action and the effects must be clearly produced °Comparison with initial state must take place °If subjective points of view of the people concerned are included in the evaluation and feedback is obtained from those people °Considering also the benefit of the principal °If the measure is effective | °Primary effects °Costs also important °Broad effectiveness important °Premise is that a program has an effect. Program should be realizable in terms of costs °Validity is important: does the project produce what it expected to cause? °Important if target group is achieved °Successful project should have concrete questioning, should be able to be broadly used, or should address special target groups purposefully | °If targets are achieved °If achievement of objectives can be attributed to the measure °Has to be feasible under real conditions °Sustainability regarding effects and transferability °Must not be offered only once as a project |
C2. Which parameters would make such success practically measurable? | °Protective factors as important as target figure °Risk and protective factors with scales °Representative studies | °There are hard and soft factors; however, in the purely preventive area, the soft factors are more important (growth, etc.) | °By observation of acceptance, availability | °Investigation of long-term stability of behavior °Sustainability of behavior and structures | °Reachability of participants °Achievement of objectives °Costs | °Dependence of object and operationalizability °Can be soft or hard parameters | °Regarding validity, lifestyle parameters would be interesting or physical activity °Measurements before and afterwards and longer follow-up important °Protocol, what is implemented, when and where would be important | °Effectiveness depends on the individual case °Sustainable implementation can be proven by documentation °For the transfer, it is important to create transfer aids |
C3. What has proven to be particularly easy to implement regarding the implementation of the evaluation of health promotion and prevention measures in your experience? | °Work closely together with those performing the task and introduce their expertise | °Address family/environment (sustainability) °Use settings (existing structures) | °Adapt the survey tools to target groups | °Keep effort low | °Tight communication is most helpful for the project | °Stakeholder participation and transparency | °It is helpful to use existing structures rather than to approach individuals | °Plan evaluation from the beginning °Minimal requirements that always work: intervention group, control group (not absolutely randomized) °Consider useful indicators |
C4. What obstacles are to be expected in the evaluation of health promotion and prevention measures? | °Time °Staff °Motivation °Institutions in practice overstrained by evidence base °Motivation of participants as part of evaluation | °Great effort °Understanding of the people involved °No clear causality of the measure recognizable °Cost involvement difficult due to privacy, especially when no manifest disease °Costs that are influential as a result of prevention are not directly recognizable °Consideration of costs only useful with long-term observation °Effects only recognizable after a long time | °External evaluation of those possibly not familiar enough with the project °Participation is target group dependent and cultural differences have to be noticed | °Too much effort is an obstacle °Lacking acceptance by those concerned | °Barrier if evaluation is performed during current operations instead of already planned before °Evaluation is felt as being irksome | °Wrong information °Refusal °Those concerned do not gain any benefit °Objective is adapted by principal °Those concerned do not want to be examined precisely °Evaluation abused as delay in decision-making °Different interests, mistrust (scare sponsors off) °Consent of parents for comprehension of children | °Recruitment of those who one wants to reach is difficult (prevention bias) °Difficulties in determining whether effects also appear outside laboratory conditions (efficiency vs. efficacy) °Long-term follow-up | °In children, effects appear time-lagged °Effects of evaluations are never very strong °Objectives are never reached at all °Missing evaluation and quality culture °Evaluation is felt as a threat °Measures are complex °Cost aspects °Control group does not see any benefit from participation |