Table 3 Synopsis of the interviews transcribed, interviewer 2
Respondent | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
A1. What does the term “evaluation” mean to you? | To verify whether targets aspired to were reached ➔ achievement of objectives | Proof of efficacy, proof that positive results emerge, arrangement of evaluation not defined | Approach or process for the evaluation of measures, important: systematic approach, systematic review of achievement of objectives. Evaluation of endpoints as well as process evaluation is important to optimize results | Measure effects, but also learn something about cost–benefit relation | Collection of data or findings for the evaluation of a measure | Socio-scientific-based assessment of value or benefit of an object; object can be program, institution, structure, does it work, and are objectives achieved? ➔ achievement of objectives | Evaluation but also the question after the effect of a measure | Collection of data and analyzing an object. Important here: systematic implementation has to satisfy scientific requirements. Evaluation should be included so that position statement also possible, questioning of the evaluation has to be answered |
A2. What role does the evaluation of health promotion and prevention measures play in your daily work? | Majority of work ➔ main business, involved in development, implementation, and interpretation | For assessment of programs always a factor, but itself not involved in evaluation, only accompanying action | Important role, main task, partly evaluations self-designed, implemented, and analyzed, partly main business, involved in development, accompanies implementation and analysis | Important role, because while developing new prevention programs, evaluation is also developed immediately | Evaluation broad: Analysis of own work: constantly, evaluation closely: after a fix frame for specific projects ➔ sideline, focus on projects, evaluation as additive | Quietly business field, everyday business, evaluation of federal programs, legislative initiative, practical approaches➔main business, involved in development, implementation and analysis | Main business, involved in development and analysis | Not involved in development, implementation, and analysis itself but everyday business is the assessment of programs and projects, where evaluations should or must also be included |
A3. Do you experience evaluations to be perceived in practice as meaningful or more disruptive? Please describe one example each of a meaningful evaluation or a less meaningful evaluation from your experience. | Basically useful, if you have a specific question and if it serves for further development, clear target criteria less useful: measuring instruments which are not understood by those people who should apply them | Both, useful for investor, disturbing for program suppliers, implementers, involved persons; example: Safari-Kids | Useful vs. disturbing does not exclude each other, can be useful and nevertheless disturb ➔ if evaluation is useful or not also depends on whether the data are used sensibly or not, e.g., early aids | In practice, often sensed as disturbing due to effort, e.g., AOK-absence-time report | Useful if prior defined criteria are checked; problematic or disturbing for the peoples involved because of high effort ➔ depends on what is done and who is affected | Partly useful, but aware that one possibly disturbs the “evaluated” ➔ more useful Example (less useful): program evaluated, objectives not reached, nevertheless no change and vice versa, program evaluated, very good results, nevertheless program terminated ➔whether evaluation useful or not also depends on whether data are used in a proper way or not | Possibly sensed as disturbing or can sometimes be sensed as disturbing by people or prevention workers who work in the field and do not immediately get in touch with the results. There are no useless evaluations; if ever there are bad evaluations | Partly useful, but from experience, evaluations are often seen as useless in practice, possibly even as a threat. Most important insight from the examples: clear objective, which one wants to reach and is verifiable; this often goes wrong |
B. What according to your assessment is the practical importance of evaluation (in terms of measuring the costs and effects) of health promotion and prevention measures, especially in children and adolescents? | Scientific view on something that is done in everyday life in practice and can give impulses. Not: this makes sense or not, but an impulse for further development of the measures, therefore may have practical importance but not the non plus ultra | Useful if long-term observation is included | Basically important for decision-making, therefore a cost–benefit analysis would also increase importance | It is important to make sure what has an effect and what does not, especially regarding children and adolescents; evaluations are important to find out what has an effect or what is useful and what is not | Important: quality of the measure, thus quality in planning and implementation has to be present and has to be improved; projects get more and more costly and bigger, here an involvement of evaluation is important | Evaluations are initially not directly addressed to children and adolescents, but rather to specialists and structures in which children move, i.e., school. Implementation of evaluation results or possible recommendations also always dependent on how this fits into existing structures. The closer one is to the decision-makers, the more useful and momentous evaluations can be for children and adolescents. If one involves children and adolescents directly in the evaluation (with different perspectives), then they have an opportunity to take part in the design | Of course one can also implement prevention measures completely without evaluation; thus, it has no immediate practical importance. On a meta level, I would say, however, that it actually is a condition for implementation in practice or at least should be one | Extremely important, especially regarding the costs; decision-makers can possibly be convinced better if one can give information about costs as well as effectiveness |
Please describe, based on your experience, one example each for a useful evaluation or a less useful evaluation. | °Useful: Life-competence projects in the prevention of drug dependence/Tiger-Kidsa | °Safari-Kidsb | °“Münchner Modell der frühen Hilfen”c | °Absence time report, workplace health promotion | °Participative approach of movement promotion for underprivileged women | |||
C1. What aspects of health promotion and prevention measures (e.g., costs or effects) are particularly important to you, or when is a project successful for you? What would be an example of a particularly successful project in your opinion? (Why?) | Project is successful if objectives are achieved and especially the planned target group is achieved. Transferability. Successful prevention influences risk factors and protection factors, feasibility, acceptance, sustainability, example: Tiger-Kids | Feasibility, participation, sustainability, example: Safari-Kids | Overall very general, always dependent on the question, but for us: accessibility of target group (disadvantaged people) | Successful would mean a good effect with appropriate cost deployment, dimension of effects most important, costs have to remain within reasonable bounds, participation of the people involved, i.e., participative development of quality, reference to absence report, operational health promotion as best developed approach to prevention | Participative development of quality, thus involvement of the people affected, evaluation may not change the project, fixed criteria which should be comparable, overridingly assessment concerning the contents of the success, thus with regard to the objectives of the project not of the evaluation | Connection between action and effect ➔ imputability, separation of prevention in the narrow sense and health promotion, different approaches, different prerequisites for evaluation, who uses the results? Participative design whenever possible | Premise for offering a program at all is that is has an effect. Simultaneously, it has to enable a realistic feasibility in terms of costs. Validity, attainment of the target group, achievement of objectives | Feasibility, sustainability, transferability, for example a project of movement promotion for disadvantaged women, which meets these criteria; however no dramatic effects established, only small changes |
C2. Which parameters would make such a success practically measurable? | Effectiveness, attainment of target groups, feasibility, sustainability, to measure with appropriate representative studies | Hard factors (weight, blood values,…) and soft factors (fun); in children, more soft ones, at least in the preventive area | Acceptance of target group as a parameter of accessibility | Individual: behavior; institution: to which extent are sanitary aspects, sanitary perspectives in processes and structures taken into account | Accessibility, costs, achievement of objectives | Always dependent on project and cognitive interest | Dependent on target, this can be so called lifestyle parameters, on the one hand, but also recording | Always dependent on project, but finally one can achieve much by means of documentation |
C3. What has proven to be particularly easy to implement regarding the implementation of the evaluation of health promotion and prevention measures in your experience? | Working at eye level with those involved, creating a win–win situation, also considering the practitioner’s “gut feeling”, but not only | Considering the whole environment, using networks, going into the settings | Dependent on the measure, looking at exactly which instrument is most suitable for which target group | Low effort | Close communication with those involved, identification with the project target | “Assessment of evaluability”, involvement of all people concerned, transparency, and clearly defined targets | Using the structures given | Planning evaluation from the beginning, good definition of objectives, intervention group, defined and large enough, ideally also control group |
C4. What obstacles are to be expected in the evaluation of health promotion and prevention measures? | Time, staff, motivation, oversaturation of the practice with all kinds of evaluations from different areas, communication of the effects | Effort, costs, financial and personnel resources, understanding of the people involved | Money, appropriate contractor, familiarity with the object to be evaluated, linguistic shyness to engage oneself in something written, generally finding the access | High effort | Lack of resources, effort is often felt as stress | Refusal to participate, wrong information, wrong data, boycott in every possible form, balances of power, privacy, mistrust | The one obstacle is the recruitment, worse accessibility of specific groups of people, another obstacle in evaluation is that it is relatively difficult to create such real-life evaluations, thus one distinguishes sometimes between so called efficacy and efficiency evaluation; therefore, I can actually point out on the basis of the results, which I get in such a study, that this will then have those effects in reality | Lacking quality culture of evaluation, that evaluation is sometimes felt as a threat and not as a chance, lacking competencies and capacities, methodical difficulties, money, … Participants do not want to take part, some measures are “bound politically to succeed” |