The present project examined perceptions of scientists in different fields related to public health about the requirements of a tool to measure the societal impact of publications. Based on the insight gained in the focus groups and based on the repeated applications of the tool developed to scientific studies, it proposes a practical tool to measure the societal impact factor of publications. The tool comprises an assessment of the societal impact of a publication at 3 dimensions: (1) the aim of the publication, (2) the efforts of the authors to translate their research results into societal action or practical reality, and (3) the accomplishment of the translation in terms of the geographical reach (level), status (preliminary or permanent) and size of the target group of this translation. A first exercise to evaluate the comprehensibility, relevance and practicability of the tool, which have been perceived as crucial features of a suitable tool in the focus groups, yielded encouraging results. The time needed by the reviewers for the assessment of the societal impact varied considerably between the experts and the different publications.
The tool developed provides a novel complementary view to earlier projects by exploring the perceptions of scientists working in the field of public health. Compared to previously proposed assessment models, the results of the focus group discussions yielded an assessment model that was largely research-output oriented and is consequently suitable for post-hoc analysis of translation outcomes but not for prospective assessments of possible future translations of research, which are targeted at in other proposed assessment models . In most cases, the societal impact of a publication is reached after the publication of the related study, and develops with time, often depending on the efforts of the researchers to translate their findings into societal action. The societal impact factor assignable to a publication may consequently increase over time, e.g. due to a transition of the translation accomplished from a temporary to a permanent status, or due to an expansion of the translation to the international level following a translation at the regional level. To take future developments into account, authors who apply for a societal impact factor should explicitly be invited to submit their publications for reassessment in case that there is some progress in the translation status.
One of the main differences between the current and earlier approaches is that the proposed tool does not specify which kinds of translations (e.g., which resulting products, e.g. technical devices, guidelines, patents, etc) are specifically considered as accomplished translation. For example, a recently published model differentiates between biological materials, databases, patents, pharmaceutical products, curriculum guidelines, medical devices and many more, which all can be considered some kind of translation outcome . In the present tool, it is completely left to the applicant to describe what his or her translation accomplishment actually is--the only prerequisites made are that there must be some documentation of a causal link between publication and the described beneficial translation outcome, and that the translation is accomplished in a non-scientific area relevant to the society. Compared to other definitions of societal impact used in the literature, the definition used here is wider than in other impact assessment models. This approach makes the tool easier to handle and applicable to different situations, research areas and practice fields. The reasoning behind the exclusion of impact in other scientific areas (as for example indicated by the number of citations of the paper in scientific journals) from the societal impact assessment in this model was based on earlier suggestions that scientific citation-based assessments and societal impact assessments should be synergistic and complementary .
Several adaptations and specifications to the developed instrument were made based on the feedback from the external experts. Importantly, the aim of the publication was deemed to be a necessary but - alone - not a sufficient criterion for achieving a societal impact. Additional points in either dimension 2 (efforts undertaken) or dimension 3 (accomplished translation) were deemed necessary to achieve a societal impact factor. A further important specification was made for the 'status of the translation' in dimension 3. Here, it was deemed most suitable not to define a minimum duration of 'permanence' in the light of the ever-changing state-of-the-art practices in biomedical fields. Rather, a translation should be considered permanent if it is part of the official state of the art practice in its subject area at a specific point in time.
The third aspect in dimension (3), that is, (c) 'target group' (1 point: individuals, 2 points: subpopulation, and 3 points: the public) required further specification of the terms used for the categories subpopulation and individuals. In the present form of the instrument, the category 'subpopulation" should be used for any translations targeting directly a definable group of people (e.g., migrants; mentally ill people, depressed patients), whereas the category 'individuals" is for translations affecting individual cases of patients which cannot be described as a coherent group. The latter category may, e.g., be appropriate from the translation of a published case study.
Strengths and limitations
A strength of the present research is its focus on the perceptions of scientists related to the societal impact of health-related publications, with subsequent translation of the gained insights into a practical tool to assess the societal impact of publications. The focus group interviews provided a relatively quick method to explore opinions in an interactive context and to develop creative problem-based solutions . The repeated applications of the drafted tool to scientific publications and the expert feedback resulted in the identification of several practical issues and subsequent adaptations and refinements that are useful for further research activities and practical applications.
The present study also has several limitations. The applied methodology of using focus group meetings has been criticised in the literature for not taking account of hierarchical structures in groups which influence the contents produced by the groups . In spite of the fact that opinion leaders who may have a high influence on the other participants were purposefully sampled into one group and the moderator (MM) encouraged all participants to take equal part in the process, some related bias in the outcome cannot be ruled out.
Furthermore, the tool developed can be used as an informative practical exercise, but it cannot be considered to be ready for routine implementation. Adaptations and changes will be necessary before it can be used in various applied research fields, and it may turn out to be inadequate for specific fields that were not considered in the present study. Also, the quantitative point assignments to the different categories of the included items are preliminary at the present stage. A categorization into "low impact", "medium impact" and "high impact" is planned for the future when more assessments based on the tool become available. A quantitative assessment seems required to be able to implement the tool in academic settings that are currently (nearly) exclusively considering quick and easy-to-use quantitative assessments of research outputs [5, 8].
Implications for future research
Focus group discussions seemed particularly suitable for the present study based on the small amount of preknowledge on the opinions and beliefs of researchers on the societal impact of their research publications . Future work may use the ideas generated in this analysis to conduct individual interviews or to use consensus-building methods such as Delphi approaches, which can overcome some of the shortcomings of group discussions . The analyses may be expanded to scientists from other fields than public health sciences. Additionally, other stakeholders who contribute to the formation of the societal impact may be included as participants. The tool developed in the present study acknowledges the contribution of other groups of stakeholders than scientists only indirectly by considering interdiscipinary (often non-scientific) work such as media collaborations and policy development.