How effective is the health promotion policy in Sichuan, China: based on the PMC-Index model and field evaluation

Abstract

Background

Many countries around the world highlight the health in all policies (HiAP). However, most of the related research focused on the influential factors and implementation strategies, with less concern on the evaluation of HiAP. In response to HiAP's call, the Chinese government has proposed health promotion policies (HPPs) in counties or districts, the evaluation of HPPs in sample counties or districts of Sichuan province in China is an essential basis for optimizing policy content, improving policy implementation, and ensuring health promotion's continuous and efficient operation.

Methods

This paper established an evaluation system for HPPs based on the PMC-Index model and then quantitatively analyzed 37 representative HPPs from the pilot areas in Sichuan province. In addition, a team of experts conducted a field assessment.

Results

The results showed that the average PMC index of 37 HPPs was 7.091, and correlation analysis showed that there was a significant correlation between the PMC index and expert score.

Conclusions

This study indicates that the overall consistency of HPPs was good and proves a connection between the formulation and implementation of HPPs.

Background

The Health in all Policies (HiAP) was proposed in 2013. According to the World Health Organization, to improve the population's health and promote health equity, the HiAP emphasizes the cross-sectoral search for synergies, the integration of health outcomes, and the avoidance of harmful health effects [1]. Now, HiAP is a term used to describe efforts to improve health by incorporating health considerations into decision-making across policy domains [2]. HiAP has become one of the basic guidelines in China's health career in the same year, which means that health has been highly prioritized in the development of public policy and incorporated into all stages of policy-making [3].

With an aging population, a growing health burden, an outbreak of infections, and increasing health inequalities, a switch from disease treatment-oriented to prevention-oriented health policy-making could help address the prevalent health challenge [4]. The world's health focus is shifting from disease-specific treatment to universal, whole-cycle health[5]. However, it is essential to point out that the health policies shift still localized in unique domain-specific, such as prevention and treatment of chronic diseases, management of mental health during the epidemic, the control of childhood obesity, etc [6]. There remains a lack of systematic understanding of the health policy reform. The intricate and complex relations among the health system elements further affirm the need for a systematic approach to tackling health system challenges. HiAP is a practical and valuable tool to conduct system governance.

Through the review over HiAP related research at home and abroad, three themes are summarized by analyzing documents: the conceptualization of HiAP, the adoption of HiAP, and the implementation of HiAP [7]. Firstly, In terms of the conceptualization of HiAP, which is conceptualized as a method, strategy or framework. For example, Yuan Yanfei disseminated and popularized the concept, development history, action framework and international experience of HiAP [8]. Peters highlighted the cross-sectoral collaboration of HiAP [9]. Secondly, in terms of the adoption of HiAP, the main focus is on the influencing factors. For example, Pintodiscussed the impact of economic factors on the implementation of HiAP [10]. Hofstad pointed out the impact of Norwegian bureaucracy on HiAP [11], and Zeeb advocated the consideration and strengthening of environmental themes in HiAP [12]. Thirdly, in terms of the implementation of HiAP, Newman applied HiAP strategy to obesity [13]. Qichao Chen applied HiAP to hospital management research [14], and Vassiliou advocated the application of HiAP to healthcare policy [15]. At present, the related concepts, strategies and measures of HiAP at home and abroad are relatively clearly clarified in different perspectives. Some developed countries have designed HiAP operating mechanisms according to their national conditions, such as the interdepartmental cooperation committee system in the Netherlands [16] and the HiAP working Group in Boston, USA [17]. However, domestic studies on HiAP are mostly concentrated in a single field, especially in the field of health, showing a narrow fragmented state of research. It must be emphasized that HiAP is a complex multi-factorial, multistage process, which gives priority to well understanding the precise mechanisms. And there's very little research about evaluations of HiAP, including using the scoping review approach to analyze the policy texts [7]. Therefore, the traditional single-method evaluations should be transferred to a comprehensive and reliable evaluation [18].

Universal health coverage is a global priority, effectively helping reduce the disease burden and promote sustainable development [19]. With a large population, it is strategically important for China to improve its national health [20]. In 2014, the Chinese government actively responded to the concept of HiAP, and launched the pilot project of HPPs nationwide that year. The project has become an important way to implement the HiAP strategy construction and realize the health of the whole people. In 2017, HPPs was fully implemented nationwide, and all sectors of society responded positively and participated fully.

This study focuses on the HPPs in Sichuan province, which is vast in size (486,000 km2) and densely populated (83.41 million), proper for being the research subject [21]. For the moment, Sichuan province has selected 37 counties/districts as pilot projects to implement the HPPs. Each region has developed its own "Regional Health Promotion Plan" based on the basic national policy. In 2017, the Sichuan Provincial Health and Family Planning Commission built a big health environment to integrate health into all policies. In 2018, Leshan City in Sichuan Province promoted health concepts to the public by introducing mass media. In 2019, Yaan City in Sichuan Province launched a health promotion site. In 2020, the People's Government of Sichuan Province made efforts to safeguard life-cycle health. In 2021, the Sichuan Health Action Promotion Committee standardized management and strengthened the monitoring and evaluation of the Healthy China Action. The HPPs in Sichuan Province has experienced from identification to promotion and deepening, and then to supervision and testing. Therefore, the next most important concern is the evaluation of HPPs in Sichuan Province, taking factors related into consideration, including whether HPPs is more consistent with similar policies? What is HPPs's dilemma in practice? What is next for improvement? Through the evaluation of HPPs in Sichuan, we can find out the problem more comprehensively, summarize the experience and form better policy pathways.

Policy evaluation is a very crucial part of the analysis of public policy [22]. The results can provide a basis for rational allocation of health resources and improvement of health policy planning. In the past, the government often introduced expert assessment as an essential means of policy assessment, but it is prone to subjective bias [23]. China's HPPs still lacks a comprehensive, easy-to-operate, scientific evaluation method of the pros and cons of such policies. A variety of approaches on policy evaluation has been adopted, but the more current cutting-edge approach is the PMC-index model.

To assess the strengths and weaknesses of policies at multiple levels, Ruiz Estrada presented the Policy Model Coherence Index in 2011 [24]. The PMC-Index model does not limit the number of secondary indicators included in evaluations [25], while visually presenting the advantages and disadvantages of policies and the level of internal consistency of policies in the form of curved graph. In this study, words extracted from the policy text, such as health, promotion, environment, service, etc., were used as criteria to further the PMC-Index model so as to be properly applied to our research. In recent years, the PMC-Index model has become more widely used and has become a popular way to evaluate policies. In the field of industrial economy, Hu Feng et al. and Xu Liying et al. respectively applied the PMC-Index model to China's robot industry and biomedical industry [26, 27]. In the environmental field, Dai et al. used the PMC-index model to evaluate the effectiveness of implementing green development policies in the Yangtze River Economic Belt of China [25]. Liu Yating et al. applied the PMC index model to study China's long-term and water pollution prevention and control policies [28]. In the field of administrative management, Zhou Haiwei et al. applied the PMC-Index model to reservoir migration policy, which has strong practical application value [29].

In conclusion, compared with other evaluation methods, PMC-Index model combines quantitative and qualitative methods, which is more comprehensive, objective and intuitive. Moreover, each district and county government in Sichuan Province continuously expands and improves regional health policies based on HiAP, which provides a reference template for the evaluation of policy consistency. Therefore, this study combined PMC index model and expert field evaluation to evaluate the consistency of HPPs in Sichuan Province and explore the correlation between policy formulation and implementation, to analyze the advantages and disadvantages of HPPs in Sichuan, and provide theoretical reference for policy optimization and innovation.

Methods

PMC evaluation

In this study, HPPs was quantified using the PMC-Index model, consisting of four steps. Firstly, we reviewed and summarized 52 key policy articles based on the theme and content of the research. Then text and co-word matrices were established by analyzing and selecting high-frequency words using ROSTCM 6.0 software. Secondly, based on the results of text mining and literature review, the primary variable was designed to meet the research needs, the secondary variable was expanded, and the evaluation system was established. Thirdly, we selected 37 HPPs from each pilot for analysis. By text analysis, the binary count method was used to assign values to the secondary variables, and then a multi-input policy table was established. Finally, the 37 HPPs were rated based on the policy consistency criteria (see Table 1). The validity, advantages and disadvantages of policy texts were evaluated by constructing three-dimensional matrices and painting a PMC surface.

Table 1 Policy consistency rating sheet

Data sources and research samples

The search of policies was carried out using the following keywords related: health in all policies, health promotion policies, and prefectural health promotion plan. We retrieved policies from the Chinese government website, the official website of the National Health Commission of China, the website named magic weapon of Peking University, and the official website of pilots. The retrieval dates were from January 1, 2013, to May 30, 2022. We have reviewed several types of policy documents, such as programs, regulations, observations, measures, plans, guidelines, etc., with the focus on those with specific action plans. After initial screening, 52 documents were obtained, including 15 national-level and provincial-level policies and 37 district-level or county-level policies from 37 pilot regions. These samples were identified as P1 to P52, and policy text excerpts are presented in Table 2.

Table 2 Policy Texts Summary

We conducted word segmentation analysis and semantic network analysis on the above 52 policies by ROSTCM 6.0, and we obtained high-frequency words including "health," "education," "service," and "management".

Ultimately, we recognized 37 HPPs from pilot areas as the research object, and the quantitative policy evaluation was carried out by using the PMC-Index model.

Classification of Variables and Identification of parameters

For the comprehensive evaluation strategy, the following aspects were taken into consideration in selecting variables in this study. Ruiz Estrada [24] proposed 10 main variables, namely research types, research direction, data source, econometric methods adopted, study scope, research theoretical framework, policy modelling by sector, economic framework, geographical analysis, paper citations and 50 subvariables. On this basis, working with the evaluation of the existing policy, for example, Tingting Jia, Siqi Zhao would talk about policy nature, policy limitation, and public policy content [30, 31]. Yuying Zou, Yi Jihave mentioned policy incentives, such as policy areas, publishing institutions, policy evaluation [32, 33]. In addition, Wenjin Zhang also mentioned policy supervision, function, and Jinming Xing also mentioned policy perspective, the policy mix [34, 35]. Finally, based on the relevant policy texts of health promotion districts and counties in China and Sichuan Province, 10 first-level variables and 44 S-level variables were determined. The first-level variables are shown in Table 3, numbered X1-X10, namely policy nature and timeliness, policy relevance, incentives and constraints, policy subject and policy content, policy evaluation and issuing institutions, policy object and policy disclosure. Then we need to set parameters for each variable. Then we need to set parameters for each variable. Binary assignments were used to give equal importance and weight to all secondary variables. More specifically, if something in the policy text is consistent with the expression of the second variable, the assignment value is 1; conversely, the assignment value is 0 [36].

Table 3 Variables Setting of Quantitative Evaluation of HPPs

Building a multi-input-output table

To quantify the values of the main-variables, a multi-input-output table was designed at this section (see Table 4).

Table 4 Multi-Input-Output Table

Measurement of the PMC-Index

There are usually four steps to calculate the PMC index. Firstly, a multi-input- output table was created to incorporate primary and secondary variables. Secondly, each secondary variable was assigned a value based on text analysis and formula (1) [2]. Thirdly, each primary variable was calculated according to formula (3). Fourthly-, the PMC index of policies was calculated by adding up the sum according to formula (4) [24, 37].

$$X\sim N[\mathrm{0,1}]$$

(1)

$$X=\{XR:[0:1]\}$$

(2)

$${X}_{i}(\sum_{j=1}^{n}\frac{{X}_{ij}}{T({X}_{ij})})$$

(3)

($i$ is recorded as primary variables, $i=1\dots n$;

$j$ is recorded as secondary variables, $j=1\dots m$;

${X}_{ij}$ is the score for $j$-th secondary variable in $i$-th primary variable;

$T({X}_{ij})$ is the number of secondary variables in $i$-th primary variable)

$$PMC=\sum_{i=1}^{n}{X}_{i}$$

(4)

($i$ is recorded as primary variables, $i=1\dots n$;

${X}_{i}$ is the score for $i$-th primary variable)

PMC-surface construction

PMC surfaces can visualize the policy's strengths and weaknesses. Since all policy documents are obtained from open websites, all policies receive a score of 1. Also, considering the symmetry and balance of the matrix, after removing P10, we created a matrix of size 3 * 3 to form a curved graph. The associated matrix is shown in formula (5) [24].

$$PMC-Surface=\left[\begin{array}{ccc}X1& X4& X7\\ X2& X5& X8\\ X3& X6& X9\end{array}\right]$$

(5)

Expert field evaluations

The Health Promotion Policy Criteria (HPPsC) includes six primary indicators (organization and management, health policy, healthy places, healthy culture, healthy environment, healthy people) and 39 secondary indicators. The expert group scored 37 pilot areas at the field research site. The maximum HPPsC score was 1000, but the total score was transformed to a percentage grading system for a visual representation of the results, with a total score of 100. In terms of quality control, firstly, experts should be trained before field evaluation, so that each expert has the same understanding and standard for each evaluation item. Secondly, many evaluation items should be objectively existing data. Thirdly, each site has more than two experts to evaluate. If there is any inconsistency between the two experts, the third expert will evaluate until the consensus is reached.

Correlation analysis

Correlations between the PMC-Index score and the field evaluation score by experts were estimated using the Person correlation coefficient by Spss26.0 software (IBM Corp). Significance tests were 2-tailed, with α=0.05.

Results

PMC evaluation

The PMC-Index of the policies

Access to existing research, the consultation of a policy's PMC index should be divided into four levels [25]. Scores range from 0 to 4.99 for poor consistency policies. Scores range from 5 to 6.99 for acceptable consistency policies. A score between 7 and 8.99 is a good consistency. A score between 9 and 10 is a perfect consistency for policy, and the more consistent the text, the better the overall policy guidance.

Then through text mining and content analysis, 37 policy texts of pilot areas were assigned, and the policy score was calculated according to the PMC index calculation formula. It is only P7 with perfect consistency. There are 17 policies with good consistency, namely P1, P5, P9, P10, P12, P17, P18, P20, P21, P23, P28, P29, P30, P31, P32, P34, and P37. There were 18 policies with acceptable consistency, P2, P3, P4, P6, P8, P11, P13, P14, P15, P16, P19, P22, P24, P25, P26, P27, P35, P36. One policy with poor consistency was P33. The average score of the PMC index is 7.091, which indicates that the policies of all districts and counties have good consistency (see Table 5). Among the scores of primary variables, the average score of X10 (policy disclosure) is the highest. The average score of X8 (issuing agency) is the lowest.

Table 5 PMC-Index

The PMC-surface of the typical policies

The three-order square matrix was constructed with X1, X2, X3, X4, X5, X6, X7, X8, and X9. In this study, one policy sample (P7, P1, P3, P33) from each level of ABCD was selected to construct the PMC matrix and draw the surface map.

$$\begin{array}{c}P7=\left(\begin{array}{ccc}1& 0.75& 1\\ 1& 1& 0.25\\ 1& 1& 1\end{array}\right)\\ \begin{array}{c}P1=\left(\begin{array}{ccc}0.857& 0.25& 1\\ 0.5& 0.8& 0.25\\ 0.5& 1& 1\end{array}\right)\\ \begin{array}{c}P3=\left(\begin{array}{ccc}0.714& 0.25& 1\\ 0.5& 0.8& 0.25\\ 0.25& 0.833& 0.6\end{array}\right) \\ P33=\left(\begin{array}{ccc}0.571& 0& 0.4\\ 0.25& 0.6& 0.25\\ 0.25& 0.5& 0.2\end{array}\right)\end{array}\end{array}\end{array}$$

The P7 showed a noticeable dent in the X4(incentives and constraints) and X8(issuing agency), with low scores for incentive restraint and release mechanisms (see Fig. 1).

Fig. 1

PMC-Surface chart of P7 (Perfect)

The P1 sagged the most in the X4(incentives and constraints) and X8(issuing authority), followed by the X2(policy timeliness) and X3(policy relevance) (see Fig. 2).

Fig. 2

PMC-Surface chart of P1 (Good)

The P3 was dented in all dimensions, with X3(policy relevance) and X4(incentives and constraints) having the most significant impact on curved depressions (see Fig. 3).

Fig. 3

PMC-Surface chart of P3 (Acceptable)

The P33 had the most dented surfaces, especially on the X4(incentives and constraints), where it scores zero points (see Fig. 4).

Fig. 4

PMC-Surface chart of P33 (Poor)

Comparison between P1 and P3

Based on the analysis above, most policies rank B or C, and policies in rank B and policies in rank C differ significantly mainly in the following dimensions: X2 (policy timeliness), X3 (policy relevance), X4 (incentives and constraints), X6 (policy content), X9 (policy objects) (see Table 6). To shed more light on the difference between B-level and C-level policies, we will compare P1 (representing B) with P3 (representing C) in some aspects.

Table 6 Comparison Between Rank B and Rank C

Policy timeliness: P1 scored higher than P3 in policy timeliness. Policy timeliness refers to the time length of policy implementation, which is an essential factor affecting the effect of policy implementation [38]. HPPs are a basic national policy in our country, indicating that it should be implemented and promoted over a long period. Specifically, as implementing health promotion district and county policies requires cross-sectoral collaboration, it takes sufficient time to set up specialized organizations and internal collaboration [39]. The policies of health promotion districts and counties involve creating healthy places, promoting a healthy culture, cultivating healthy people, and creating a healthy environment. With enough time, it can be proved that culture is deeply rooted in the hearts of the people, and the formation of healthy consciousness and habits can be promoted so as to maintain the health and friendly living environment for people.

Policy relevance: P1 scored higher than P3 in policy relevance. Policies not only concern the continuity of time but also the unity and inheritance between policies. Policy relevance is implemented through interpreting the policy content from the perspectives of breadth and depth, to identify the level of government policy formulation stance, and constantly achieve the transition from the guiding ideology to the implementation of the program [40]. Suppose the government does not strengthen the interpretation of relevant documents at all levels in the policy formulation process. In that case, there may be a shortage of policy basis, it may be challenging to get the support of the higher level of government, and there may be a contradiction between the policies.

Policy content: P1 scored higher than P3 in policy content. Based on the available physiological-psychological-social theory, many factors affect health [41]. Therefore, health promotion policies should be formulated comprehensively. We should give concern about system development and organization management, and strengthen the construction of public healthy places and a healthy environment. In addition, we should make detailed plans to spread a healthy culture and foster healthy citizens. Only by ensuring the joint efforts of each component in the policy implementation can the health promotion level of the districts and counties be improved as a whole.

Policy objects: P1 scored higher than P3 in the policy objects. From the ecological perspective, the health level of an area is assessed on all groups, so the layout of HPPs should be carried out from a more macro perspective, including enterprises, institutions, public environment, communities, families, and individuals mentioned in this study [42]. HPPs for enterprises and institutions will be radiated to the healthy transformation of various industries. Considering the construction of the healthy public environment is conducive to providing environmental protection for the physical and mental health of the residents. Community, family, and individual HPPs are the last mile and the final destination for improving the health of the residents.

Expert field evaluations

Naming each pilot region using Arabic numerals. The name of the pilot area corresponds to the name of the policy. For example, the policy issued by pilot area 1 is P1 (see Table 7).

Table 7 Expert Field Evaluations

Correlation analysis

The PMC-Index score of the variable and the total score of field evaluation by experts both were continuous variables undergoing the normality test. After correlation analysis, r=0.415 and P= 0.011 (P < 0.05) were obtained, from which it could be concluded that there was a significant correlation between the PMC-Index score and the total score of field evaluation by experts (see Fig. 5).

Fig. 5

The Relationship Between PMC Index and Total Score of Field Evaluation

Discussion

The overall evaluation of HPPs

The policy is the guidance of actions [43]. When analyzing HiAP's practical logic, some researchers pointed out that "health first" was the premise in politics [44], and it’s the practical way to formulate health promotion policies from the central government to local governments, targeting at realizing the "Healthy China 2030" strategy. Quantitative evaluation on local health promotion policies is helpful to evaluate whether the influence of policies extends to the grass-roots level and provides scientific basis for deepening health promotion policy reform.

As a strong province of population, economy and culture in western China, Sichuan Province is a demonstration site of healthy culture construction. Therefore, the evaluation of HPPs in this region has important reference value for the formulation of national HPPs. In this study, the health promotion policies of 37 districts (counties) were quantified and evaluated. Overall, the average score of the PMC index was 7.091, indicating that policy consistency was good, and more than 90% of the policy consistency was at a reasonable and acceptable level. However, there is still a gap between policy consistency from these policies and excellent consistency, which shows that there exist some flaws in the process of local policy formulation. In other relevant studies, the PMC index of most studies is acceptable consistency or good consistency, and the results of this study are consistent with those of other studies. This may be related to the fact that local policies are detailed implementation rules by following central policies and local conditions.

The main factors affecting PMC index score

According to the evaluation dimensions, X2 (policy timeliness), X3 (policy relevance), X4 (incentives and constraints), and X8 (publishing agency) mainly affect the PMC index score.

Most 37 sample policies are temporary and short-term (P1, P2, P3, etc.). However, such health threats as obesity, smoking, diabetes, air pollution, etc., require long-term policy guidance and prevention to make a difference [45]. In addition, some researchers pointed out that the continuity of policies is the task requirement of policy implementation in the new era, and local policies should actively cooperate with the central policies. "Healthy China 2030" is a comprehensive policy covering the short, medium, and long term, so local governments must strengthen the formulation of long-term health promotion policies [36].

The policy relevance can reflect the local government's priority to the superior policy and whether the local policy formulation has a scientific basis. In this study, most sample policies had the best correlation with national-level policies and the worst correlation with municipal-level policies. Some researchers analyzed that the policy is a top-down hierarchical diffusion model many times [46], and the specific policy-making path is meta-policy-basic policy-specific. The districts (counties) strengthen the correlation with municipal-level policies, which is conducive to ensuring the scientificity of the policies and building an excellent political alliance, so as to improve the probability of policy success [47].

The 37 sample policies included in this study seldom involve incentives. From the perspective of policy instruments, appropriate, effective, and incentive policies can promote political stability and economic development [48, 49], but they lack supply-oriented and environmental policy tools in the study. Therefore, in order to ensure the smooth implementation of the health promotion programme, it is necessary to formulate tax relief, administrative approval priority, talent incentives, financial incentives, etc., to motivate all units to participate in health promotion activities and expand the scope of health promotion [50].

From the perspective of publishing agencies, 37 sample policies are all published by government offices, without multi-department joint publication of policies. However, many studies have proved that cross-department collaboration is crucial in promoting HiAP, and the same is valid for health promotion areas [51]. In the process of policy promotion, the health sector cannot be neglected. In fact it plays a leading role, so all relevant departments should actively interact and cooperate [52]. Moreover, the health sector should seek and create shared interests with relevant departments to establish a platform to solve problems [53]. In the future, all districts (counties) should establish a cooperative working mechanism: "government is responsible for leading, departments are responsible for cooperation, and industries are responsible for implementation".

A significant correlation between policy formulation and implementation

According to the correlation analysis, there is a significant correlation between the PMC index score and the total score of field evaluation, which indicates a correlation between policy formulation and implementation, and proves the importance of establishing an integrated mechanism for policy formulation and implementation [54]. For example, the personnel involved in implementing any policy must have sufficient knowledge of the policy, even those involved in policy implementation must also be involved in the process of policy formulation.

Through the analysis of policy consistency and the correlation between policy formulation and policy implementation, this study theoretically provides a reference basis for HPPs policy formulation and enriches the evaluation research of HPPs. In fact, integrating health into all policies is an important and complex project, which refers to integrating the three levels of government, society and individuals to form a strong synergy to maintain and promote health [55]. In policy-making, local governments should follow the template of policies, strengthen departmental collaboration, and promote policy cohesion. In policy practice, all industries in the society follow HPPs related rules and regulations to promote the horizontal and vertical development of HPPs. At the same time, through various forms, such as publicity, cultural activities, increase of sports equipment and facilities, it is advocated to integrate HPPs into personal life, to promote the health of the whole society.

Limitations

However, the policies selected in this study may have some limitations due to policy openness and website construction, such as insufficient representation. In the future, we can optimize the problem by cooperating with government departments or using crawler mining to achieve more objective and scientific policy evaluation.

Conclusions

Based on PMC - index model, HPPs evaluation system related to the counties/districts of Sichuan province is established in this study through conducting the quantitative analysis of HPPs in the counties/districts of Sichuan province, and combining the expert field evaluations, which explored the Sichuan HPPs consistency as well as the correlation of policy formulation and implementation, for the continuous reform of HPPs to provide theoretical reference.