 Research
 Open access
 Published:
Evaluation of the quality of COVID19 prevention and control by a novel comprehensive evaluation model in a tertiary general hospital: a prospective observational study
BMC Public Health volume 21, Article number: 2022 (2021)
Abstract
Background
Prevention and control (P&C) of Corona Virus Disease 2019 (COVID19) is still a critical task in most countries and regions. However, there are many single evaluation indexes to assess the quality of COVID19 P&C. It is necessary to synthesize the single evaluation indexes reasonably to obtain the overall evaluation results.
Methods
This study was divided into three steps. Step 1: In February 2020, the improved Delphi method was used to establish the quality evaluation indexes system for COVID19 P&C. Step 2: in March 2020, the CRITIC method was used to adjust the Order Relation Analysis (G1) method to obtain the subjective and objective (S&O) combination weights. The comprehensive evaluation value was obtained using the weighted Efficacy Coefficient (EC) method, weighted TOPSIS method, weighted ranksum ratio (RSR) method, and weighted Grey Relationship Analysis (GRA) method. Finally, the linear normalization method was used to synthesize the evaluation values of different evaluation methods. Step 3: From April 2020 to May 2021, this evaluation method was used to monitor and assess COVID19 P&C quality in critical departments prospectively. The results were reported to the departments monthly.
Result
A quality evaluation indexes system for COVID19 P&C was established. Kendall’s consistency test shows that the four evaluation method had good consistency (χ^{2} = 43.429, P<0.001, Kendall’s consistency coefficient = 0.835). The Spearman correlation test showed that the correlation between the combined evaluation results and the original method was statistically significant(P < 0.001). According to the MannKendall test, from March 2020 to May 2021, the mean value of COVID19 P&C quality in all critical departments showed an upward trend (P < 0.01).
Conclusions
The combined comprehensive evaluation method based on the S&O combined weight was more scientific and comprehensive than the single weighting and evaluation methods. In addition, monitoring and feedback of COVID19 P&C quality were helpful for the improvement of P&C quality.
Background
The COVID19 pandemic has been the world’s largest infectious disease pandemic in decades [1, 2]. At present, the COVID19 epidemic in China has been controlled and entered the regular prevention and control (P&C) stage of “Outside defense import, inside defense rebound” [3]. However, the COVID19 epidemic in most countries and regions outside China is still severe. By April 25, 2021, more than 146 million confirmed cases of COVID19 [4] had been reported outside China, more than 1500 times the total number of confirmed cases in China. The success of China’s fight against COVID19 is related to the implementation of scientific P&C measures, ranging from a province or a city to a community or a medical institution.
From the perspective of a medical institution, to do well in COVID19 P&C, it should first understand the current situation of COVID19 P&C in their institutions. Then, it is easy to grasp the key and take targeted improvement measures to continuously improve work quality, which requires regular evaluation of the quality of P&C work. However, there are many indicators to evaluate COVID19 P&C work. For example, there are process indicators, including hand hygiene (HH) compliance rate, disinfection qualification rate, personal protective equipment (PPE) reserve complete rate, complete rate of protective facilities, COVID19 P&C knowledge training rate, COVID19 P&C knowledge awareness rate, the compliance rate of various COVID19 P&C measures implementation. There are also outcome indicators, including the incidence of nosocomial infection, occupational exposure, and adverse events. The values of multiple indicators vary from high to low, and it isn’t easy to evaluate the overall quality of work only by comparing a single index. Therefore, a comprehensive evaluation of all indicators is needed to comprehensively and accurately reflect COVID19 P&C work quality.
The comprehensive evaluation method refers to a systematic and standardized method to evaluate multiple indexes and units simultaneously. It is generally divided into three steps: 1. Select evaluation indexes and establish an evaluation index system. Whether the comprehensive evaluation result is objective and accurate depends on whether the comprehensive evaluation index is accurate and comprehensive. Therefore, the selection of an evaluation index is an essential basic work in a comprehensive evaluation. 2. The evaluation indicators should be changed to dimensionless and trendless to eliminate the influence of dimensions and trends. 3. Determine the weights of evaluation indicators. The index is weighted by weight to reflect the relative importance of the index. 4. Summary and synthesis of comprehensive indicators. According to the standard value, evaluation method, and each indicator’s weight in the index system, each evaluation object is summarized and synthesized into an index, and the value of the synthesis index is compared to judge the quality of each object.
The application field and scope of comprehensive evaluation are vast. It is widely used in natural science, social science, and economics to evaluate the characteristics and properties of various things. Such as comprehensive evaluation of environmental monitoring, drug clinical trials, social security, quality of life, economic benefits, and production mode.
It also plays a vital role in health management. During the COVID19 pandemic, some literature reported the issue of prioritizing COVID19 patients using multiindicator decisionmaking. For example, Mohammed Ma et al. [5] used EntropyTOPSIS method for evaluation and benchmarking methodology to select the best classifiers for COVID19 diagnosis. Majumder P et al. [6] presented a realtime death assessment monitoring of COVID19 using TOPSIS MCDM to choose the most critical risk factors and applied GMDH to estimate death value within all confirmed cases. Singh R, et al. [7] suggested that many activities should not be performed during the COVID 19 pandemic period. These activities were considered as criteria and then applied to an analytic hierarchy process (AHP) to calculate their weights and assigned ranks/priorities according to their effect. Albahri As et al. [8] used AHP and Vikor to get COVID19 patient prioritization dependent on their health conditions.
Furthermore, some methods had been reported in the literature to evaluate nosocomial infection P&C quality comprehensively. For example, Yun Z [9] evaluated hospital infection P&C quality using the relative distance and AHP. Suner A et al. [10] collected expert opinions regarding the criteria influencing the best HH preference. Afterward, these opinions were examined with the MultiAttribute Utility Theory (MAUT) and the AHP. The result indicated that rubbing the hands with ABAS was the most favorable choice for IDCM specialists to prevent nosocomial infection. Xiaomeng C [11] evaluated ICU target monitoring in a hospital from 2010 to 2016 by the RSR method. However, these studies listed above used only a single weighting method and one or two results synthesis methods.
In the comprehensive evaluation process, the relative importance of indicators, i.e., the index’s weight, significantly influences the results. The determination methods of the index’s weight can be divided into subjective and objective weighting methods [12]. The subjective weighting method takes complete account of the personal wills of the evaluators but ignores the nature of the data. Therefore, in some cases, the evaluation objects can’t be effectively distinguished. The objective weighting method fully considers the nature of data, such as correlation and difference. However, it lacks subjective control and often produces results inconsistent with the facts. Therefore, only a good combination of S&O weights can reflect the evaluation’s comprehensiveness and scientificity. In addition, different results may be obtained due to different principles of various evaluation methods and different sensitivities of indicators to evaluation methods. Therefore, it is inappropriate to use only one evaluation method. At present, it is the development trend to apply various methods in a comprehensive evaluation [13].
Therefore, in this study, the combined evaluation methods weighted by S&O combined weights were put forward to improve the scientific nature of evaluation and the stability of results. Using this method to evaluate the quality of COVID19 P&C, the overall situation and focus of COVID19 P&C work can be grasped more scientifically and accurately, and targeted measures can be taken to improve the quality of COVID19 P&C work continuously. This study will share an example of a tertiary general hospital in Shenzhen, China, using this approach to assess and improve the quality of COVID19 P&C work since the COVID19 pandemic. Furthermore, it provides a reference for medical institutions to do well in COVID19 P&C.
Methods
Study institutions and design
The study was carried out from February 2020 to May 2021 in a tertiary general hospital with 1350 beds, 1115 doctors, and 1334 nurses in Guangming District, Shenzhen, Guangdong, China. It was divided into three steps. The first step was the establishment step of the evaluation indexes system: In February 2020, all single indicators for evaluating the quality of COVID19 P&C were first listed, questionnaires were made, representative evaluation indicators were screened by the Delphi method [14], and an evaluation indexes system for the quality of COVID19 P&C was established. The second step was the establishment of the evaluation method step. In March 2020, the evaluation indexes data were collected, and the objective weight obtained from the CRITIC method was used to adjust the G1 subjective weighting method. As a result, the S&O combined weights were obtained.
Then, the comprehensive evaluation value was obtained by four methods weighted by the S&O combined weights: weighted EC method, weighted TOPSIS method, weighted RSR method, and weighted GRA method. In addition, the methods of pre and post combination tests evaluated the consistency of the results. Finally, the linear normalization method was used to synthesize the outcomes of various evaluation methods to obtain the value of COVID19 P&C quality in critical departments. The third step was the prospective monitoring, evaluation, and feedback step. This evaluation method was used to prospectively monitor and evaluate the COVID19 P&C quality in the critical department from April 2020 to May 2021. The evaluation results were reported to the departments monthly. The study flow chart is shown in Fig. 1.
Establishment of evaluation indexes system
The modified Delphi method [14] was adopted. First of all, all indicators of COVID19 P&C quality evaluation were preliminarily listed through national diagnosis and treatment guidelines [15], P&C plans [16], and relevant literature search [17], including three firstlevel indicators, 17 secondlevel indicators, and 48 thirdlevel indicators. Then, an expert consulting group consisting of 11 people was established, including three people in the field of infection and control, two people in the area of medical treatment, two people in the field of nursing, two people in the field of epidemiology, and two people in the field of medical laboratory science. All the experts were from our hospital, and had been engaged in relevant profession for more than 10 years, and were proficient in their profession. An expert consultation table was prepared, and the Likert 5level scoring method [18] was used to score the purpose, completeness, operability, and independence [19] of the indicators one by one. Each attribute of the indicators with 1–5 points was successively increased. Experts ranked the importance of the indicators on a scale of 1 to 10. The experts selfevaluated the judgment basis and familiarity of the indicators by using the preestablished scoring criteria. An expert supplementary opinion column is designed in the consultation table, and experts can add, modify, and delete various indicators. The number of indicators removed by each expert must be less than 50% of the total indicators. Finally, the threshold value method was used to screen the evaluation indexes. The full mark frequency, the arithmetic mean, and the coefficient of variation were calculated according to each index’s scores.
The calculation method of the threshold value for full mark frequency and score’s arithmetic mean are as follow:
The calculation method of the threshold value for score variation coefficient is as follow:
Indicators with scores below the threshold value were retained and vice versa. To prevent essential indicators from being removed, only three scales that did not meet the criteria at the same time would be released. If one or two scales did not meet the criteria, the choice should be made after a discussion based on the principles of purpose, completeness, operability, and independence.
After two rounds of expert consultation, the quality evaluation indexes system of COVID19 P&C was finally determined. The expert consultation results were measured by the expert positive coefficient (Response rate), degree of expert opinion concentration (Mean value), degree of specialist opinion coordination (Coefficient of variation and coordination), and expert authority coefficient (Cr). The Cr is the arithmetic mean of the judgment coefficient (Cs) and the familiarity coefficient (Ca). The calculation formula is as follow:
Indicators data collection
The process indicators, such as the HH compliance rate, consumption of HH products, awareness rate of COVID19 P&C knowledge, and compliance rate of various infection control measures, resulted from manual investigation and monitoring conducted by professionally trained infection control professionals twice a week. Moreover, the outcome indicators, such as the incidence rate of nosocomial infections and the number of occupational exposure cases, were derived from the nosocomial infection information system. The system has a data interface with the hospital information system (HIS), laboratory information system (LIS), radiology information system (RIS), mobile nursing system, operating room anesthesia information system, and could obtain relevant data.
The objective weighting of indicators (CRITIC method)
The CRITIC method is an objective weighting method proposed by Diakoulaki [20]. Its basic idea is to determine the objective weight of indicators based on two fundamental concepts. The first is the contrast intensity, which represents the value difference of each evaluation object in the same index and is expressed in the form of standard deviation. The larger the standard deviation, the more significant the value difference of each evaluation object. The second is the conflict between indicators, which is based on the correlation between indicators. If there is a strong positive correlation between two indicators, the conflict between two indicators is low.
The quantification index of conflict between the j index and other indexes is:
where r_{tj} is the correlation coefficient between the evaluation indexes t and j.
Therefore, the conflict and correlation of data are considered more comprehensively in this method than in other objective weighting methods. Let the C_{j} represents the information content contained in the j evaluation index, then C_{j} can be expressed as:
Where j = 1,2, …, m; the δ_{j} represents the standard deviation of the j valuation index.
The larger the C_{j} is, the more information in the j evaluation index is, and the greater the relative importance is. Therefore, the objective weight of the j index is as follows:
Where j = 1,2, …, m.
Adjusting the calculation of subjective weight with objective weight
The G1 method is a subjective weighting method proposed by Professor Yajun G through improved the AHP method [21]. Compared with AHP, it is no need for a consistency test. And it solves the problem that it is difficult to judge and reach a consensus on the relative importance of indexes in AHP when there are many indexes.
The order relation of each index was determined by each index weight obtained during the establishment of the evaluation indexes system: x_{1} > x_{2} > … > x_{m}. Then, the objective weights obtained above were used as the comparative judgment of the relative importance between x_{k − 1} and x_{k}:
Where k = m, m − 1, ⋯, 3, 2.
Then the combined weight of the m index was:
From the v_{m}, the m1,m2…, 3, 2 indicator weights were:
Where j = m, m1, m2, …, 3, 2.
In the formula, the v_{j − 1} was the G1 combined weight modified by the objective weight of the j1 index, the r_{j} was the ratio calculated through Eq. (8), and the v_{j} was the G1 combined weight modified by the objective weight of the j index.
Calculation of comprehensive evaluation value
Weighted GRA method [22]
The GRA was established and developed by Professor Deng Julong from Huazhong University of Science and Technology in the 1980s. Its basic idea is to distinguish the correlation degree by analyzing and comparing the influence of the sequence index’s change on the reference index. The analysis steps are as follows:

(1)
Determine the reference sequence
According to the meaning of the evaluation index, the optimal value of each index is selected from n evaluated objects to form a reference sequence x_{0j}. If j is a positive indicator, the bigger, the better, then x_{0j} is the maximum value of the j index of the n evaluation object; if it is a reverse indicator, it is the minimum value; if it is a moderate indicator, it is the moderate value of the index.

(2)
Calculate twolevel maximum difference ∆(Max) and twolevel minimum difference ∆(min)
Calculate the absolute difference sequence between each evaluation object sequence and the optimal reference sequence, and the calculation formula is as follows:
Where i = 1,2, …,n, j = 1,2, …,p.
On this basis, according to Eqs. (12) and (13), twolevel maximum ∆(Max) and twolevel minimum difference ∆(min) can be obtained.

(3)
Calculate the correlation degree
The correlation coefficient is calculated according to the formula:
Where the ρ (0 < ρ < 1) is the resolution coefficient, and the ρ for this study was 0.05.

(4)
Calculate the grey correlation degree according to the formula:
Where w_{j} is the weight of item j.

(5)
Permutation correlation degree
Since the E_{i} reflects the degree of correlation between the i evaluated object and the evaluation standard sequence x_{0j}, if E_{i} > E_{j}, it indicates that the i object is better than the j object. So the E_{i} can sort and compare the evaluated objects.
Weighted EC method [23]
EC method also called the efficiency function method, is based on the principle of multiobjective programming to determine a satisfactory value and a disallowed value for each evaluation index, with the satisfactory value as the upper limit and the disallowed value as the lower limit. The degree of each indicator’s satisfaction is calculated. Then the score of each indicator is determined based on this. The comprehensive status of the studied object is evaluated through the weighted average. The calculation steps are as follows:

(1)
Determine the satisfactory value and the disallowable value of each evaluation index.
The satisfaction value of the index is χ_{hij} (i = 1,2…, n. j = 1,2…, m) and the unallowable values of the index is χ_{sij} (i = 1,2…, n. j = 1,2…, m).

(2)
Calculate the EC of each index according to this formula:
Where the d_{ij} represents the i (i = 1, 2, …, n) EC values of the j (j = 1,2…m) evaluation indexes. The x_{ij} represents the i (i = 1, 2, …, n) measurement values of the j (j = 1, 2, …, m) evaluation indexes.

(3)
The total EC of each evaluation object is calculated:
Where D_{i} represents the total EC value of the i (i = 1,2…, n) evaluation object; The d_{ij} denotes the j (j = 1,2…, m) power coefficient values of the i (i = 1,2…, n) evaluation indexes; The w_{j} represents the j (j = 1,2…m) weight of evaluation indexes.
The evaluation object is evaluated according to the total EC value. Thus, the greater the total EC value is, the better the comprehensive condition of the evaluation object is.
Weighted RSR method [24]
The method of RSR was proposed by Professor Tian Fengtian, a Chinese statistician, in 1988. The statistical analysis method, which combines the advantages of classical parametric statistics and modern nonparametric statistics, is not only suitable for the comprehensive evaluation of fourgrid data but also for row × table data, as well as for metrological data and classified data. The calculation steps are as follows:

(1)
Rank compiling
The rank of the high superior index is ranked from small to large. That is, the minimum value of the index is ranked as rank 1, the small value of the index is ranked as rank 2, …, the maximum value of the index is ranked as rank n; On the contrary, the low superior index varies from large to small rank. The mean rank is used for those with the same index value. The average rank is used for the intermediate superior index.

(2)
The weighted RSR is calculated:
Where the i = 1,2…, n, j = 1,2…, m, the R_{ij} represents the rank of the element in row i and column j, and the W_{j} means the weight of the j evaluation index.
The value of WRSR is used to determine the merits and demerits of the evaluation object. Thus, the larger the WRSR is, the better the evaluation object is.
Weighted TOPSIS method [25]
C.L. Hwang and K. Yue first proposed the TOPSIS method in 1981. TOPSIS is a method to rank a limited number of evaluation objects according to their proximity to the ideal goal, which is to evaluate the relative merits of existing objects. The calculation steps are as follows:

(1)
Indexing the same trend
The counting backward technique is used to change the low superior indexes to high superior indexes.

(2)
The dimensionless of the indexes
After the same trend, the original data matrix is normalized according to the following formula to eliminate the influence of indexes’ measurement and establish a normalized matrix Ζ.
When the original data is a high superior indicator:
When the original data is a low superior indicator:

(3)
Determine positive and negative ideal solutions
According to the normalized matrix Ζ, the positive ideal solution (optimal vector) is obtained:
And the negative ideal solution:
Where i = 1,2…, n. j = 1,2…, m. \( {Z}_{i1}^{+} \) + and \( {Z}_{i1}^{} \) represent the maximum and minimum value of the evaluation object in the j index, respectively.

(4)
Calculate the Euclide distance
\( {D}_i^{+} \) + and \( {D}_i^{} \) between the index values of each evaluation object and the positive and negative ideal solutions:
Where the w_{j} represents the weight of index j.

(5)
Calculate the C_{i} value
The C_{i} value of the relative proximity between the index value of each evaluation object and the positive and negative ideal solution is calculated:
According to the C_{i}’s relative proximity coefficient, the order of the merits and demerits of evaluation objects is sorted. The value range of C_{i} is [0,1]. The closer the C_{i} value is to 1, the closer the evaluation object is to the positive ideal solution. Conversely, the closer the C_{i} value is to 0, the further away the evaluation object is from the positive ideal solution.
Precombination test
The Kendall rank correlation [26] was used to test whether the ranking results of every single evaluation method are consistent. Only evaluation methods that passed the precombination test could be used for combinations.
Synthesis of comprehensive evaluation results (information synthesis method)
Constructing the matrix of comprehensive evaluation results
The results obtained from the comprehensive evaluation method passed by the precombination test, such as total EC, WRSR, C_{i} value, and GR degree, are summarized to form the evaluation result matrix. The evaluation result matrix is marked by (x_{ij})_{m × n}, where x_{ij} represents the evaluation result of the j object by the i evaluation method. The m represents m evaluation methods, and the n represents n evaluation objects.
Equation (26) is used to linearize the evaluation result matrix by row
Where y_{ij} represents the value of each evaluation object after normalization.
Calculate the mean normalized values of each evaluation object according to Eq. (27)
Rearrange according to the size of \( {\overline{y}}_j \) value. The larger the value is, the better the evaluation object is.
Postcombination test
The Spearman rank correlation test method [26] was used to test the degree of closeness between the ranking results obtained after the combination and the results obtained from the initial method. If the order of the combined evaluation method was inconsistent with one of the evaluation methods in the combination, the method should be eliminated and then recombined.
Statistics and analysis
Input the data into Excel 365 to establish the database. In excel, The indexes weighting and comprehensive evaluation were completed, and the MannKendall [27] was used to test the trend of critical departments’ comprehensive evaluation value. Descriptive analysis, Kendall consistency test, and Spearman rank correlation test were performed using SPSS 26.0 (Armonk, NY, USA: IBM Corp) software. Continuity variables conforming to normal distribution were described by mean ± standard deviation. The nonnormal distribution data was described by the median (interquartile interval). Pvalues of less than 0.05 were considered significant.
Results
Establishment of COVID19 P&C quality evaluation indexes system
After two rounds of letter consultation by 11 experts (average age: 36 ± 3.5 years), the indexes system for the quality evaluation of COVID19 P&C was finally determined, including one firstlevel indicator, eight secondlevel indicators, and 13thirdlevel indicators. The composition and the calculation formula of the indicators are shown in Table 1.
In the first round of expert consultation, 11 questionnaires were sent out, ten were returned, and all of them were valid, with a response rate of 90.91% and a valid rate of 100%. Six experts put forward a total of 12 opinions. Ten questionnaires were sent out in the second round, ten were returned, and all were valid. The response rate and valid rate were all 100%. A total of four opinions were put forward by three experts, which indicated that experts were more active in the two rounds of letter consultation. The authority coefficients of the two rounds of expert consultation were all greater than 0.8, indicating that the authority of the experts selected in this study was relatively high, and the results had certain representativeness. The Kendall consistency test showed that the coordination coefficient greater than 0.3 (P < 0.0001) indicated a strong evaluation consistency. The mean value, standard deviation, and full mark ratio are shown in Table 2. Indicator selection was determined by the threshold values of full mark frequency, mean value, and coefficient of variation in the two rounds of letters consultation. The values of the two rounds are shown in Table 3.
The combination of the S&O weighting of evaluation indexes
According to the data in March 2020, the indicators’ objective weights were obtained, and the subjective weight calculation process of the G1 method was adjusted with the objective weight. As a result, the combined S&O weights of each evaluation indicator were obtained. The weighted rankings of each index of the combined weight were the same as the subjective weighting method, reflecting the personal wills. In addition, the size of the combined weight was finetuned by the objective weight, reflecting the nature of the data. The weighting process of the S&O combination is shown in Table 4.
The evaluation results of each comprehensive evaluation method
The evaluation results of the weighted EC method, weighted RSR method, weighted TOPSIS method, and weighted GRA method for each evaluation object were different, as shown in Table 5.
Precombination test
The Kendall consistency test of the four comprehensive evaluation methods on the evaluation objects showed the Kendall synergy coefficient was 0.835 (χ^{2} = 43.429, P < 0.001), more significant than 0.8. Thus, the evaluation consistency of the four methods was good, and the results of the four evaluation methods could be integrated.
Synthesize the results of various comprehensive evaluation methods
After linear normalization, the mean value of evaluation results was obtained. Then the P&C quality of COVID19 in each department was ranked by the mean value. The top three departments were endoscopy, ophthalmology, and emergency department, while the bottom three departments were delivery room, stomatology, and pediatrics, as shown in Table 6.
Postcombination test
The correlation coefficients between the combined evaluation method and each evaluation method were above 0.8, with statistical significance, as shown in Table 7.
The quality trend of COVID19 P&C in critical departments
The mean quality value showed an upward trend by the Mann Kendall test, as shown in Table 8, Figs. 2, 3, and 4.
Discussion
COVID19 P&C is currently one of the critical tasks in most countries and regions. The essential point of P&C work is implementing the scientific P&C norms and guidelines. However, to evaluate the quality of implementation, a single evaluation index is often not comprehensive enough. Multiple indicators are difficult to judge the overall situation. Therefore, multiple indicators should be integrated to obtain a scientific and comprehensive evaluation result [28].
An essential function of the multiindex comprehensive evaluation method is “dimensionality reduction,” that is, many index values are integrated into a single evaluation value to facilitate comparison. However, if too many indicators are involved in the evaluation, a correlation may exist, affecting the results. It also increases the workload and difficulty in the assessment. Therefore, it was necessary to choose the representative index by discarding the dross and selecting the essential to establish a scientific evaluation system. This study thoroughly considered the evaluation indexes’ purpose, completeness, operability, and independence through two rounds of the Delphi letter consultation method. Therefore, the opinions of evaluation experts could be consistent and unified to establish a quality evaluation indexes system of COVID19 P&C in line with the actual situation of our hospital.
At present, there are more than ten commonly used methods of the multiindex comprehensive evaluation methods, such as expert opinion method, Delphi method, AHP method, G1 method, coefficient of variation method, EC method, entropy method, maximum deviation method, RSR method, TOPSIS method, GRA method, principal component analysis method, factor analysis method, and fuzzy evaluation method. Different evaluation methods may obtain different results due to diverse principles and the sensitivity of indexes to evaluation methods. Therefore, it is not possible to use only one evaluation method for evaluation. For minimizing the impact of evaluation methods on evaluation results, four methods, including the weighted EC method, weighted RSR method, weighted TOPSIS method, and weighted GRA method, were adopted in this study to evaluate the quality of COVID19 P&C in critical departments. The evaluation results of different methods were reasonably integrated, and the pre and post combination testing ensured the reliability and stability of the results.
Traditional weight combination methods generally include 1. Addition synthesis methods, such as the linear weighting method with preference coefficient proposed by Liu X et al. [29], and R. Md Saad et al. [30] proposed the minimum difference method between S&O evaluation information and the ideal scheme. However, the additive synthesis method is too rigid, which is not conducive to treating the superior and inferior information in the S&O weight information and is not conducive to the rational explanation of the combined weights. 2. Multiplication method. For example, Chen Ch [12] used the multiplication method to synthesize the entropy weight and AHP methods to obtain the S&O combined weight. However, the normalization of S&O weights after multiplication is easy to produce a multiplier effect. This results in the combined weights of the indexes with high weights are large. In contrast, the combined weights of the indexes with low weights are small. Which only applies to the situation where the number of indicators is large, and the weight distribution is uniform. Moreover, the significance of combined weights obtained by addition and multiplication methods is not easy to explain. In addition, the combined weighting method of addition and multiplication also needs to solve the allocation of S&O weight coefficients or preference coefficients. There is no ideal solution yet.
According to the research in recent years, many experts have improved the defects of traditional combination weighting. For example, Wenyu Z. et al. [31] based on AHP to stratify the indicators, processed the weights of the indicators by using the twobase point entropy weight method, worked out the objective weight, and obtained the combined weight through the Delphi method. XinXin G. et al. [32] effectively combined the entropy method with AHP to avoid single S&O weighting defects. Yunfei L. et al. [33] proposed a method to calculate the combined weight of indicators based on the principle of vector similarity. First, the importance orders of indicators were determined by the superior belonging degree method of adjacent targets. Then the objective weight of indicators was obtained by the maximum entropy criterion. At last, the final index weight was obtained using vector similarity to find the one closest to the order of subjective weight importance. Gang L. [34] proposed the combined weighting method based on standard deviation and the G1 method, a weighting method that adjusts subjectivity with objectivity. Firstly, the indexes’ importance ranking was given by experts according to their experience, and the superiority of the subjective weighting method was reflected through the ranking. Then, the objective weighting method was used to calculate the indexes’ objective weight. The indexes’ objective weight was used as the basis for calculating the importance ratio between the indexes. And the advantages of the objective weighting method in data information were reflected by calculating the importance ratio of adjacent indexes. Finally, the G1 weighting method was used to calculate the combined S&O weights.
These weighting methods solved the mechanical combination of S&O weights in the traditional weighting methods. Therefore, there is no need to determine the preference coefficient of S&O weights when combining. Thus, at present, they are relatively reasonable combination methods. However, the standard deviation only considered the conflict between the data, didn’t consider the correlation of the data. Therefore, to entirely use the data’s nature, this study used the CRITIC objective weighting method to integrate the conflict and correlation of data.
As an effective measure of quality improvement, monitoring and feedback were widely used in nosocomial infection P&C practice. For example, the monitoring and feedback on the HH of HcWs were helpful to improve the HH compliance rate [35]. In addition, monitoring and feedback of the implementation rate of P&C measures for catheterassociated urinary tract infections were conducive to reducing urinary tract infections in patients [36]. In this study, monitoring and feedback were used to monitor the quality of COVID19 P&C work in critical departments monthly. The rankings of work quality were timely fed back to departments. The longitudinal evaluation of the key departments showed that the mean value of the combined assessment showed an upward trend, suggesting that monitoring and feedback of the quality of COVID19 P&C could promote the improvement of the P&C quality.
This study used a scientific and rigorous method to establish a quality evaluation index system for COVID19 P&C. The combination weight was obtained through the subjective weight modified by the objective weight, which avoided the defects of the addition and multiplication combination method. The index’s sensitivity to the method was reduced by using various data synthesis methods to obtain scientific and stable evaluation results. However, there were still some shortcomings in this study. First of all, the quality indexes system of COVID19 P&C, in line with the actual situation of our hospital, was established through two rounds of the Delphi method. However, the indicators data were mainly obtained through investigators’ observation, with high subjectivity and low objectivity. Although the investigators were all professional personnel with more than 5 years of experience in infection control and had received unified training before the investigation, observational bias was unavoidable. In addition, no adverse events such as HcWs’ COVID19 nosocomial infection, occupational exposure to COVID19, and psychological stress occurrence during the study. Therefore, the indexes system consisted of structural and process indexes, lacking outcome indexes. Then in the combined comprehensive evaluation process, various weighting methods and resultssynthesis methods were used. The calculation process was more complex, not convenient for practical application. For this reason, a particular evaluation procedure was developed by Excel’s functions and formulas. When the relevant data were inputted, the results would be obtained automatically to facilitate the practical application of the evaluation method. Finally, this study was an observational study without a control group. The COVID19 P&C quality in critical departments was monitored and fed back every month in the prospective monitoring step. Although other quality promotion measures had not been implemented, there might still be other confounding factors. A randomized controlled trial needs to be further verified whether monitoring and feedback can improve COVID19 P&C quality in departments.
Conclusions
Summarily, in this study, a COVID19 P&C indexes system with purpose, completeness, operability, independence, and conformity to the actual situation of our hospital was established by the Delphi method. The S&O combined weight was obtained through the G1 weighting adjusted by the CRITIC method, which took into account both the data’s nature and the personal needs. COVID19 P&C quality in key departments was comprehensively evaluated by the weighted RSR method, weighted EC method, weighted TOPSIS method, and weighted GRA method. The information synthesis method was used to synthesize the comprehensive evaluation results of the four methods reasonably. The combination rationality of the four methods was evaluated by the pre and post combination tests. More scientific and comprehensive evaluation could be achieved in the combined method than the single weighting and single evaluation methods. Finally, this method was used to prospectively monitor and fed back the quality of COVID19 P&C in critical departments. As a result, COVID19 P&C quality has been continuously improved through continuous monitoring and feedback, which medical institutions could use as a reference to do excellent COVID19 P&C works.
Availability of data and materials
The data used in the study was available from the Tertiarygrade & Aclass office of the Shenzhen Hospital of the University of Chinese Academic of Science.
Abbreviations
 COVID19:

Corona Virus Disease 2019
 P&C:

Prevention and Control
 CRITIC:

Criteria Importance Though Intercriteria Correlation
 G1:

Order Relation Analysis
 S&O:

Subjective and Objective
 EC:

Efficacy Coefficient
 TOPSIS:

Technique for Order Preference by Similarity to an Ideal Solution
 RSR:

Ranksum ratio
 GRA:

Grey Relationship Analysis
 HH:

Hand hygiene
 PPE:

Personal Protective Equipment
 AHP:

Analytic Hierarchy Process
 MAUT:

MultiAttribute Utility Theory
 Cr:

Expert authority coefficient
 Cs:

Judgment coefficient
 Ca:

Familiarity coefficient
 HIS:

Hospital Information System
 LIS:

Laboratory Information System
 RIS:

Radiology Information System
 NO.:

Number
 HcWs:

Healthcare Workers
 Sd.:

Standard Deviation
 CV:

Coefficient of Variation
 CT:

Computerized Tomography
 ICU:

Intensive Care Unit
 ENT:

EarNoseThroat
 ED:

Emergency Department
 OR:

Operating Room
References
Richardson S, Hirsch JS, Narasimhan M, Crawford JM, McGinn T, Davidson KW, et al. Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID19 in the new York City area. JAMA. 2020;323(20):2052–9. https://doi.org/10.1001/jama.2020.6775.
Monshi MMA, Poon J, Chung V. Deep learning in generating radiology reports: a survey. Artif Intell Med. 2020;106:101878. https://doi.org/10.1016/j.artmed.2020.101878.
China NHCotPsRo. COVID19 technical guide for infection prevention and control in healthcare settings (7th edition). http://www.nhc.gov.cn/jkj/s3577/202009/318683cbfaee4191aee29cd774b19d8d.shtml. Accessed 20 Sept 2020.
WHO. COVID19 weekly epidemiological update. https://www.who.int/publications/m/item/weeklyepidemiologicalupdateoncovid19%2D%2D27april2021. Accessed 25 Apr 2021.
Mohammed MA, Abdulkareem KH, AlWaisy AS, et al. Benchmarking methodology for selection of optimal COVID19 diagnostic model based on entropy and TOPSIS methods. IEEE Access. 2020;8:99115–31. https://doi.org/10.1109/ACCESS.2020.2995597.
Majumder P, Biswas P, Majumder S. Application of new TOPSIS approach to identify the most significant risk factor and continuous monitoring of death of COVID19. Electron J Gen Med. 2020;17(6):em234. https://doi.org/10.29333/ejgm/7904
Singh R, Avikal S. COVID19: a decisionmaking approach for prioritization of preventive activities. Int J Healthc Manag. 2020;13(3):257–62. https://doi.org/10.1080/20479700.2020.1782661.
Albahri AS, AlObaidi JR, Zaidan AA, Albahri OS, Hashim M. Multibiological laboratory examination framework for the prioritisation of patients with COVID19 based on integrated AHP and group VIKOR methods. Int J Inf Technol Decis Mak. 2020;275(05):1247–69. https://doi.org/10.1142/S0219622020500285.
Yun Z. Application of comprehensive evaluation method in the evaluation of nosocomial infection control and prevention quality. Chin J Disinfection. 2018;35(05):52–4 [in Chinese].
Suner A, Oruc OE, Buke C, Ozkaya HD, Kitapcioglu G. Evaluation of infectious diseases and clinical microbiology specialists’ preferences for hand hygiene: analysis using the multiattribute utility theory and the analytic hierarchy process methods. BMC Med Inform Decis Mak. 2017;17(1):129. https://doi.org/10.1186/s129110170528z.
Xiaomeng C. Comprehensive evaluation of target monitoring of nosocomial infection in ICU in a hospital of Henan province. J Med Forum. 2018;v.39(04):85–7 [in Chinese].
Chen CH. A novel multicriteria decisionmaking model for building material supplier selection based on entropyAHP weighted TOPSIS. Entropy (Basel). 2020;22(2):259. https://doi.org/10.3390/e22020259
Alao MA, Ayodele TR, Ogunjuyigbe ASO, Popoola OM. Multicriteria decision based waste to energy technology selection using entropyweighted TOPSIS technique: the case study of Lagos, Nigeria. Energy. 2020;201:117675. https://doi.org/10.1016/j.energy.2020.117675.
Tchouaket Nguemeleu E, Boivin S, Robins S, Sia D, Kilpatrick K, Brousseau S, et al. Development and validation of a time and motion guide to assess the costs of prevention and control interventions for nosocomial infections: a Delphi method among experts. PLoS One. 2020;15(11):e0242212. https://doi.org/10.1371/journal.pone.0242212.
China NHCotPsRo. COVID19 diagnosis and treatment protocol (trial version 4). http://www.nhc.gov.cn/yzygj/s7653p/202001/4294563ed35b43209b31739bd0785e67.shtml. Accessed 27 Jan 2020.
China NHCotPsRo. COVID19 technical guide for infection prevention and control in healthcare settings (1st edition). http://www.nhc.gov.cn/yzygj/s7659/202001/b91fdab7c304431eb082d67847d27e14.shtml. Accessed 23 Jan 2020.
Liyun X. Construction of nursing quality index system for COVID19 infection prevention and control. Chin Gen Pract Nurs. 2020;18(8):9624. [In chinese].
Heikkilä S. The Dougados functional index with the 5point Likert scale is sensitive to change due to intensive physiotherapy and exercise in spondyloarthropathy. Clin Exp Rheumatol. 2002;20(5):689–92.eng.
Zhanglin P, Aiping Z, Sufeng W, Yu B. Designing principles and constructing processes of the comprehensive evaluation indicator system. Sci Res Manag. 2017;38(S1):20915. [in Chinese].
Diakoulaki D, Mavrotas G, Papayannakis L. Determining objective weights in multiple criteria problems: the critic method. Comput Oper Res. 1995;22(7):763–70. https://doi.org/10.1016/03050548(94)00059H.
Yajun G. The theory and methods of comprihensive evaluation. Beijing: Science Press; 2002. [in Chinese]
Peng X, Tang X, Chen Y, Zhang JH. Ranking the healthcare resource factors for public satisfaction with health system in Chinabased on the grey relational analysis models. Int J Environ Res Public Health. 2021;18(3):995. https://doi.org/10.3390/ijerph18030995
Du M, Qiu Y, Li Q, Li Y. Efficacy coefficient method assisted quadrupleactivities 3DQSAR pharmacophore model for application in environmentally friendly PAE molecular modification. Environ Sci Pollut Res Int. 2020;27(19):24103–14.eng.
Tian F. Application of rank sum ratio method. Beijing: People’s Medical Publishing House; 2002. [in Chinese]
Albahri AS, Hamid RA, Albahri OS, Zaidan AA. Detectionbased prioritisation: framework of multilaboratory characteristics for asymptomatic COVID19 carriers based on integrated entropyTOPSIS methods. Artif Intell Med. 2021;111:101983. https://doi.org/10.1016/j.artmed.2020.101983.
Böttcher HF, Posthoff C. Mathematical treatment of rank correlationa comparison of Spearman’s and Kendall’s coefficients. Z Psychol Z Angew Psychol. 1975;183(2):201–17.ger.
Santos CAG, Brasil Neto RM, da Silva RM, Passos JSA. Integrated spatiotemporal trends using TRMM 3B42 data for the upper Sao Francisco River basin, Brazil. Environ Monit Assess. 2018;190(3):175. https://doi.org/10.1007/s1066101865363.
Nguyen HT, Dawal SZ, Nukman Y, Rifai AP, Aoyama H. An integrated MCDM model for conveyor equipment evaluation and selection in an FMC based on a fuzzy AHP and fuzzy ARAS in the presence of vagueness. PLoS One. 2016;11(4):e0153222. https://doi.org/10.1371/journal.pone.0153222.
Liu X, Xu Y, Ge Y, Zhang W, Herrera F. A group decision making approach considering selfconfidence behaviors and its application in environmental pollution emergency management. Int J Environ Res Public Health. 2019;16(3):385. https://doi.org/10.3390/ijerph16030385
Saad R, Ahmad MZ, Abu MS, Jusoh MS. Hamming distance method with subjective and objective weights for personnel selection. Sci World J. 2014;2014:865495–9. https://doi.org/10.1155/2014/865495.
Wenyu Z, Lei W. Fuzzy comprehensive evaluation based on integrated enduing coefficients to assess wechat marketing effect. Comput Engin Appl. 2016;5(11):6876. [in Chinese].
Xinxin G, Qian R, Rui A. Evaluation method for regional road network service level based on combination weighting and similitude degree of grey incidence. Transp Res. 2015;1(3):16. [in Chinese].
Yunfei L, Linlin L, Zhuang Z. Research and application of decision index combination weighting method. Comput Eng. 2018;044(001):84–90 [in Chinese].
Gang L. Research on method of determing combination weights based on the standard deviation revised grounpG1. J Syst Eng. 2012;27(1):9–18 [in Chinese].
Zhong X, Wang DL, Xiao LH, Mo LF, Wu QF, Chen YW, et al. Comparison of two electronic hand hygiene monitoring systems in promoting hand hygiene of healthcare workers in the intensive care unit. BMC Infect Dis. 2021;21(1):50. https://doi.org/10.1186/s12879020057483.
Zhong X, Xiao LH, Wang DL, Yang SW, Mo LF, He LF, et al. Impact of a quality control circle on the incidence of catheterassociated urinary tract infection: an interrupted time series analysis. Am J Infect Control. 2020;48(10):1184–8. https://doi.org/10.1016/j.ajic.2020.01.006.
Acknowledgments
We want to thank the HcWs of Shenzhen Hospital of the Chinese Sciences Academy for supporting and cooperating with our research.
Funding
This study was funded by Shenzhen’s Guangming District Economic Development Special Fund (2020R01101).
Author information
Authors and Affiliations
Contributions
ZX is responsible for the study design, data analysis, interpretation of the data, statistical analysis, and the article’s drafting. WDL, MLF, ZW, XLH, WXL, and CYW were responsible for implementing the study, collecting data, and final approval. XLH and WDL are responsible for interpreting the data, critically revising the article for important intellectual content, and final approval. YL was responsible for designing the calculation formulas and functions for this study in Excel. All the authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The ethics committee of the Shenzhen hospital University of the Chinese Academy of Sciences approved the study. Furthermore, because this study was a quality improvement activity, the ethics committee waived the mandate for obtaining written informed consent from subjects. All methods were performed following the relevant guidelines and regulations.
Consent for publication
Not applicable.
Competing interests
All authors declared that they had no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Zhong, X., Wang, DL., Mo, LF. et al. Evaluation of the quality of COVID19 prevention and control by a novel comprehensive evaluation model in a tertiary general hospital: a prospective observational study. BMC Public Health 21, 2022 (2021). https://doi.org/10.1186/s12889021120329
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12889021120329