Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Explaining variance in health literacy among people with type 2 diabetes: the association between health literacy and health behaviour and empowerment

BMC Public Health volume 20, Article number: 161 (2020) Cite this article

Abstract

Background

To reflect the health literacy (HL) skills needed for managing type 2 diabetes (T2DM) in everyday life, HL in people with T2DM should be measured from a broader perspective than basic skills, such as proficiency in reading and writing. The HLS-Q12, based on the European Health Literacy Survey Questionnaire (HLS-EU-Q47), assesses four cognitive domains across three health domains. International studies on people with T2DM show inconsistent results regarding the association between HL and general health and the association between HL and glycaemic control. Moreover, knowledge is needed related to the link between HL and empowerment for those with T2DM. The aims of this study were to examine the association between i) HL and general health and diabetes outcomes, ii) HL and health behaviours and iii) HL and empowerment in people with T2DM.

Methods

During March and April 2015, 388 adults with T2DM responded to a paper-and-pencil self-administered questionnaire. A sequential multiple regression analysis was applied to explore the association between HL, as measured by the HLS-Q12, and health conditions, HbA1c, health behaviours and empowerment.

Results

For people with T2DM, higher levels of HL were associated with higher levels of education, better overall health conditions and higher self-perceived empowerment. No empirical evidence strengthening either the link between HL and glycaemic control or the link between HL and health behaviours was found.

Conclusions

The independent variables education level, overall health condition and empowerment explained about one-third of the total observed variance in HL.

Peer Review reports

Background

Health literacy (HL) is an essential individual competence necessary for dealing with health information provided, making appropriate health-related decisions and managing health in different situations [1]. Sørensen et al. [2] defined HL as follows: Health literacy is linked to literacy and entails people’s knowledge, motivation and competence to access, understand, appraise, and apply health information in order to make judgments and take decisions in everyday life concerning healthcare, disease prevention and health promotion to maintain or improve quality of life during the life course (p. 3). Having sufficient HL might be more important than ever before because people are expected to participate in health decisions and to take responsibility for their own health despite more complicated health problems and the need to navigate a more complex health system [3].

Studies have linked poor diabetes knowledge [4,5,6], poor glycaemic control [5, 7,8,9] and diabetic retinopathy [7] in people with type 2 diabetes (T2DM) to low HL. Using a modified version of the Functional, Communicative and Critical Health Literacy scale (FCCHL) [10], Finbråten et al. [11] found that lower HL is linked with poorer self-reported general health in people with T2DM. Similarly, Al Sayah, Qiu and Johnson [12] found that low HL is associated with lower health-related ‘quality of life’ measures in people with T2DM. However, neither the links between low HL and diabetes outcomes, such as poor glycaemic control [13,14,15], nor the links between HL and other self-reported health-related measures, such as general health and quality of life, are consistent [13, 16, 17]. Based on their systematic review of reviews, Caruso et al. [18] also revealed gaps in knowledge about the relations between HL and health outcomes. Thus, further studies are needed.

The prevalence of T2DM is increasing worldwide, and T2DM is strongly associated with lifestyle and health behaviours [19]. Physical activity has a beneficial effect on blood glucose levels, whereas physical inactivity, the use of tobacco and high alcohol consumption could increase the risk for complications, such as cardiovascular diseases [19, 20]. Despite T2DM having been associated with health behaviours, such as low physical activity [19], few studies have investigated the association between HL and health behaviours in people with T2DM. Some studies have linked low HL to physical inactivity, but these studies did not differentiate between type 1 and type 2 diabetes [4, 21]. The associations between HL and smoking behaviours and alcohol consumption are not strongly supported [22,23,24,25,26].

HL may facilitate the self-management of T2DM [27], active participation in diabetes treatment [28] and interactive communication with health professionals. ‘Empowerment’—individuals’ ability to make decisions and to maintain control over their personal lives [29]—refers to self-management and control over one’s own health. Empowerment has become a major goal in health communication [30] and is also viewed as a goal of HL itself [31]. Theoretical relationships have been established between HL and empowerment [1, 2, 32], where high HL is believed to increase opportunities for autonomy and empowerment in health-related decision-making [32, 33]. Wang et al. [27] emphasised the importance of HL and empowerment for self-management in people with T2DM. However, the association between HL and empowerment in people with T2DM has not yet been fully explored.

Research on the links between HL and health outcomes in general and between HL and health outcomes related to T2DM specifically have measured HL as a set of basic skills in clinical settings, such as numeracy and reading, which one might refer to as ‘functional’ HL. Examples of such measurement scales are the Test of Functional Health Literacy in Adults (TOFHLA) [34] (or the S-TOFHLA [35]), the Rapid Estimate of Adult Literacy in Medicine (REALM) [36] and Chew’s ‘brief questions to identify patients with inadequate HL’ [37]. However, people with T2DM require proficiencies that surpass basic skills if they are to utilise health information in their everyday management of the disease. The relationship between HL and health outcomes in general and between health outcomes of T2DM specifically should be further explored using instruments that capture the broader aspects of HL, such as the capacity to find, understand, evaluate and apply health information across different contexts; health care, disease prevention and health promotion. These capacities are related to the definition of HL used in this article. The European Health Literacy Survey Questionnaire (HLS-EU-Q47) was developed on the basis of this framework [2, 38, 39]. However, Finbråten et al. [40] and Huang et al. [41] have revealed several psychometric weaknesses of the HLS-EU-Q47 when validated in people with T2DM and in people with stroke, respectively. The newly developed and shortened version of this instrument, the HLS-Q12, which has enhanced psychometric properties [42, 43], was well-suited for HL screenings in the study sample of people with T2DM.

Against this background, the aims of this study were to examine the association between i) HL and general health and diabetes outcomes, ii) HL and health behaviours and iii) HL and empowerment in people with T2DM.

Methods

Study design

This cross-sectional study was conducted based on data from adults with T2DM recruited from the Norwegian Diabetes Association (NDA)—a member of the International Diabetes Federation. NDA is an independent organisation that aims to serve people with diabetes, and membership is voluntary.

Participants and data collection

According to the Norwegian Prescription Database 149,057 people were pharmacologically treated for T2DM in 2014 (87% of these had an age of 50 or older and 43% were females) [44]. At the time of sampling, 16,754 people with T2DM (average age 68 years and 53% females) were members of the NDA (Norwegian Diabetes Association, personal communication, 7 May 2015). After careful instructions from the researchers, the NDA drew a random sample of 999 people with T2DM from the 7655 members residing in nine of Norway’s 19 counties (members from the other counties participated in a pilot study that was conducted 1 year prior). All parts of the country were represented in the current study. An inclusion criteria was being diagnosed with T2DM. Further, the NDA was asked to draw a random sample from the member list, but make sure that the gender and age composition of the sample matched the distribution of gender and age of the members in the selected counties. Assuming 80% power, a small to medium effect size and having six predictors, a sample size of 142 would be sufficient for regression analysis. When using a sample size calculator and considering a confidence level of 95% and a confidence interval of 5, a sample of 376 is recommended. Thus, invitation was sent to 999 people, expecting that about one third should return a completed questionnaire.

Using a self-administrated paper-and-pencil questionnaire, the data were collected between March and April 2015. As the questionnaire was distributed via regular mail, two questionnaires were returned due to unknown addresses. Responses were received from 406 individuals. Eighteen respondents who reported type 1 diabetes were excluded as they did not meet the inclusion criteria. Furthermore, 31 individuals reported health conditions that were incompatible with responding to the questionnaire and were consequently excluded. Hence, the analyses were based on responses from 388 individuals (response rate 41% [388/948]).

In our sample, just over half of the 388 respondents were men (Table 1). The average age was 73 years. Just below one-third had compulsory comprehensive school as their highest completed education, and about the same fraction had completed education at a university or university college level. The majority of the respondents reported fairly good to very good general health.

Table 1 Sample characteristics (n = 388)
Full size table

Measures

HL was measured using the HLS-Q12 instrument developed by Finbråten et al. [42]. The respondents were asked to report demographic variables (gender, age and education level), health behaviours (physical activity, alcohol consumption and tobacco use), general health condition (‘How is your health in general?’), items intended to be indicators of empowerment and their latest measured glycated haemoglobin (HbA1c) level. In Norway, HbA1c is measured at least twice a year in people with T2DM [45]. They were also asked whether they had communicated with a registered nurse about health-related issues during the past 12 months.

Health literacy

Using the 12-item HLS-Q12 [42], which has been validated in the Norwegian population [42, 43] and for Norwegians with T2DM (see below) [43], HL was measured across the four cognitive domains (access, understand, appraise and apply health information) and three health domains (health care, disease prevention and health promotion) of the model developed by Sørensen et al. [2]. The 12-item HLS-Q12 scale, for which each cell of this 4 × 3 matrix of HL is represented by one item, has a 4-point rating scale; very difficult (1), difficult (2), easy (3) and very easy (4), where the higher score sums indicate higher HL proficiencies. The additional response category ‘don’t know’ was recoded as ‘systematic missing’ during the analyses. Using the scoring model 0–3, the HLS-Q12 raw score sums represent a so-called sufficient statistic for the unidimensional partial credit Rasch model [46, 47]. Using a confirmatory factor analysis (CFA) and treating the data as categorical (Robust Maximum Likelihood [RML] and Diagonally Weighted Least Squares [DWLS] estimators), it was found that the HLS-Q12 data also displayed acceptable ‘goodness-of-fit’ indexes.

When applied to people with T2DM, the HLS-Q12 is sufficiently reliable with high reliability indexes (person separation index estimated to 0.847 using the RUMM2030 statistical package [48] and Coefficient H estimated to 0.923). All 12 items conformed well to the partial credit parameterisation of the unidimensional Rasch model, and all items had ordered response categories / no reversed Rasch-Andrich thresholds. At the overall level, the HLS-Q12 is sufficiently unidimensional (lower binominal 95% CI proportion of 0.04) with no significantly locally dependent items (cf. ‘response dependency’) and no items displaying ‘differential item functioning’ for available person factors [43].

After applying CFA using LISREL9.3 software [49], the HLS-Q12 displayed acceptable goodness-of-fit (GOF) indexes (normed Satorra-Bentler chi square of 1.67, standardised root mean square residual [SMSR] of 0.066, comparative fit index [CFI] of 0.976 and a root mean square error of approximation [RMSEA] of 0.132). The item communalities varied between 0.36 and 0.65 with an average value of 0.48 [43].

Health behaviour

Health behaviour was reported using single items related to physical activity, tobacco use and alcohol consumption. These items were developed based on similar items of the HLS-EU-Q47 [38]. The participants reported their physical activity by responding to the single item ‘How often have you been physically active for at least 30 minutes during the last 30 days?’, which had four response categories: none of the days (1), some of the days (2), some days every week (3) and every or almost every day (4). Respondents were classified as physically active if they reported being physically active ‘every or almost every day’. The respondents reported their tobacco use by answering the single item ‘How would you best describe your use of tobacco (such as cigarettes, cigars and tobacco pipes)?’, which had four response categories: smoke every day (1), smoke now and then (2), former smoker (3) and never smoked (4). Those who answered that they were former smokers or had never smoked were all classified as non-smokers. Alcohol consumption was reported by responding to the single item ‘How often did you drink alcohol during the last 30 days?’ with the response categories: none of the days (1), once (2), two-three times (3), once a week (4), two-three times a week (5), four-six times a week (6) and every day (7). Those who answered ‘none of the days’ were classified as non-consumers. For all items, the response category ‘do not want to answer’ was offered and later recoded as ‘systematic missing’.

Empowerment-related indicators

Empowerment-related indicators were measured using four self-developed items. The items were developed based on the Diabetes Empowerment Scale [50], considering the World Health Organisation’s definition of individual empowerment [29] and the aspects of patient empowerment as described by Schulz and Nakamoto [1, 51]. The four items asked how difficult or easy it is to: 1) take control of one’s own health in daily life, 2) implement goals related to one’s own health, 3) take responsibility for one’s own physical and mental health by being physical active, eating healthily and being social and 4) participate actively in health communication with health professionals by asking questions and prompting a plan of action for one’s own health needs. The items were used as a proxy for empowerment and had a four-point response scale: very difficult (1), difficult (2), easy (3) and very easy (4), where higher scores indicated higher empowerment.

Data analysis

When investigating HL across different levels of person factors, the raw data obtained from the HLS-Q12 were transformed into logit values using RUMM2030 statistical package [48]. Hence, the analyses were performed with person-location estimates of HL, providing continuous data and interval levels. Furthermore, the demographic variables and variables related to general health, health behaviours and empowerment-related indicators were dichotomised before being entered into the analyses. Differences in HL across person factors were analysed using independent t-tests.

A sequential multiple linear regression analysis was performed to further investigate the association between HL and the variables that showed significant differences in HL based on the independent t-tests. The general linear regression model requires a continuous outcome variable and assumes normally distributed residuals with constant variance (homoscedasticity) that are independent (residual covariances are zero) and have a mean of zero. To meet the requirement of a continuous outcome variable, the HLS-Q12 person-location estimates from the Rasch modelling were used. There are no requirements for or assumptions related to the independent variables [52].

In the regression analysis, the independent variables ‘educational level’ and ‘general health condition’ were dichotomised in the same way as for the t-tests, whereas the empowerment-related indicators were treated as categorical with a four-point rating scale. The independent variables were entered in three sequential steps. According to Field [52], variables that are already known to be predictors of the dependent variable should be entered first. As several studies have reported a link between HL and education level [4, 12, 53,54,55,56,57], this variable was introduced first in Model 1. Some studies have reported a link between HL and general health, whereas the link between HL and empowerment has been investigated in people with T2DM only to a small extent. Hence, in Model 1, ‘education level’ was entered as the independent variable. In Model 2, the independent variable ‘general health condition’ was entered in the next step along with ‘educational level’. In the final model (Model 3), the independent variables for empowerment-related indicators were added. Hence, it was possible to assess the contribution of each independent variable. Adjusted R squares were used to assess the model fit, whereas a change in an R square was used to assess the contribution of newly entered independent variables [52]. The standardised β coefficient with p-value was used to assess the unique contribution of each variable. Initial analyses indicated no violation of the assumptions of multicollinearity, normality, linearity or homoscedasticity. The significance level was set at 5%. It was also investigated whether gender and age could be possible confounders.

Using a sequential multiple regression in SPSS 24 [58], missing data were treated listwise, resulting in an effective sample size of 252–257.

Results

In Table 2, the proportion of responses (%) in each response category of the 4-point rating scale is reported for each HLS-Q12 item. On average, 38% responded in either category 1 or 2 and perceived the task as (very) difficult.

Table 2 The proportion of responses (%) in each category of the 4-point rating scale for each HLS-Q12 item
Full size table

HL and demographic variables, general health, diabetes outcome, health behaviours and empowerment-related indicators

Based on the investigation of HL across the levels of person factors and general health using independent t-tests, those with an education level at the university/university college level and those who reported their general health as good had significantly higher HL (person-location estimates of HL) with p-values of 0.001 and 0.002, respectively. No significant differences were observed for the person factors gender, age or HbA1c (Table 3). In addition, there were no significant differences in HL observed across age groups when treating the variable age as categorical (not shown in the table). When investigating HL between those who reported (n = 312) their latest HbA1c against those who did not (n = 76), it was found that the former group had significantly higher HL (p = 0.044; not shown in the table). No significant differences in mean values were observed for the health behaviour variables, but those who reported the empowerment-related indicators as easy had significantly higher HL (p < 0.001 for all four items).

Table 3 Health literacy across levels of person factors, general health, diabetes outcome, health behaviours and empowerment
Full size table

Those who reported that they had been in contact with a nurse with regard to their health situation during the last year (n = 214) had significantly higher HL (p = 0.021) than those who had not (n = 150; not reported in Table 3).

The association between HL and education, general health and empowerment-related indicators

After entering the variables that showed significant differences in HL proficiency (education, self-reported general health and empowerment-related indicators) as independent variables in a sequential multiple linear regression analysis, the variables of the final model (Model 3) explained 33% of the total variance in HL as the dependent variable. The empowerment-related indicators explained more of the observed variance in HL than the other independent variables (change in R square = 28%; Table 4). The items ‘take control of one’s own health in daily life’ and ‘participate actively in health communication with health professionals’ made the strongest and most significant contributions to the variance in HL in the final model (β = 0.23, p = 0.001 and β = 0.28, p < 0.001, respectively), followed by the variable ‘education’ (β = 0.13, p = 0.015; Table 4).

Table 4 Regression analysis for education, self-reported general health and empowerment-related indicators on health literacy
Full size table

Despite the significant differences in HL between those who reported that they had been in contact with a nurse with regard to their health situation and those who had not, this variable did not make a significant contribution in the regression analysis (not shown in the table). The variables gender and age did not correlate with the dependent variable, nor display any significant contribution to the models (not shown in the table).

Discussion

On average, 38% reported that the HL tasks were difficult or very difficult. When measuring HL by means of the HLS-Q12, higher HL was associated with education at the university/university college level, good general health and higher empowerment.

HL and demographic variables, general health, diabetes outcome, health behaviours and empowerment-related indicators

According to the proportion of responses in each response category, over one-third marked the HL tasks as difficult or very difficult, which implies that health professionals should pay attention to the individual’s HL and should adapt their health communication to the appropriate level. In contrast to van der Heide [4] and Hussein, Almajran and Albatineh [53] but in accord with Al Sayah et al. [16] and Vandenbosch et al. [54], no significant differences in HL with regard to age were found. However, the average age of the sample was relatively high, so differences related to age may not have been evident.

Significant differences in HL were found in relation to education. People with T2DM who had completed a university-level education reported a significantly higher HL (as measured by the HLS-Q12) than those with a lower education level, which is similar to the findings of previous studies [4, 12, 53,54,55,56,57]. This finding was also supported when using the FCCHL to measure HL in the same population [11]. According to this finding, nurses and other health professionals should tailor health information to the individual’s HL level and their educational level, and they should be prepared to devote more time to explaining relevant health information, using different learning aids and ensuring that the individual thoroughly understands the information. Health professionals should also ensure that individuals have the capacity to use the information to promote their health in everyday life. Regarding the significant association between HL and education, actions aimed at strengthening HL should be implemented as early as primary and secondary school. Such interventions should aim to strengthen reading and writing skills, strengthen knowledge related to health issues and the determinants of health, strengthen competence in the critical assessment of different sources of health information and strengthen knowledge regarding where to find evidence-based health information.

Those with a higher HL reported significantly better general health than those with lower HL estimates. However, only 4% of the variance in HL was explained by general health condition. Whether low HL could result in worse health or whether the opposite occurs is unclear. If low HL causes worse health, adapting health information to the individual’s HL is important to avoid a negative impact on health. Strengthening HL could improve an individual’s health, and thus a particular effort should be made to develop HL in people with T2DM. This is also supported by Al Sayah et al. [12], who called for studies investigating whether such interventions could lead to improvements in health.

It was also found that those who had been in contact with a registered nurse regarding their health situations had significantly higher HL. In Norway, people with T2DM are usually offered a follow-up by their regular general practitioners [45]. Those who are offered additional follow-ups with registered nurses may receive more comprehensive information and information better adapted to their needs. Registered nurses may conduct longer consultations than general practitioners due to time constraints, and thus they have time to adapt and to explain the information more thoroughly. This is potentially supported by Tshiananga [59], who found that nurse-led follow-ups and diabetes education were associated with improved glycaemic control. Thus, it could be recommended that people with T2DM should be offered follow-ups with registered nurses who are educated in diabetes treatment and health communication in addition to the follow-ups provided by general practitioner. However, many nurses have limited knowledge related to HL [60, 61], which could affect the learning outcomes of diabetes education and self-management in people with T2DM. Hence, nursing education should include a greater emphasis on HL as well as how health communication could be better adapted to people’s HL levels.

No significant differences were found in HL related to HbA1c, which is in accord with Lee et al. [57], Al Sayah et al. [16] and Finbråten et al. [11] (the latter using the FCCHL in the same population) but in contrast to van der Heide et al. [4], Powell et al. [5], Schillinger et al. [7], Tang et al. [8] and Cavanaugh et al. [9]. However, significant differences were found in HL between those reporting and those not reporting their latest HbA1c levels. The reason for people not reporting this value is uncertain. One reason could be that they do not know their latest HbA1c levels. Hence, it is possible that there is an association between HL and knowing the HbA1c level. The findings confirm the claim put forward by Al Sayah et al. [13] and Bailey et al. [14] that the association between HL and HbA1c is inconsistent. This inconsistency may be due to HL being measured using different instruments. All the studies mentioned [4, 5, 7,8,9, 16] with the exception of Lee et al. [57] and Finbråten et al. [11] measured HL using instruments that were limited to the functional HL level. Another explanation might be different operationalisations of the variable HbA1c. When treating HbA1c as a dichotomous variable, van der Heide et al. [4] did not find a significant difference in HL.

Despite the association between HL and self-reported general health, there was no association between HL and health behaviours. The absence of an association between HL and both smoking behaviours and alcohol consumption is in line with the findings of Friis et al. [21]. However, the association between HL and physical activity, as described by Friis et al. [21] and van der Heide et al. [4], was not supported by this study. Measuring HL using the FCCHL, Shin and Lee [62] did not find any significant direct effect of HL on physical activity in elderly people with diabetes. The average age in the sample was quite high. Hence, T2DM in the sample could be linked more to age than to health behaviours. The association between HL and health behaviours should be further explored in younger individuals with T2DM. Since the study was initiated, the recommendation for physical activity has changed from 30 min of physical activity per day to 150 min of physical activity each week [63]. However, dividing 150 min by 5 days equals 30 min 5 days per week. As this change can be considered quite small, it is not expected to influence the results of this study.

Theoretically, HL has been linked to empowerment [2, 32]. A significant association between HL and all four empowerment-related indicators was found. A relationship between personal empowerment and HL was also identified by Wang et al. [27]. The empowerment-related indicators also contributed the most to the variance in HL when performing the regression analysis. If Nutbeam’s theory [32] is assumed to be correct and thus higher HL implies higher empowerment, strengthening HL might lead to greater empowerment. Hence, to enhance empowerment in people with T2DM, nurses and other health professionals should strengthen their HL, such as by providing valid and reliable health information and guiding the individuals to the actual sources of the information. Furthermore, nurses and other health professionals should guide individuals in evaluating the advantages and disadvantages of the different treatment options and should facilitate their active participation in health communication. In addition, they should enable individuals to take control of their own health in their daily lives by providing adapted health information. The association between HL and taking control of one’s own health could also confirm the theoretical association between HL and empowerment, as ‘control’ is a central part of the definition of individual empowerment. However, higher HL does not necessarily entail empowerment [1]. Schulz and Nakamoto [1] pointed out that mismatches of these two factors can have dangerous consequences, as emphasising a high degree of empowerment without adequate knowledge and a high level of HL may increase the risk of making inexpedient health choices.

By defining empowerment as a process, Lee et al. [57] claimed that there is a direct pathway from empowerment to HL. Hence, if the individual’s active participation and involvement in health communication is facilitated, it may also strengthen HL. It is important that nurses and other health professionals facilitate this involvement by using language that is adapted to the patient’s HL level and framework of understanding. This is in accord with Engström, Leksell, Johansson and Gudbjörnsdottir [64], who found that some health information was difficult to understand due to the use of jargon and that people wanted the information to be tailored to their needs. Engström et al. [64] also found that most people with diabetes (both Types 1 and 2) wanted to participate in shared decision making. Hence, nurses and other health professionals should ask the patients about their preferences and should invite them to participate in a dialogue regarding their health and the requirements associated with living with T2DM.

The final regression model explained 33% of the variance, which means that other factors also contributed to the variance in HL. In light of the definition provided by Sørensen et al. [2], it is reasonable to assume that factors such as motivation, diabetes knowledge and general literacy may contribute to the variance in HL. Additional factors, such as self-efficacy, may also contribute to the variance in HL. There may also be person factors other than those mentioned that contribute to the variance.

Methodological considerations

The sample was drawn from the member list of the NDA. Hence, the sample might not be representative of the entire population of people with T2DM, as members of such an organisation may be better informed and motivated in managing their chronic disease than non-members. However, it was difficult to recruit non-members as we were not allowed to use Norwegian Patient Registry as a sampling frame. In addition, the mean age of the sample was quite high and was higher than the average age in the member list. Consequently, the results may be considered a reflection of older adults with T2DM.

We do not have access to information about age and educational level in Norwegians with T2DM not being members of the NDA. However, in a study describing the incidence of T2DM in Norway [44] from 2009 to 2014, people aged 55 and older constituted 75% of all individuals newly diagnosed with T2DM. Ruiz et al. [44] reported that about the half of the individuals newly diagnosed with T2DM had completed education at upper secondary school-level, which is similar to our study. Further, Ruiz et al. [44] reported that just below one fifth had completed higher education, which is however in contrast with our sample, where 30% had such education.

Despite the high average age in our study sample, the majority of the respondents considered their health as good or even very good. However, in Norway there is not unusual that elderly people rate their health as good. In fact, according to Statistics Norway [65], 60% of Norwegians 80 years old rate their health as good or very good. The associations between HL and general health, health behaviours and empowerment should be further explored in younger people with T2DM. According to Statistics Norway [66], 11% of the entire population includes daily smokers. Hence, there is a risk of a response bias in the study because only 4% reported being a smoker.

Responding to self-administered measures could be quite challenging for elderly people and for those with limited HL, as it requires reading and reading comprehension abilities. However, during cognitive interviews, the participants reported that the items were clearly stated and easy to understand. Advantages of using paper- and-pencil questionnaire are that there are lower risk for interviewer bias and that the respondents answering according to what they think is expected [67].

The sample size was rather small, which might lower the generalisability of the findings. However, assuming 80% power, the number of independent variables and a small to medium effect size, the sample size could be considered sufficient [52, 68].

To ensure the plausibility of the interpretations of the scores obtained from the HLS-Q12 for people with T2DM, the psychometric properties of the scale were assessed. Applying the Rasch model analysis and CFA, the HLS-Q12 was found to have acceptable psychometric properties for people with T2DM. These findings are similar to those of Finbråten et al. [42], who validated the HLS-Q12 in the Norwegian population. However, in contrast to Finbråten et al. [42], a discrepancy between the fit indexes CFI and RMSEA was found, which suggested a good and a relatively bad fit, respectively. This discrepancy was further explored by comparing estimates obtained from ML and DWLS, which displayed quite similar results. Moreover, the LISREL output did not suggest any modification indices. However, Brown [69] claimed that a high value of RMSEA may be of less concern if other GOF indexes suggest acceptable model fit. Altogether, the HLS-Q12 showed several strengths, as the data displayed a good fit to the unidimensional Rasch model, which implies that all requirements of the fundamental measurement were met [70]. Hence, the HLS-Q12 could be used as a valid and reliable scale for measuring HL in people with T2DM in relation to different health settings. However, the GOF indexes should be further explored in future studies. The HLS-Q12 has thus far been validated for people with T2DM and in the Norwegian population, but the psychometric properties of the instrument should be further assessed in other populations and across different languages and cultures.

Conclusions

When applying the HLS-Q12 to measure HL in people with T2DM, it was found that a higher HL was associated with education at the university/university college level, good self-reported general health and higher empowerment-related indicators. Nurses and other health professionals should be aware that people have different levels of HL and should pay particular attention to the prerequisites for dealing with health information among those with less education and worse self-reported health. Furthermore, the health information should be adapted to the individual’s HL to support the individual’s active participation in diabetes management and to enable individuals to take control of their own health in their daily lives. No evidence of an association between HL and glycaemic control nor between HL and health behaviours was found.

The association between HL and health behaviours should be further investigated, especially in younger people with T2DM. Other factors that could contribute to the variance in HL should also be investigated.

Availability of data and materials

The dataset used and analysed during the current study is available from the corresponding author upon reasonable request.

Abbreviations

CFA:

Confirmatory factor analysis

CFI:

Comparative fit index

CI:

Confidence interval

DWLS:

Diagonally Weighted Least Squares

FCCHL:

Functional, Communicative, and Critical Health Literacy scale

GOF:

Goodness-of-fit

HbA1c:

Glycated haemoglobin

HL:

Health literacy

HLS-EU-Q47:

European Health Literacy Survey Questionnaire

ML:

Maximum Likelihood

NDA:

Norwegian Diabetes Association

RMSEA:

Root-mean-squared error of approximation

SRMR:

Standardised root mean square residual

T2DM:

Type 2 diabetes

References

  1. Schulz PJ, Nakamoto K. Health literacy and patient empowerment in health communication: the importance of separating conjoined twins. Patient Educ Couns. 2013;90(1):4–11.

    Article  PubMed  Google Scholar 

  2. Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, Brand H. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12(1):1–13.

    Article  Google Scholar 

  3. Johnson A. Health literacy, does it make a difference? Aust J Adv Nurs. 2014;31(3):39.

    Google Scholar 

  4. van der Heide I, Uiters E, Rademakers J, Struijs JN, Schuit AJ, Baan CA. Associations among health literacy, diabetes knowledge, and self-management behavior in adults with diabetes: results of a Dutch cross-sectional study. J Health Commun. 2014;19(Suppl 2):115–31.

    Article  PubMed  Google Scholar 

  5. Powell CK, Hill EG, Clancy DE. The relationship between health literacy and diabetes knowledge and readiness to take health actions. Diabetes Educ. 2007;33(1):144–51.

    Article  PubMed  Google Scholar 

  6. Hahn EA, Burns JL, Jacobs EA, Ganschow PS, Garcia SF, Rutsohn JP, Baker DW. Health literacy and patient-reported outcomes: a cross-sectional study of underserved English-and Spanish-speaking patients with type 2 diabetes. J Health Commun. 2015;20(Suppl 2):4–15.

    Article  PubMed  Google Scholar 

  7. Schillinger D, Grumbach K, Piette J, Wang F, Osmond D, Daher C, Palacios J, Sullivan GD, Bindman AB. Association of health literacy with diabetes outcomes. JAMA. 2002;288(4):475–82.

    Article  PubMed  Google Scholar 

  8. Tang YH, Pang SM, Chan MF, Yeung GS, Yeung VT. Health literacy, complication awareness, and diabetic control in patients with type 2 diabetes mellitus. J Adv Nurs. 2008;62(1):74–83.

    Article  PubMed  Google Scholar 

  9. Cavanaugh K, Huizinga MM, Wallston KA, Gebretsadik T, Shintani A, Davis D, Gregory RP, Fuchs L, Malone R, Cherrington A, et al. Association of numeracy and diabetes control. Ann Intern Med. 2008;148(10):737–46.

    Article  PubMed  Google Scholar 

  10. Ishikawa H, Takeuchi T, Yano E. Measuring functional, communicative, and critical health literacy among diabetic patients. Diabetes Care. 2008;31(5):874–9.

    Article  PubMed  Google Scholar 

  11. Finbråten HS, Guttersrud Ø, Nordström G, Pettersen KS, Trollvik A, Wilde-Larsson B. Validating the functional, communicative and critical health literacy scale using Rasch modeling and confirmatory factor analysis. J Nurs Meas. 2018;26(2):341–63.

    Article  PubMed  Google Scholar 

  12. Al Sayah F, Qiu W, Johnson JA. Health literacy and health-related quality of life in adults with type 2 diabetes: a longitudinal study. Qual Life Res. 2016;25(6):1487–94.

    Article  PubMed  Google Scholar 

  13. Al Sayah F, Majumdar SR, Williams B, Robertson S, Johnson JA. Health literacy and health outcomes in diabetes: a systematic review. J Gen Intern Med. 2013;28(3):444–52.

    Article  PubMed  Google Scholar 

  14. Bailey SC, Brega AG, Crutchfield TM, Elasy T, Herr H, Kaphingst K, Karter AJ, Moreland-Russell S, Osborn CY, Pignone M. Update on health literacy and diabetes. Diabetes Educ. 2014;40(5):581–604.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97.

    Article  PubMed  Google Scholar 

  16. Al Sayah F, Majumdar SR, Johnson JA. Association of inadequate health literacy with health outcomes in patients with type 2 diabetes and depression: secondary analysis of a controlled trial. Can J Diabetes. 2015;39(4):259–65.

    Article  PubMed  Google Scholar 

  17. Boren SA. A review of health literacy and diabetes: opportunities for technology. J Diabetes Sci Technol. 2009;3(1):202–9.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Caruso R, Magon A, Baroni I, Dellafiore F, Arrigoni C, Pittella F, Ausili D. Health literacy in type 2 diabetes patients: a systematic review of systematic reviews. Acta Diabetol. 2018;55(1):1–12.

    Article  PubMed  Google Scholar 

  19. International Diabetes Federation. IDF Diabetes Atlas. 2017. https://www.idf.org/our-activities/advocacy-awareness/resources-and-tools/134:idf-diabetes-atlas-8th-edition.html. Accessed 12 Feb 2019.

  20. World Health Organization (WHO). Diabetes. http://www.who.int/mediacentre/factsheets/fs312/en/. Accessed 20 Feb 2019.

  21. Friis K, Vind BD, Simmons RK, Maindal HT. The relationship between health literacy and health behaviour in people with diabetes: a Danish population-based study. J Diabetes Res. 2016;2016:1–7.

  22. Fernandez DM, Larson JL, Zikmund-Fisher BJ. Associations between health literacy and preventive health behaviors among older adults: findings from the health and retirement study.(Report). BMC Public Health. 2016;16(1):596.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Levin-Zamir D, Baron-Epel OB, Cohen V, Elhayany A. The association of health literacy with health behavior, socioeconomic indicators, and self-assessed health from a national adult survey in Israel. J Health Commun. 2016;21(Suppl 2):61–8.

    Article  PubMed  Google Scholar 

  24. Geboers B, Reijneveld SA, Jansen CJM, de Winter AF. Health literacy is associated with health behaviors and social factors among older adults: results from the LifeLines cohort study. J Health Commun. 2016:1–9.

  25. Al Sayah F, Johnson ST, Vallance J. Health literacy, pedometer, and self-reported walking among older adults. Am J Public Health. 2016;106(2):327–33.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Jayasinghe UW, Harris MF, Parker SM, Litt J, van Driel M, Mazza D, Del Mar C, Lloyd J, Smith J, Zwar N. The impact of health literacy and life style risk factors on health-related quality of life of Australian patients. Health Qual Life Outcomes. 2016;14(1):68.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Wang R-H, Hsu H-C, Lee Y-J, Shin S-J, Lin K-D, An L-W. Patient empowerment interacts with health literacy to associate with subsequent self-management behaviors in patients with type 2 diabetes: a prospective study in Taiwan. Patient Educ Couns. 2016;99(10):1626–31.

    Article  PubMed  Google Scholar 

  28. Bagnasco A, Di Giacomo P, Da Rin Della Mora R, Catania G, Turci C, Rocco G, Sasso L. Factors influencing self-management in patients with type 2 diabetes: a quantitative systematic review protocol. J Adv Nurs. 2014;70(1):187–200.

    Article  PubMed  Google Scholar 

  29. World Health Organization (WHO): Health promotion glossary. 1998. http://www.who.int/healthpromotion/about/HPR%20Glossary%201998.pdf?ua=1. Accessed 28 June 2018.

  30. Schiavo R. Health communication: from theory to practice. 2nd ed. San Francisco: Jossey-Bass; 2014.

    Google Scholar 

  31. Paasche-Orlow MK, Schillinger D, Weiss BD, Bickmore T, Cabral H, Chang P, Bailey SC, Dewalt DA, Fernandez A, Fransen M. Health literacy and power. Health Lit Res Pract. 2018;2(3):e132–3.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promot Int. 2000;15(3):259–67.

    Article  Google Scholar 

  33. Nutbeam D. The evolving concept of health literacy. Soc Sci Med. 2008;67(12):2072–8.

    Article  PubMed  Google Scholar 

  34. Parker RM, Baker DW, Williams MV, Nurss JR. The test of functional health literacy in adults: a new instrument for measuring patients' literacy skills. J Gen Intern Med. 1995;10(10):537–41.

    Article  CAS  PubMed  Google Scholar 

  35. Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss J. Development of a brief test to measure functional health literacy. Patient Educ and Couns. 1999;38(1):33–42.

    Article  CAS  Google Scholar 

  36. Davis TC, Long SW, Jackson RH, Mayeaux E, George RB, Murphy PW, Crouch MA. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993;25(6):391–5.

    CAS  PubMed  Google Scholar 

  37. Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med. 2004;36(8):588–94.

    PubMed  Google Scholar 

  38. HLS-EU Consortium. Comparative report of health literacy in eight EU member states. European health Literacy Survey HLS-EU 2012.

    Google Scholar 

  39. Sørensen K, Van den Broucke S, Pelikan JM, Fullam J, Doyle G, Slonska Z, Kondilis B, Stoffels V, Osborne RH, Brand H. Measuring health literacy in populations: illuminating the design and development process of the European health literacy survey questionnaire (HLS-EU-Q). BMC Public Health. 2013;13(1):948.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Finbråten HS, Pettersen KS, Wilde-Larsson B, Nordström G, Trollvik A, Guttersrud Ø. Validating the European health literacy survey questionnaire in people with type 2 diabetes. Latent trait analyses applying multidimensional Rasch modelling and confirmatory factor analysis. J Adv Nurs. 2017;73(11):2730–44.

    Article  PubMed  Google Scholar 

  41. Huang Y-J, Chen C-T, Lin G-H, Wu T-Y, Chen S-S, Lin L-F, Hou W-H, Hsieh C-L. Evaluating the European health literacy survey questionnaire in patients with stroke: a latent trait analysis using Rasch modeling. Patient. 2018;11(1):83–93.

    Article  PubMed  Google Scholar 

  42. Finbråten HS, Wilde-Larsson B, Nordström G, Pettersen KS, Trollvik A, Guttersrud Ø. Establishing the HLS-Q12 short version of the European health literacy survey questionnaire. BMC Health Serv Res. 2018;18(506):1–17.

    Google Scholar 

  43. Finbråten HS. Measuring health literacy - evaluating psychometric properties of the HLS-EU-Q47 and the FCCHL, suggesting instrument refinements and exploring health literacy in people with type 2 diabetes and in the general Norwegian population. Karlstad: Karlstad University Studies; 2018.

    Google Scholar 

  44. Ruiz PLD, Stene LC, Bakken IJ, Håberg SE, Birkeland KI, Gulseth HL. Decreasing incidence of pharmacologically and non-pharmacologically treated type 2 diabetes in Norway: a nationwide study. Diabetologia. 2018;61(11):2310–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. The Norwegian Directorate of Health. Nasjonal faglig retningslinje for diabetes [National professional guideline for diabetes]. 2017. https://helsedirektoratet.no/retningslinjer/diabetes. Accessed 28 June 2018.

  46. Rasch G. Probabilistic models for some intelligence and attainment tests. Expanded ed. Chicago: University of Chicago Press; 1960.

    Google Scholar 

  47. Masters GN. A Rasch model for partial credit scoring. Psychometrika. 1982;47(2):149–74.

    Article  Google Scholar 

  48. Andrich D, Sheridan B, Luo G. Rasch Unidimensional Measurement Model. RUMM2030 (Software). RUMM Laboratory Pty Ltd.; 2003.

  49. Jöreskog K, Sörbom D. LISREL 9.3 software. Skokie: Scientific Software International, Inc; 2017.

    Google Scholar 

  50. Anderson RM, Funnell MM, Fitzgerald JT, Marrero DG. The diabetes empowerment scale: a measure of psychosocial self-efficacy. Diabetes Care. 2000;23(6):739–43.

    Article  CAS  PubMed  Google Scholar 

  51. Schulz PJ, Nakamoto K. " Bad" Literacy, the Internet, and the Limits of Patient Empowerment. In: AAAI Spring Symposium: AI and Health Communication; 2011.

    Google Scholar 

  52. Field A. Discovering statistics using IBM SPSS statistics. 4th ed. Los Angeles: SAGE; 2013.

    Google Scholar 

  53. Hussein SH, Almajran A, Albatineh AN. Prevalence of health literacy and its correlates among patients with type II diabetes in Kuwait: a population based study. Diabetes Res Clin Pract. 2018;141:118–25.

    Article  PubMed  Google Scholar 

  54. Vandenbosch J, Van den Broucke S, Schinckus L, Schwarz P, Doyle G, Pelikan J, Muller I, Levin-Zamir D, Schillinger D, Chang P. The impact of health literacy on diabetes self-management education. Health Educ J. 2018;77(3):349–62.

    Article  Google Scholar 

  55. Lai AY, Ishikawa H, Kiuchi T, Mooppil N, Griva K. Communicative and critical health literacy, and self-management behaviors in end-stage renal disease patients with diabetes on hemodialysis. Patient Educ Couns. 2013;91(2):221–7.

    Article  PubMed  Google Scholar 

  56. Heijmans M, Waverijn G, Rademakers J, van der Vaart R, Rijken M. Functional, communicative and critical health literacy of chronic disease patients and their importance for self-management. Patient Educ Couns. 2015;98(1):41–8.

    Article  PubMed  Google Scholar 

  57. Lee Y-J, Shin S-J, Wang R-H, Lin K-D, Lee Y-L, Wang Y-H. Pathways of empowerment perceptions, health literacy, self-efficacy, and self-care behaviors to glycemic control in patients with type 2 diabetes mellitus. Patient Educ Couns. 2016;99(2):287–94.

    Article  PubMed  Google Scholar 

  58. IBM Corporation. SPSS version 24. 2016.

    Google Scholar 

  59. Tshiananga JKT, Kocher S, Weber C, Erny-Albrecht K, Berndt K, Neeser K. The effect of nurse-led diabetes self-management education on glycosylated hemoglobin and cardiovascular risk factors: a meta-analysis. Diabetes Educ. 2012;38(1):108–23.

    Article  PubMed  Google Scholar 

  60. Macabasco-O'Connell A, Fry-Bowers EK. Knowledge and perceptions of health literacy among nursing professionals. J Health Commun. 2011;16(Suppl 3):295–307.

    Article  PubMed  Google Scholar 

  61. Rajah R, Ahmad Hassali MA, Jou LC, Murugiah MK. The perspective of healthcare providers and patients on health literacy: a systematic review of the quantitative and qualitative studies. Perspect Public Health. 2017;138(2):122–32.

    Article  PubMed  Google Scholar 

  62. Shin KS, Lee EH. Relationships of health literacy to self-care behaviors in people with diabetes aged 60 and above: empowerment as a mediator. J Adv Nurs. 2018;74:2363–72.

    Article  PubMed  Google Scholar 

  63. World Health Organization (WHO). Global Strategy on Diet, Physical Activity and Health. Recommended levels of physical activity for adults aged 18–64 years. http://www.who.int/dietphysicalactivity/factsheet_adults/en/. Accessed 12 Feb 2019.

  64. Engström MS, Leksell J, Johansson U-B, Gudbjörnsdottir S. What is important for you? A qualitative interview study of living with diabetes and experiences of diabetes care to establish a basis for a tailored patient-reported outcome measure for the Swedish National Diabetes Register. BMJ Open. 2016;6(3):e010249.

    Article  Google Scholar 

  65. Statistics Norway. State of health. 2018. https://www.ssb.no/helse/artikler-og-publikasjoner/over-halvparten-av-oss-blir-over-80-ar-og-stadig-sprekere. Accessed 18 Sept 2019.

    Google Scholar 

  66. Statistics Norway. Smoking habits. 2018. https://www.ssb.no/en/helse/statistikker/royk. Accessed 28 June 2018.

    Google Scholar 

  67. Polit DF, Beck CT. Nursing research: generating and assessing evidence for nursing practice. Philadelphia: Wolters Kluwer Health Lippincott Williams & Wilkins; 2012.

    Google Scholar 

  68. Knofczynski GT, Mundfrom D. Sample sizes when using multiple linear regression for prediction. Educ Psychol Meas. 2008;68(3):431–42.

    Article  Google Scholar 

  69. Brown TA. Confirmatory factor analysis for applied research. New York: Guilford Publications; 2015.

    Google Scholar 

  70. Kreiner S. The Rasch model for dichotomous items. In: Christensen KB, Kreiner S, Mesbah M, editors. Rasch models in health. Hoboken: John Wiley & Sons Inc.; 2013. p. 5–25.

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors thank Jari Appelgren for guidance during the multiple regression analysis.

Funding

The data collection was funded by the Norwegian Nurses’ Organisation [grant number 14/0023] and the Inland Norway University of Applied Sciences [grant number 2013/1565].

Author information

Authors and Affiliations

  1. Department of Health and Nursing Sciences, Faculty of Social and Health Sciences, Inland Norway University of Applied Sciences, PO Box 400, N-2418, Elverum, Norway

    Hanne Søberg Finbråten, Gun Nordström, Anne Trollvik & Bodil Wilde-Larsson

  2. Department of Health Sciences, Faculty of Health, Science and Technology, Nursing science, Karlstad University, SE-65188, Karlstad, Sweden

    Hanne Søberg Finbråten & Bodil Wilde-Larsson

  3. Norwegian Centre for Science Education, Faculty of Mathematics and Natural Sciences, University of Oslo, PO Box 1106, Blindern, N-0317, Oslo, Norway

    Øystein Guttersrud

  4. Department of Nursing and Health Promotion, Faculty of Health Sciences, OsloMet – Oslo Metropolitan University, PO Box 4, St Olavs plass, N-0130, Oslo, Norway

    Kjell Sverre Pettersen

Authors
  1. Hanne Søberg Finbråten
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Øystein Guttersrud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gun Nordström
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kjell Sverre Pettersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anne Trollvik
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bodil Wilde-Larsson
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors (HSF, ØG, GN, KSP, AT and BWL) contributed to the study design. HSF performed the data analyses under supervision from ØG and Jari Appelgren. HSF wrote the draft of the manuscript. All authors (HSF, ØG, GN, KSP, AT and BWL) contributed to reading, revising and approving the final manuscript.

Corresponding author

Correspondence to Hanne Søberg Finbråten.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Norwegian Social Science Data Service (NSD), ref. 38917. In Norway, all research projects processing personal data must apply for an approval from NSD. The NSD do an overall assessment whether collecting, registering, storing, sharing and/or publishing personal data in the research project meet the requirements of data protection legislation and are in line with ethical guidelines. On basis of this assessment, NSD decides whether it is necessary to receive an additional approval from the Regional Commitees for Medical and Health Research Ethics. According to NSD, our study did not require ethics approval from an ethics committee. Subject participation was voluntary, and questionnaires were completed anonymously. We considered returning a completed questionnaire as consent to participate in the project. This procedure for obtaining informed consent was stated in the information letter that was sent to the participants, and this procedure was also approved by the NSD.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Finbråten, H.S., Guttersrud, Ø., Nordström, G. et al. Explaining variance in health literacy among people with type 2 diabetes: the association between health literacy and health behaviour and empowerment. BMC Public Health 20, 161 (2020). https://doi.org/10.1186/s12889-020-8274-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12889-020-8274-z

Keywords

BMC Public Health

ISSN: 1471-2458

Contact us