How do healthcare consumers process and evaluate comparative healthcare information? A qualitative study using cognitive interviews

Information processing

From cognitive science and decision making literature, we know how information can be processed by consumers. Broadly, people either think in an analytical (rule-based) or experiential mode [19]. The analytical mode concerns conscious, deliberative, attribute-by-attribute reasoning, which is relatively slow. Dijksterhuis and colleagues [13] argued that human consciousness has limited capacity; causing consumers to take into account only a subset of relevant information. Therefore, the analytical mode is usually applied when information is relatively simple. In contrast, the experiential mode consists of more associative, automatic reasoning, occurring relatively fast. People often apply this processing mode, using shortcuts or intuitive heuristics, especially when large amounts of information are concerned [20–22]. Usually, consumers only scan information [23], looking for information they want [24], and in the light of questions they already have in mind, their knowledge, and their expectations [25, 26].

Choices based on comparative healthcare information typically involve the following demands: 1) processing technical terms and complex ideas; 2) comparing multiple alternatives on several attributes; and 3) weighting various factors according to individual preferences [27]. These processes and trade-offs are known to be difficult [28] and provoke fast and frugal decision making [29]. Furthermore, comparative healthcare information seems to produce preferences that are 'constructed' while sorting through information ('constructed preference') [30, 31]. This means that consumers have no fixed ideas about their priorities in healthcare quality, and construct them depending on how the information is provided.

To summarize, it is known which general processing strategies can be applied by consumers, but relatively little is known with respect to comparative healthcare information. The literature suggests that it is a complex job to process comparative healthcare information, and Internet research has identified many guidelines to improve website usability. However, hardly any studies have comprehensively examined the information processing strategies of consumers themselves. To be able to understand the difficulties and bottlenecks consumers face, an open, qualitative approach using real online information is therefore needed. With this study, we aimed to gain insight into consumers' own thoughts, interpretations, and evaluations of this kind of information. Our research question was: "How do consumers process and evaluate comparative healthcare information?'"

Cognitive interviews

We performed the interviews in a small, private room, and an assistant made notes during the interview. Interviews were recorded on audio tape with permission of the interviewees. Participants filled out an informed consent form and a questionnaire about demographic variables. After that, they were rewarded with a token gift, namely a gift voucher of 15 euro. Each interview lasted about one hour. The interviews were performed by a team of five researchers, who had a joint instruction before the start of the interviews. After a first round of interviews, interviewers were debriefed on the main findings and aspects to pay attention to in the next set of interviews.

Materials

Participants were provided with three Dutch web pages containing comparative healthcare information as visual prompts: 1) information on the quality of hospital care concerning hip surgery (Figure 1[33]); (2) information on the quality of health plans (Figure 2[34]); and 3) information on both quality and premiums of health plans (Figure 3[35]). At the time of the interviews, these websites were relatively well-known public reporting initiatives in the Netherlands. In addition, we chose to test these websites because clinical performance indicators -defined by the Dutch Inspectorate for healthcare [36] are presented- as well as patient experience information measured with the Consumer Quality Index (a set of standardized patient surveys) [37]. The pages were presented in six different orders (3*2) to control for potential order effects.

Data collection and sample

Analysis

The original audio tapes were transcribed and the transcriptions were analyzed by one researcher. A second researcher independently analyzed a subset of the transcriptions. Both researchers conducted descriptive thematic analysis, consisting of an open coding and an axial coding phase [38, 39]. Open coding was characterized by fragmenting [40]: relevant themes were extracted, categorized and classified. After the research team verified the themes, relationships between categories were identified in the axial coding phase. Since we were most interested in consumers' own spontaneous interpretations and information processing, we focused on data derived from thinking aloud. Subsequently, we analyzed answers to specific questions. In the descriptions of the emerging themes, we used the following guideline to connote the quantity of how often themes were mentioned [41]. Many, often, frequently and generally are used when a theme is mentioned by more than 75% of participants; common and several when mentioned by about 50% to 75% of participants; some when mentioned by about 25% to 50% of participants; few when mentioned by less than 25% of participants.

Response to the design

Participants often spoke about the design of the website, focusing on aspects such as the amount of information on one webpage (theme 1), information complexity and organization (theme 2), usability of the webpage (theme 3) and the appearance of the information (theme 4). Participants wanted to go quickly through the information and preferred information that is clear at first sight. Generally, negative comments were made about the amount of information (theme 1), such as the following:

How I feel about it? It is too much. I have to consider it line by line. It's too much for one webpage.

The number of health plans is overwhelming. You should view all and then wonder 'what was at the top?'. So you must actually move back and forth. I would not prefer this.

Well, I have to go through a lot, based on this information. Because if you have a number of your own criteria, you still got to do a lot of work to specifically find out.

It is clear that participants felt overwhelmed by the amount of information, which sometimes caused them to stop considering it. It was striking how often consumers said that it was too much immediately after providing them with the information. Some people described their feeling by words as 'overwhelming', 'confused', and 'disorderly'. In contrast, some participants were satisfied with the presented quantity.

Comments were made about the complexity and organization of the design (theme 2):

Well, I think that this website appears calm, compared to the other one. It is more conveniently arranged, and has clear components. This really works for me.

I mean the structure of the information. I feel that the structure is not straightforward. But that's also a personal matter, I think.

From these quotes we see that interviewees' evaluation of the complexity was related to how the information was organized. Interviewees also frequently touched upon the usability or user-friendliness of the website (theme 3):

I have to read the information vertical. That's very bad, because I have to turn my head.

It is not clear that these aspects are clickable.

These quotes tell us that the vertical display of quality indicators and the clickability of aspects of choice are barriers to an easy use of the information. Other barriers were mentioned as well, namely the absence of an option to fasten text in the upper part of a table while scrolling down, and the ambiguous display and content of mouse-overs.

Concerning the appearance of the information (theme 4), interviewees criticized the layout, type face, and the use of colors, as the following quotes illustrate:

I think it is just a messy layout. Letters that don't fit in a box. It is a noisy site, Look, holes and corners are everywhere.

This site is nice and open. A lot of white and bright colors. And a large font.

Response to the content of information

Participants' thoughts focused on different aspects related to the actual information content: the importance of quality indicators (theme 5), the interpretation of information (theme 6), a comparison of the information to their own experiences and ideas (theme 7), and the quality of the presented information (theme 8).

Many interviewees spontaneously attributed importance to the presented quality indicators (theme 5), and further focused on those aspects that they prioritized:

I focus on the opinion of family doctors. That's something that I find important. What my family doctor would think about the quality of hospitals.

Let's see: I think waiting times are important. I see that hospital A has waiting time period of 7 weeks, which I think is just too long.

Almost all information was considered important, and some interviewees even wanted more detailed information, which is hard to reconcile with their feelings of being overwhelmed by the amount of information. The following quote nicely illustrates this inconsistency:

I'd like to have more background information. What's the meaning of the stars? How much stars are there? What's the purpose of "performance indicators"? The number of beds? The number of single and double rooms? That can be included in the information. But it has to be more straightforward than it is now.

Most participants tried to interpret the information, and misunderstood a lot of it (theme 6):

"Opinion of ex-patients" means that these patients had a new hip and evaluate whether they are satisfied about it. Perhaps these patients had to fill out a questionnaire about that. But whether these questionnaires are used for this website. I don't know. Perhaps through the Internet. But it contains an opinion on satisfaction, I guess.

The bar chart says nothing. It is not clear what this actually means, "reimbursement of claims". Then you get scores of never, sometimes, usually, and always. Does this mean that a claim is always reimbursed in one year?

The second quote illustrates that bar charts were incorrectly interpreted. The scores of never, sometimes, usually, and always actually indicate how often claims were reimbursed correctly. Similarly, interviewees had difficulty interpreting symbols, especially when these were based on relative performance scores (performance that is worse than average, average, or better than average). The association between relative and absolute performance was not always clear, as can be seen in the following quote:

Health plan A has one star on all aspects. That's very bad. In my mind, they shouldn't have given one star to a rating of 7.4. That is too high. So, I doubt whether this rating of 7.4 really is an answer of respondents. I don't believe that.

This participant had trouble to understand that one star means 'a worse than average performance', and not an absolute 'bad performance'. Presenting absolute global ratings simultaneously (7.4) caused confusion.

Furthermore, participants found it hard to understand conflicting information when, for example, some hospitals performed good on one quality aspect, but bad on another aspect. One participant stated,

But that's very strange. Look at this. Here we find a contradiction. Look, that can't be possible. The performance of this hospital according to patients is very high. But the "public availability of data" is not so well. Oh, but wait a minute. Oh, I see. If you look at it a little bit longer, all sorts of questions come up. But now I think I understand. Hospital C is very reserved as to providing quality information. Oh dear, I find this very annoying. If I'd only had a fast glance, I wouldn't have understood.

The naming of several quality indicators was poorly interpreted, such as 'reimbursement', 'restitution', 'test opinion', 'public availability of data', 'quality indicators', and 'clinical specialty'. For example,

I don't understand the term "reimbursement". Perhaps I can read somewhere what that means? "Restitution", or "mixed", or "in kind". Does "reimbursement" mean that I get my medication directly?

"Quality indicators" represent the extent to which they pay attention to the patient. That's interesting, of course. Because it indicates whether they find patients important. Well, not always, but more and more, though.

In reality, the term 'reimbursement' refers to how insurance claims are processed: either directly, without interposition of the consumer, or indirectly via the consumer. 'Quality indicators' do in reality reflect objective performance indicators, and not merely patient-centeredness. Global ratings were often misinterpreted as well. Many participants thought that global ratings were composed of other presented indicators. In fact, global ratings are given by patients on a scale from zero to ten on a questionnaire item. In contrast, some terms were well understood, such as 'opinion of ex-patients', 'information', and 'telephone assistance'.

We further learned that many participants were comparing the presented information to their own experiences and ideas (theme 7):

Overall I think the score of health plan A is a bit low. My experience is that they are not that bad.

But Hospital C is my first choice, although I live in place A. I just don't like the two hospitals near place A.

We observed this tendency in interviewees' responses to the information content, but it appeared to be related to their hypothetical choices as well:

I don't want to go to hospital B because of an old-fashioned idea that I have. Because there were several incidents in my surrounding in that hospital. And that's why I'm not inclined to go to that hospital, as good as it may be now.

This quote tells us that the interviewee would not choose for hospital B, because its performance conflicted with ideas already in mind.

A final aspect related to participants' reactions to information content was that the quality of the information itself was frequently questioned (theme 8):

The "opinion of ex-patients". Well, maybe only two patients were questioned? So I'd like to know more about this website. I'd like to know how the opinion of ex-patients, how that works. Was the sample large enough?

When there are question marks, just like here, you can question the adequacy of the information.

From these quotes, we see that questions were raised about the completeness and reliability of the information. In addition to these issues, interviewees also commented on the magnitude of quality differences.

Use of information

Participants' thoughts often focused on the potential use of the information in daily life (theme 9):

I'd never make a decision based on this kind of information. Perhaps rather on personal experiences of others, and I would ask others.

I didn't know that this kind of information is available. So, now that I know, I think it's interesting information. It's tempting to look at it at some time. So I think I would look at it.

If I had to make a choice, I would look for things that I find important. But I think I know to which hospital I'd want to go. That's because I have experience with that hospital and I'm satisfied. If you are very satisfied with a particular hospital, and that hospital does not have so many stars, I'd rely on my own experience.

These quotes illustrate the variation among consumers' interest to use the information in daily life. Some interviewees thought that comparing providers on different quality aspects is a tough and time consuming activity. Others felt that information could be a helpful tool for their healthcare decisions. One agreed that other information sources were required to make an informed choice, either instead of or complementary to comparative information. Frequently cited information sources were their own experiences and perceptions, experiences of relevant others, provider image, advice of family doctor and health insurer, and media reports. How the information could be used in daily life (theme 9) appeared to be associated with the design and content of the website, such as the amount of information, and with the perceived relevance of quality indicators.

Interviewees also differed concerning the decision strategies used to make a hypothetical decision during the choice task of the interview (theme 10):

Well, I find quality of care most important. Yes, the score of "clinical specialty". And then I'd choose for Hospital A. Because that hospital is the nearest. And because Hospital A still has a good reputation. That reputation is not contradicted on this website. But, apparently, 37% of the requested data were provided. I'm not immediately sad with a performance of three stars on "clinical specialty". And the "opinion of patients". I think that's important, but they do not highly differ from each other, I see. And besides, this score is all right for hospital A, a score of 7.7.

If I have to choose now, on basis of these data, I would find it hard and complicated. Perhaps then I'd focus on, God help me, the global rating of 8.2.

I concentrate on aspects where large differences exist. These are found on "clinical specialty". That's where differences exist. "Opinion of family doctor" is not available. But especially this one with two stars. I think that's bad, compared to the others.

Many of the strategies listed in Table 1 were used. Several participants systematically weighted the information. For example, they examined quality aspects one by one (WADD) or first defined most important aspects and then compared performance (LEX). Additionally, strategies by which providers were excluded one by one when performance did not meet requirements were often used (EBA). Frequently observed as well was the strategy to count up the number of good and bad scores on different aspects (FRQ). However, more simple associative strategies and shortcuts were also used. Some interviewees, for instance, chose the provider with the highest global rating (Performance Oriented). Yet, even more simple strategies, such as choosing the provider first named (In Store), providers with a familiar name (Affect Referral), providers chosen before (Habitual Heuristic), or the cheapest provider (Price Oriented), were used. Most participants adopted a mix of the above mentioned decision strategies, particularly those who systematically weighted information.

Many interviewees had difficulty making the hypothetical decision. First, several participants were not able to complete this task, because they needed additional information from other sources. Second, it took most consumers lots of time to complete the task. Third, several participants used shortcuts to decide, which indicates that the amount of information was too large for them to process systematically. Apart from these difficulties, we found incongruity between what consumers said to find important or what they would do, and what they actually did when making a choice. For example, during thinking aloud, several participants came up with aspects that they prioritized. Later on, however, these aspects were not weighted in their decisions.

Purpose of the information

Participants had clear ideas about the direct purpose of the information (theme 11). Although a few consumers thought that the information was designed to inform health insurance companies or hospitals themselves, most participants related the information to consumer choice in healthcare:

This information attempts to rate hip surgery quality. The aim is to get some insight into this quality. Then I can choose what's important to me. Should the clinician be excellent? Should the hospital be near? You get some information on these aspects.

The intention is to provide a summary of all options, so we can make choices in healthcare and live happily ever after.

If people want more freedom of choice in their health insurance, they obviously want to know what they can ensure, what is available, how fast and reliable such insurers are. That's what you are looking for when using this information.

We see that consumers generally knew why the information is presented to them. However, this does not necessarily mean that they actually wanted to use it, for example, if there are few provider-differences.

Participants' thoughts also concentrated on the purpose of different quality indicators (theme 12). Most consumers were able to describe the purpose of different quality indicators, in particular when they saw benefits of presenting the information:

"Opinion of family doctor". Family doctors do have an idea about how clinicians do their work. And these doctors give their view as well. They give stars, or they say they have no idea.

The global rating for health plans is presented because people are used to think in numerals. Therefore, a rating from 0 to 10 immediately says something. If a health plan has a global rating of 5, everyone thinks 'Oh no, that's not where you'd have your insurance'. It's as simple as that.

Study strengths and limitations

Our study is the first to investigate in-depth consumers' own thoughts about Dutch comparative healthcare information. An important strength is that real online information was used, with all its complexities included. We used three different websites which are typical for websites internationally [42], and the results were of the same order for these three websites. The open qualitative approach resulted in detailed information about the interpretations and experiences of consumers themselves. Our findings therefore provide a thorough and valid understanding of consumers' experiences and the difficulties that they face. However, our small scale study does not allow for specific recommendations concerning presentation formats. More controlled experiments and observational studies are needed to further investigate decision making using online comparative healthcare information.

A limitation of our study is that neither low educated people nor ethnic minorities participated, although they were invited. This might suggest that certain consumer subgroups are not interested in comparative healthcare information, think that participating is too difficult, or that their jobs or lives are less flexible. Lower educated people are known to have more difficulty understanding healthcare quality information. In addition, the use of Internet is limited among lower educated people and ethnic minorities [43]. This means that their use of the information might even be more complicated than was shown among our participants. Further research should be conducted to investigate these potential problems concerning accessibility of information and equity.

Our findings were also limited by the fact that our participants were not facing a real decision. We forced consumers to choose, which can bias the results towards the 'safer', more average option [44]. Patients facing a real decision in healthcare might weigh other aspects than volunteers in hypothetical choices. Real healthcare consumers usually do not have a 'no choice' option either, though they can decide to leave the choice of a provider to their family doctor who refers them, or-in market research terms- who acts as a 'surrogate consumer' [45]. Is it important to realize that real decisions in healthcare involve many factors within a healthcare trajectory, rather than merely visiting one website to get informed [46].

Important findings

A key finding is the tension between the great amount of information consumers stated to find important and how sporadically they actually incorporated this information into their decisions. Furthermore, ideas on which quality aspects are important to consider changed during the course of the interview. This inconsistency between (initial) interest in certain information and (later) leaving out of consideration has been found previously [15, 16, 28]. It suggests that preferences are constructed gradually during the interview [16, 30, 47], and are not as predictable as is sometimes assumed. As already mentioned as a study limitation, the prescriptive nature of our question (what would you do.?) might contribute to differences in what people said to what they actually did. Another explanation might be found in the data itself; when there are few provider-differences on aspects that one considers important, that aspect is not weighted in the eventual choice, though it is still considered important.

Considering the difficulties that participants experienced when processing the amount of presented information and making a choice, we want to emphasize the perceived barrier of too much information. It is known that people can only process about six pieces of information at a time and are easily overwhelmed by information [11]. Therefore, providing all available information is not the most effective way to stimulate informed choices [48, 49]. As argued by Eysenbach, websites do not always need to be complete and present the full information spectrum about a particular disease or healthcare topic. Indeed, consumers are able to gather information from various sources and sites [16, 50]. Therefore, websites should rather provide conceivable overviews with small numbers of providers and the most relevant quality aspects, and offer more detailed information into step-by-step pages, an approach corresponding to humans' need for generic to specific information [14, 51]. This deep-linking approach, which has been frequently cited in the broader context of consumer health informatics, [16, 23] could reconcile consumers' desire for more information without overwhelming them. Gerteis and colleagues [52] suggested to use evaluative formats (for example stars) on a first page and let consumers drill down to more detailed bar graphs.

Consumers found it hard to process contradictory information, such as a hospital with high performance on one quality aspect, and low on another aspect, which also corresponds to previous findings [28]. Conflicting information asks for more cognitive effort, which forces consumers to make trade-offs of important aspects and to rely on intuitive heuristics. Comparative healthcare information usually contains contradictory information. Initiatives to prepare or train consumers about potential contradictions might remove some confusion. However, effectively processing contradictory information requires relatively complex strategies and will continue to be difficult.

Only a few consumers deliberately processed all information. More often, only parts of the information were considered, particularly information about familiar providers. This suggests that consumers are not interested in all information, but rather want to check how particular providers perform compared to others. This corresponds to what we know from cognitive science about interpretation in light of questions and information already in mind when viewing information, such as reviews of other patients or media reports [16, 26]. Therefore, it seems important to relate comparative healthcare information to alternative information that consumers find familiar [48]. For example, anecdotal or patient review information (such as on NHS Choices in England [53] and Consument en de Zorg in the Netherlands [54]) might be an interesting source of additional consumer information. Further research is needed to assess whether and how these different types of information should be integrated.

Various strategies were applied to choose providers, varying from systematic reasoning to more intuitive, experiential reasoning using only parts of information. Both alternative-based reasoning and attribute-based reasoning were used, which are both known to be used when information is presented in a matrix format [55, 56]. In terms of web design, it means that pages presenting information need to be highly flexible, and preferably allow selections on both prioritized aspects and particular providers of interest.

A substantial number of the participants was interested in the presented information, and understood the purpose of the information. In line with a previous qualitative study [57], consumers appear to comprehend information among main lines, but have difficulty understanding more detailed information and concepts. Findings seem to contradict the notion of some researchers that consumers are not interested in comparative healthcare information. Perhaps the healthcare market is different from other markets where people prefer not to choose, e.g. the energy market [58, 59], in the sense that healthcare is a product that is of interest to people. There are many documentaries and talk shows about health and healthcare, and hardly any about gas and electricity. So even if consumers are not willing to choose, they can still find healthcare information interesting.