Data analyses resulted in the identification of twelve themes, which are described in this section and illustrated by interviewees' quotes. All quotes were translated from Dutch by the first author, and checked by the second author. We categorized the themes according to the main areas of interest in the study: a response to the design and content of the information (thinking aloud), the purpose of the information (probing), and the use of the information (choice task).
Response to the design
Participants often spoke about the design of the website, focusing on aspects such as the amount of information on one webpage (theme 1), information complexity and organization (theme 2), usability of the webpage (theme 3) and the appearance of the information (theme 4). Participants wanted to go quickly through the information and preferred information that is clear at first sight. Generally, negative comments were made about the amount of information (theme 1), such as the following:
How I feel about it? It is too much. I have to consider it line by line. It's too much for one webpage.
The number of health plans is overwhelming. You should view all and then wonder 'what was at the top?'. So you must actually move back and forth. I would not prefer this.
Well, I have to go through a lot, based on this information. Because if you have a number of your own criteria, you still got to do a lot of work to specifically find out.
It is clear that participants felt overwhelmed by the amount of information, which sometimes caused them to stop considering it. It was striking how often consumers said that it was too much immediately after providing them with the information. Some people described their feeling by words as 'overwhelming', 'confused', and 'disorderly'. In contrast, some participants were satisfied with the presented quantity.
Comments were made about the complexity and organization of the design (theme 2):
Well, I think that this website appears calm, compared to the other one. It is more conveniently arranged, and has clear components. This really works for me.
I mean the structure of the information. I feel that the structure is not straightforward. But that's also a personal matter, I think.
From these quotes we see that interviewees' evaluation of the complexity was related to how the information was organized. Interviewees also frequently touched upon the usability or user-friendliness of the website (theme 3):
I have to read the information vertical. That's very bad, because I have to turn my head.
It is not clear that these aspects are clickable.
These quotes tell us that the vertical display of quality indicators and the clickability of aspects of choice are barriers to an easy use of the information. Other barriers were mentioned as well, namely the absence of an option to fasten text in the upper part of a table while scrolling down, and the ambiguous display and content of mouse-overs.
Concerning the appearance of the information (theme 4), interviewees criticized the layout, type face, and the use of colors, as the following quotes illustrate:
I think it is just a messy layout. Letters that don't fit in a box. It is a noisy site, Look, holes and corners are everywhere.
This site is nice and open. A lot of white and bright colors. And a large font.
Response to the content of information
Participants' thoughts focused on different aspects related to the actual information content: the importance of quality indicators (theme 5), the interpretation of information (theme 6), a comparison of the information to their own experiences and ideas (theme 7), and the quality of the presented information (theme 8).
Many interviewees spontaneously attributed importance to the presented quality indicators (theme 5), and further focused on those aspects that they prioritized:
I focus on the opinion of family doctors. That's something that I find important. What my family doctor would think about the quality of hospitals.
Let's see: I think waiting times are important. I see that hospital A has waiting time period of 7 weeks, which I think is just too long.
Almost all information was considered important, and some interviewees even wanted more detailed information, which is hard to reconcile with their feelings of being overwhelmed by the amount of information. The following quote nicely illustrates this inconsistency:
I'd like to have more background information. What's the meaning of the stars? How much stars are there? What's the purpose of "performance indicators"? The number of beds? The number of single and double rooms? That can be included in the information. But it has to be more straightforward than it is now.
Most participants tried to interpret the information, and misunderstood a lot of it (theme 6):
"Opinion of ex-patients" means that these patients had a new hip and evaluate whether they are satisfied about it. Perhaps these patients had to fill out a questionnaire about that. But whether these questionnaires are used for this website. I don't know. Perhaps through the Internet. But it contains an opinion on satisfaction, I guess.
The bar chart says nothing. It is not clear what this actually means, "reimbursement of claims". Then you get scores of never, sometimes, usually, and always. Does this mean that a claim is always reimbursed in one year?
The second quote illustrates that bar charts were incorrectly interpreted. The scores of never, sometimes, usually, and always actually indicate how often claims were reimbursed correctly. Similarly, interviewees had difficulty interpreting symbols, especially when these were based on relative performance scores (performance that is worse than average, average, or better than average). The association between relative and absolute performance was not always clear, as can be seen in the following quote:
Health plan A has one star on all aspects. That's very bad. In my mind, they shouldn't have given one star to a rating of 7.4. That is too high. So, I doubt whether this rating of 7.4 really is an answer of respondents. I don't believe that.
This participant had trouble to understand that one star means 'a worse than average performance', and not an absolute 'bad performance'. Presenting absolute global ratings simultaneously (7.4) caused confusion.
Furthermore, participants found it hard to understand conflicting information when, for example, some hospitals performed good on one quality aspect, but bad on another aspect. One participant stated,
But that's very strange. Look at this. Here we find a contradiction. Look, that can't be possible. The performance of this hospital according to patients is very high. But the "public availability of data" is not so well. Oh, but wait a minute. Oh, I see. If you look at it a little bit longer, all sorts of questions come up. But now I think I understand. Hospital C is very reserved as to providing quality information. Oh dear, I find this very annoying. If I'd only had a fast glance, I wouldn't have understood.
The naming of several quality indicators was poorly interpreted, such as 'reimbursement', 'restitution', 'test opinion', 'public availability of data', 'quality indicators', and 'clinical specialty'. For example,
I don't understand the term "reimbursement". Perhaps I can read somewhere what that means? "Restitution", or "mixed", or "in kind". Does "reimbursement" mean that I get my medication directly?
"Quality indicators" represent the extent to which they pay attention to the patient. That's interesting, of course. Because it indicates whether they find patients important. Well, not always, but more and more, though.
In reality, the term 'reimbursement' refers to how insurance claims are processed: either directly, without interposition of the consumer, or indirectly via the consumer. 'Quality indicators' do in reality reflect objective performance indicators, and not merely patient-centeredness. Global ratings were often misinterpreted as well. Many participants thought that global ratings were composed of other presented indicators. In fact, global ratings are given by patients on a scale from zero to ten on a questionnaire item. In contrast, some terms were well understood, such as 'opinion of ex-patients', 'information', and 'telephone assistance'.
We further learned that many participants were comparing the presented information to their own experiences and ideas (theme 7):
Overall I think the score of health plan A is a bit low. My experience is that they are not that bad.
But Hospital C is my first choice, although I live in place A. I just don't like the two hospitals near place A.
We observed this tendency in interviewees' responses to the information content, but it appeared to be related to their hypothetical choices as well:
I don't want to go to hospital B because of an old-fashioned idea that I have. Because there were several incidents in my surrounding in that hospital. And that's why I'm not inclined to go to that hospital, as good as it may be now.
This quote tells us that the interviewee would not choose for hospital B, because its performance conflicted with ideas already in mind.
A final aspect related to participants' reactions to information content was that the quality of the information itself was frequently questioned (theme 8):
The "opinion of ex-patients". Well, maybe only two patients were questioned? So I'd like to know more about this website. I'd like to know how the opinion of ex-patients, how that works. Was the sample large enough?
When there are question marks, just like here, you can question the adequacy of the information.
From these quotes, we see that questions were raised about the completeness and reliability of the information. In addition to these issues, interviewees also commented on the magnitude of quality differences.
Use of information
Participants' thoughts often focused on the potential use of the information in daily life (theme 9):
I'd never make a decision based on this kind of information. Perhaps rather on personal experiences of others, and I would ask others.
I didn't know that this kind of information is available. So, now that I know, I think it's interesting information. It's tempting to look at it at some time. So I think I would look at it.
If I had to make a choice, I would look for things that I find important. But I think I know to which hospital I'd want to go. That's because I have experience with that hospital and I'm satisfied. If you are very satisfied with a particular hospital, and that hospital does not have so many stars, I'd rely on my own experience.
These quotes illustrate the variation among consumers' interest to use the information in daily life. Some interviewees thought that comparing providers on different quality aspects is a tough and time consuming activity. Others felt that information could be a helpful tool for their healthcare decisions. One agreed that other information sources were required to make an informed choice, either instead of or complementary to comparative information. Frequently cited information sources were their own experiences and perceptions, experiences of relevant others, provider image, advice of family doctor and health insurer, and media reports. How the information could be used in daily life (theme 9) appeared to be associated with the design and content of the website, such as the amount of information, and with the perceived relevance of quality indicators.
Interviewees also differed concerning the decision strategies used to make a hypothetical decision during the choice task of the interview (theme 10):
Well, I find quality of care most important. Yes, the score of "clinical specialty". And then I'd choose for Hospital A. Because that hospital is the nearest. And because Hospital A still has a good reputation. That reputation is not contradicted on this website. But, apparently, 37% of the requested data were provided. I'm not immediately sad with a performance of three stars on "clinical specialty". And the "opinion of patients". I think that's important, but they do not highly differ from each other, I see. And besides, this score is all right for hospital A, a score of 7.7.
If I have to choose now, on basis of these data, I would find it hard and complicated. Perhaps then I'd focus on, God help me, the global rating of 8.2.
I concentrate on aspects where large differences exist. These are found on "clinical specialty". That's where differences exist. "Opinion of family doctor" is not available. But especially this one with two stars. I think that's bad, compared to the others.
Many of the strategies listed in Table 1 were used. Several participants systematically weighted the information. For example, they examined quality aspects one by one (WADD) or first defined most important aspects and then compared performance (LEX). Additionally, strategies by which providers were excluded one by one when performance did not meet requirements were often used (EBA). Frequently observed as well was the strategy to count up the number of good and bad scores on different aspects (FRQ). However, more simple associative strategies and shortcuts were also used. Some interviewees, for instance, chose the provider with the highest global rating (Performance Oriented). Yet, even more simple strategies, such as choosing the provider first named (In Store), providers with a familiar name (Affect Referral), providers chosen before (Habitual Heuristic), or the cheapest provider (Price Oriented), were used. Most participants adopted a mix of the above mentioned decision strategies, particularly those who systematically weighted information.
Many interviewees had difficulty making the hypothetical decision. First, several participants were not able to complete this task, because they needed additional information from other sources. Second, it took most consumers lots of time to complete the task. Third, several participants used shortcuts to decide, which indicates that the amount of information was too large for them to process systematically. Apart from these difficulties, we found incongruity between what consumers said to find important or what they would do, and what they actually did when making a choice. For example, during thinking aloud, several participants came up with aspects that they prioritized. Later on, however, these aspects were not weighted in their decisions.
Purpose of the information
Participants had clear ideas about the direct purpose of the information (theme 11). Although a few consumers thought that the information was designed to inform health insurance companies or hospitals themselves, most participants related the information to consumer choice in healthcare:
This information attempts to rate hip surgery quality. The aim is to get some insight into this quality. Then I can choose what's important to me. Should the clinician be excellent? Should the hospital be near? You get some information on these aspects.
The intention is to provide a summary of all options, so we can make choices in healthcare and live happily ever after.
If people want more freedom of choice in their health insurance, they obviously want to know what they can ensure, what is available, how fast and reliable such insurers are. That's what you are looking for when using this information.
We see that consumers generally knew why the information is presented to them. However, this does not necessarily mean that they actually wanted to use it, for example, if there are few provider-differences.
Participants' thoughts also concentrated on the purpose of different quality indicators (theme 12). Most consumers were able to describe the purpose of different quality indicators, in particular when they saw benefits of presenting the information:
"Opinion of family doctor". Family doctors do have an idea about how clinicians do their work. And these doctors give their view as well. They give stars, or they say they have no idea.
The global rating for health plans is presented because people are used to think in numerals. Therefore, a rating from 0 to 10 immediately says something. If a health plan has a global rating of 5, everyone thinks 'Oh no, that's not where you'd have your insurance'. It's as simple as that.