An observed, prospective field study to evaluate the performance and acceptance of a blood-based HIV self-test in Canada

Abstract

Background

Self testing for HIV is a targeted intervention with the potential to increase the access, uptake and frequency of HIV testing and more effectively reach the undiagnosed, especially in priority populations. The objectives of this study were to (1) evaluate the INSTI HIV self-test performance compared with laboratory reference testing, (2) document if intended users can perform the steps to use the HIV self-test device, and (3) document if intended users can successfully interpret contrived positive, negative, and invalid results. Study was intended to be submitted to Health Canada for review for regulatory approval purposes.

Methods

The study used a cross-sectional design and recruited consenting adults who were representative of intended users of HIV self-testing from four community sites across Ontario, Québec, and Manitoba between August 2019 and March 2020. The results of the observed HIV self-test were compared with results of the Abbott Architect HIV Ag/Ab Combo test. Usability outcomes for critical (e.g., lancing finger, blood droplet into bottle, shaking bottle four times) and noncritical self-test procedure steps were also determined.

Results

Overall, 77% (n = 522) of participants were between 18 and 45 years of age, 61% (n = 410) were male, 71% (n = 480) had some college or more education, and 45% (n = 307) were employed; identity for race and ethnicity: Caucasian (44%; n = 296), African, Caribbean or Black (17%; n = 113), Indigenous [First Nations, Métis or Inuit] (14%; n = 95), Asian (16%; n = 106), Latin American (7%; n = 46). Primary performance analysis on 678 completed HIV self-tests revealed a positive percent agreement of 100% (5/5, 95% CI: 43.6–97.0%) and a negative percent agreement of 99.5% (614/617, 95% CI: 98.6–99.8%) with the comparator method. The overall percent agreement of results interpretation between participant and observer was 93.5% (n = 633). For the 708 participants who took part in the usability study, the average success rate for steps determined to be “critical” for successful completion of the test was 92.4%. 97% (n = 670) of participants found the instructions easy to follow, and 95% (n = 655) of participants indicated that they would use the test again. Of the 404 participants who interpreted the strong positive, weak positive, negative, and invalid contrived results, successful interpretation ranged from 90.6% (for weak positive, n = 366) to 99.3% (for negative, n = 401).

Conclusions

The addition of a regulatory-approved self-test into the Canadian HIV testing landscape could significantly increase HIV testing rates. Having a blood-based HIV self-test approved in Canada can offer an accurate, acceptable, and simple alternative to facility-based HIV testing, particularly when impacted by Coronavirus pandemic restrictions.

Background

Unlike other G7 countries, Canada is not seeing a reduction in the overall number of new people being diagnosed with HIV. Recent data from the Public Health Agency of Canada (PHAC) indicate that an estimated 2242 new HIV infections occurred in Canada in 2018 compared to 2165 in 2016, which is an increase of 3.6%. Approximately 13% of people living with HIV in Canada are still not diagnosed, which represents over 8000 individuals living with HIV who are unaware of their status [1]. Although Canada had endorsed the initial UNAIDS 90–90-90 target (90% diagnosed, 90% of those on treatment and in care, and 90% of those achieving viral suppression) [2], only 87% of the estimated 62,050 people living with HIV are diagnosed (1st 90 target) – indicating these 8000+ individuals who have undiagnosed HIV infection across the country may not be adequately served by or connected to our health care system. For those people diagnosed with HIV, 85% were on antiretroviral treatment (2nd 90 target), and of those, 94% had suppressed viral load (3rd 90 target) by the end of 2018 [3]. While we have reached one of three key UNAIDS targets, there are nearly 20,000 people who have not yet been diagnosed or who are not on antiretroviral therapy in Canada, which represents nearly a third of the total people living with HIV in Canada. Targeted interventions for testing, such as using HIV self-testing to reach the undiagnosed particularly amongst key and hard to reach populations, and supporting more people to manage and adhere to treatment, and achieve viral suppression are needed to achieve Canada’s original UNAIDS commitment to all three of the 90–90-90 targets and now move successfully towards ending its HIV epidemic in the next 5 years. Indeed, diagnosing HIV is critically important for reaching targets aimed at controlling the HIV epidemic [4].

In 2016, the WHO recommended HIV self-testing as an alternative to conventional facility-based testing [5]. By 2018, 59 countries had developed national policies on HIV self testing, however Canada had not finalized a national policy [6]. In many countries, self-testing for HIV is an established targeted intervention with the potential to increase the access, uptake and frequency of HIV testing, and could potentially become a high impact, low cost, and empowering alternative for those who may not otherwise test, notably for populations at high risk for HIV infection [7]. In a review on modern diagnostic technologies for HIV, the authors conclude that these novel technologies, including blood-based HIV self tests, show promise as they are associated with ease of use, high diagnostic accuracy, rapid detection, and the ability to detect HIV-specific markers [8]. There is a growing body of supporting evidence showing the acceptance and usability of HIV self-testing in various global key populations and groups [7, 9, 10], however similar study data for blood-based HIV self testing within the Canadian population had not existed prior to this study.

Three earlier studies were conducted in Montreal, Quebec with oral fluid HIV self tests in key populations and low risk populations. In these studies, HIV self-testing using oral fluid was demonstrated to be an acceptable and feasible testing solution, with data on technology innovations that can help increase testing and expedite linkages to care [11, 12]. A survey of Canadian stakeholders demonstrated that HIV self testing is desired by Canadians yet, operational models and linkage data need to be thought through [13]. Despite these favourable outcomes supporting the use of HIV self-testing, the first blood-based HIV self test has only just been approved by Health Canada on November 3, 2020 with implementation and scale-up critical as next steps.

To assess if favourable outcomes similar to those of the oral fluid self test studies conducted earlier can be achieved with a blood-based self test, this study aims to provide independent data on the INSTI HIV Self-Test (BioLytical Laboratories, Richmond, BC, Canada) performance, acceptance, and usability in the hands of intended users in Canada. The study objectives were to (1) evaluate the device performance, i.e., sensitivity, specificity, and percent agreement compared with laboratory reference testing, (2) document if intended users (non-professional and inexperienced in HIV self-testing) can successfully perform the steps to use the HIV self-test device without product demonstration, and (3) document if intended users can successfully interpret contrived strong positive, weak positive, negative, and invalid results.

Methods

Study design

The study used a cross-sectional design and recruited consenting adults from four community-based sexual health and medical clinics or health centres across Ontario, Québec, and Manitoba between August 2019 and March 2020. Due to the emergence of the COVID-19 infection in populations throughout Canada and related restrictions concerning clinical practice, study sites were forced to discontinue study enrolment by March 18, 2020.

Inclusion criteria

The study was open to the English and French-speaking general public over the age of 18 who met the inclusion criteria (i.e., consented to have venous blood drawn and undergo self-testing by an investigational device) and did not meet any of the exclusion criteria (see Appendix A for full list of inclusion and exclusion criteria). No participant was excluded based on race, gender, ethnicity or sexual orientation. Participants were compensated with $50 at their initial study site enrollment and an additional $25 when they returned for their laboratory testing results.

Study population and sample size

Self-reported HIV risk activity for the year prior to the study enrolment was collected for each participant. Recruitment occurred during routine clinic visits through use of brochures, waiting room posters, and word of mouth. The original sample size for the study was intended to be 900 individuals with unknown HIV status including a minimum of 400 “at-risk” of HIV infection, which is the number required for HIV self-test prospective studies in the published Health Canada guidelines (Health Canada, 2017).

Study protocol

The results of the INSTI HIV Self-Test performed and interpreted by intended users were compared with results of licensed laboratory-based Comparator Methods (CM). The CM used for all sites was the HIV-1/HIV-2 combination antigen/antibody test (Abbott Architect HIV Ag/Ab Combo test, Abbott Laboratories, Abbott Park, IL) which is licensed by Health Canada and is in routine use at the central public health laboratories of each of the participating provinces. The Geenius™ HIV 1/2 Confirmatory Assay (Bio-Rad Laboratories, Marnes-la-Coquette, France) was used for positive confirmation where necessary.

After written informed consent was obtained, one venous blood sample was collected from each participant for testing at a central laboratory by the CM. Each participant then started the self-test – they self-collected a fingerstick blood specimen and performed the INSTI HIV Self-Test, including result interpretation, according to only the manufacturer’s instructions for use. Each self-test performance was directly observed by a trained healthcare professional (Observer). The Observer did not tutor or interact with the participant conducting the INSTI HIV Self-Test but noted errors and other observations about the participant’s test performance. There were 24 items developed, similar to those used in a usability assessment of multiple HIV self-test devices in South Africa [14] to capture usability information including instructions for use, pre-ST preparation, procedure for self-collecting blood sample, self-test procedure, and follow-up procedures. Half of the items (12 in total) were considered “critical” with respect to correct usability. The Observer also interpreted the HIV self-test result immediately after the participant and recorded the result separately. Lastly, a sub-set of participants who volunteered were provided the contrived “mock” result membrane cartridges individually and asked to interpret the results. All survey instruments were developed specifically for this study (see Appendix C for full survey instruments).

The provincial public health laboratories conducted CM testing on serum from venous blood samples collected from study participants and the results were provided back to the sites and subsequently to the central data coordination centre at St. Michael’s Hospital, Toronto. Each central laboratory test result was used for study device performance evaluation with the INSTI HIV Self-Test, however study participants were able to return to the clinic site at a later date to receive their laboratory testing results if desired. All clinical care decisions were based solely on the results of the standard of care HIV testing in place at each site.

Electronic data capture

All participant data were collected by the study observers through the use of pre-loaded tablets at the study sites and stored using Qualtrics (Qualtrics, Provo, UT, USA https://www.qualtrics.com), a survey collection tool that encrypts data using Hypertext Transfer Protocol Secure (HTTPS) and enforces HTTP Strict Transport Security (HSTS). Participant data were stored on local encrypted servers in Toronto at St. Michael’s Hospital for further data analysis. Only designated research team members accessed the password-protected data from the Qualtrics servers. The survey data for each participant were coded with a corresponding unique study ID to maintain privacy and confidentiality.

Ethics approval

This study was approved by respective Research Ethics Boards (REB) for the Ontario, Québec and Manitoba sites participating in the study: University of Toronto REB, St. Michael’s Hospital REB, Veritas IRB Québec, and University of Manitoba HREB. This study was conducted under a Health Canada approved Investigational Testing Authorization (ITA), application No. 276320, issued on December 19, 2018.

Data analysis

All data analysis was conducted using IBM SPSS Statistics version 24. The Primary Efficacy Analysis (performance) includes calculation of positive and negative percent agreement between study participants’ self-interpreted results of the INSTI HIV Self-Test versus the 4th generation Abbott Architect test results. The proportion of study participants’ interpretation of their self-test which is confirmed by the confirmatory test algorithm, and thereby considered “true” will be confirmed following determination of study participant’s data inclusion in either the positive percent agreement (PPA) population or the negative percent agreement (NPA) population. Invalid self-test results and results indicated as “do not know” (n = 56) were excluded from the calculation of PPA and NPA.

The overall 95% confidence interval will be determined where: PPA = [TP / (TP + FN)] × 100, where TP (true positive) is positive self-test in agreement with positive Architect test, and FN (false negative) is negative self-test discordant with positive Architect test; and NPA = [TN / (TN + FP)] × 100, where TN (true negative) is negative self-test in agreement with negative Architect test, and FP (false positive) is positive self-test discordant with negative Architect test.

As a secondary analysis, study participants’ self-interpreted results were compared with the self-test results interpreted by the observer.

Results

Study participants

Due to the impact of COVID-19 on restrictions for facility visits, a total of 767 participants were recruited over the study period instead of the 900 that were targeted for enrollment. Of the 767 participants that started the process, 89 participants were excluded from the Primary Efficacy Analysis for reasons including failure to provide a venous blood sample or failure to provide a self-test result (see Appendix B for full details of excluded participants). Recruited participant numbers varied across the performance, usability and mock results interpretation arms of the study as not all recruited participants completed all elements.

Table 1 shows the demographic profile of participants included in the performance analysis. Over 75% (n = 522) of the sample was between the ages of 18 and 45 years of age. Over 60% (n = 410) identified as “male” and 37% (n = 247) as “female” with approximately 3% (n = 20) choosing “other”, including transgender, non-binary, queer, and Two Spirit gender identities. In terms of ethno-racial composition, approximately 44% (n = 296) identified as “White”, 17% (n = 113) as “Black”, 14% (n = 95) as “First Nations, Métis or Inuit”, 16% (n = 106) as “Asian”, 7% (n = 46) as “Latin American”, and the remaining 2% (n = 22) as “other”. In terms of educational achievement, approximately 27% (n = 184) of the sample completed High School, 23% (n = 156) completed College, and 48% (n = 322) a University degree or higher. Less than 5% (n = 30) indicated having any kind of reading or writing impairment. Approximately 45% (n = 307) were employed, 23% (n = 153) were unemployed, 13% (n = 87) were students, 2% (n = 16) were retired, and 17% (n = 113) preferred not to answer.

Table 1 Demographic Profile of Participants included in the Primary Efficacy Analysis (N = 678*)

In terms of self-reported HIV status, 20% (n = 137) indicated that their status was unknown or had never been tested, while 80% (n = 540) indicated that they were HIV-negative. When asked about HIV testing, 74% (n = 500) indicated having prior experience with testing. When evaluating self-reported risk category for HIV, 600 participants (89%) were considered “high risk” for HIV infection. For specific self-reported sexual behaviours in the year prior to survey, 30% (n = 185) identified as having condomless sex with men while 18% (n = 108) had condomless sex with women. Approximately 30% (n = 181) of participants indicated having multiple sexual partners in the past year, 26 individuals (4%) indicated that their sexual partner was living with HIV, and 51 participants (8%) indicated having ever injected drugs.

Primary efficacy (performance) analysis

After removing the 89 subjects who met the exclusion criteria from the 767 recruited participants, the primary efficacy analysis on the 678 who completed INSTI HIV Self-Test revealed a positive percent agreement of 100% (5/5, 95% CI: 43.6–97.0%) and a negative percent agreement of 99.5% (614/617, 95% CI: 98.6–99.8%) when comparing the valid self-tester results to the Abbott Architect (see Table 2 for full comparison of participant and laboratory test results and Table 3 for PPA and NPA calculations). Of the 6 HIV previously undiagnosed participants living with HIV that were confirmed by the Abbott Architect and Geenius comparator methods, 5 were positive with the self-test and 1 was invalid due to a low volume of fingerstick blood that was used in the INSTI self-test procedure. The overall percent agreement between participant and observer for the self-test results was 93.5% (n = 633) (see Table 4 for full comparison of participant and observer interpretations of HIV-ST results). The overall “invalid” rate for the HIV Self-Test was 5.6% (n = 38).

Table 2 Agreement between participant interpreted HIV-ST result and Laboratory Result (n = 678)

Table 3 Positive and Negative Percent Agreement with Laboratory Results (Abbott Architect) based on 678 completed HIV Self-Tests

Table 4 Agreement between self-tester interpreted result and observer interpreted result (_N = 678)

Usability assessment analysis

From the total of 767 study participants, 59 subjects withdrew after consenting, did not have a venous blood sample collected or did not complete the self-test, leaving 708 available for analysis with the usability assessment. Overall, the average for all expected outcomes was 91.8% and for the 12 critical items, the average was 92.4% (see Table 5 for full breakdown of usability questions). Despite as instructed by the kit instructions for use, only 47.2% (n = 334) of participants were observed to have washed and dried their hands. Just over 92% (n = 650) of participants were able to lance their finger correctly, and 88.7% (n = 627) were able to form a blood droplet, but there was a considerably lower number of participants (81.2%; n = 575) who were able to get the blood droplet to fall directly into Bottle 1.

Table 5 Usability Assessment (n = 708)

Following the fingerstick blood collection, there were three types of procedures carried out as part of the test process: (1) shaking bottles (accomplished with range of 88.8 to 90.2% of time), (2) pouring contents into the test device (97.3 to 99.0% of participants did so correctly), and (3) waiting for liquids to disappear before adding next liquid into device (participants did this procedure 98.0 to 98.2% of the time). Over 89% (n = 632) of participants performed the procedural steps of the self-test in the correct order, while less than 10% (n = 67) missed any steps and yet continued the process despite this, and only 1.0% (n = 7) quit the process before the test was completed.

Table 6 shows the results from the self-test questionnaire given to each participant. Regarding the instructions for use, 96.7% (n = 670) of participants found them easy to follow. In terms of device use, over 90% (n = 626) of participants indicated that they were confident with performing the test on their own, and 96.2% (n = 666) found the device easy to use. 94.7% (n = 655) of participants indicated that they would use the test again and 95.5% (n = 658) would recommend this test to a sexual partner or friend. In terms of where participants would prefer using the self-test, 58.6% (n = 399) indicated that they would do the test at home, while 41.4% (n = 282) indicated that they would prefer to do the test at a clinic.

Table 6 Self-Test Questionnaire (n = 708)

Mock test interpretation results

A total of 404 participants volunteered to complete the mock test interpretation (see Table 7 for a full breakdown of mock test results). Overall, 97.8% (n = 395) of participants correctly identified a “strong positive”, 90.6% (n = 366) a “weak positive”, 99.3% (n = 401) a “negative” result, 97.8% (n = 395) an “invalid” (with no control, no test) result, and 93.6% (n = 378) an “invalid” (with no control, with test) result.

Table 7 Mock Results Interpretation (n = 404)

Discussion

Since data from this study was intended to be submitted by the manufacturer to Health Canada for evidence of the INSTI HIV Self-Test safety and effectiveness as part of the device license application, it was important to determine if the self-test performance met published performance targets. The Health Canada Guidance for Manufacturers of Human Immunodeficiency Virus (HIV) Rapid Diagnostic Tests (RDTs) for use at the Point of Care (POC) or for Self-Testing provides performance targets that must be met for clinical sensitivity and specificity: “Evidence that the RDT intended for use at the POC or for Self-Testing has a minimum sensitivity and specificity of ≥99% for HIV antibody detection should be provided” [15]. Primary efficacy analysis with 678 participants who completed the HIV self-testing study revealed a positive percent agreement of 100% (5/5, 95% CI: 43.6–97.0%) and a negative percent agreement of 99.5% (614/617, 95% CI: 98.6–99.8%) when comparing the valid self-tester results to the Abbott Architect. The overall specificity, indicated as negative percent agreement of 99.5%, with the lower bound of the 95% CI at 98.6% meets the required performance criteria.

There was a 93.5% (n = 633) concordance in the self-test results interpretation between the untrained self-test study participants and the trained observers for the subjects in the primary efficacy analysis. Most discordant results were in the invalid or “do not know/not sure” (uncertainty) response categories.

A total of 38 participants (5.6% of sample) interpreted their self-test results as invalid, including one participant whose venous blood tested positive with the Abbott Architect. Of these, 22 were also interpreted as invalid by the observers, including the one participant living with HIV; however, 15 were interpreted as negative by the observers indicating there was a visible control dot present in the self-test which was either not observed by the self-tester, or not interpreted correctly. In some of these cases, the control dot intensity was indicated to be faint. A faint, but visible control dot may also have led to “do not know/not sure” interpretations recorded by both the self-testers and observers. Mitigations may be needed to improve the fingerstick blood collection process since usability results indicated that nearly 1 in 5 participants had difficulties in getting a free-flowing blood drop to fall into the INSTI sample diluent (Bottle 1). These mitigations could include revisions to the package insert to provide more clear instructions on the proper use of the lancet and subsequent blood drop collection, as well alerting the self-tester that even a faint dot intensity is considered valid. Consequently, self-test users should be instructed to conduct self-testing in a well-lit area.

There was an observed tendency for some self-testers to wipe their fingertip on the rim of the INSTI Bottle 1 (sample diluent) instead of allowing the blood drop to fall freely into the sample diluent. This was indicated for 60% (n = 22/37) of invalid self-test results and for several self-test results that were still valid. A further mitigation to address the critical factor for the success in getting the blood drop into Bottle 1 could be to revise the self-test package insert to indicate, with illustration, that one should not scrape or wipe the blood into Bottle 1.

The mock results interpretations for the 404 participants who volunteered to participate in this portion of the study show a high concordance with the expected results. The percent agreement with expected results ranged from 90.6% (n = 366) for the weak positive result to 99.3% (n = 401) for the negative result, indicating that untrained self-testers can successfully interpret a range of self-test results including strong positive, weak positive, negative, and invalid test results. However, mitigations to the package insert to call attention to the possibility of faint dot intensity could improve the lower success rate in interpreting weak positive results.

In general, participants expressed a high level of satisfaction with the HIV-ST experience as shown in Table 6: 96.7% (n = 670) of participants found the instructions for use easy to follow and 96.2% (n = 666) found the self-test device easy to use. 94.7% (n = 655) of participants would use the INSTI HIV Self-Test again, and 95.5% (n = 658) would recommend its use to a sexual partner or friend. Of interest and for consideration – approximately 60% (n = 399) of participants indicated a preference for using the self-test in a home environment while 40% (n = 282) would prefer to use the test at a clinic or health care facility, suggesting that both assisted (supervised) and unassisted self-testing strategies will likely be needed for “real world” access and uptake.

Overall, acceptance of the INSTI HIV Self-Test by intended users in this study was high, and similar to studies that reported that oral fluid-based HIV self-testing is highly acceptable among a variety of populations [7, 16, 17]. Although data on use of blood-based HIV self-tests remains scarce [18,19,20], performance, usability and acceptance in this study were similar to an INSTI HIV Self-Test study conducted in Kenya [21]. This suggests that populations in both settings were able to conduct and perform the blood-based test as per instructions and agreement and performance metrics were aligned. This attests to the broad use of blood based self-tests as an alternative to oral self-tests.

Conclusions

The impact of the COVID-19 pandemic and its related public health control measures on HIV treatment and prevention, including access to facility-based testing, has had profound effects on individuals, communities, and societies across the world [22]. Implementation of HIV self-testing has been shown to increase HIV testing rates in men who have sex with men [23], and we found high acceptability amongst the diverse sample of participants in the current study. So, the addition of this newly approved HIV self-test into the Canadian HIV testing landscape could have similar outcomes. An approved blood-based HIV self-test in Canada can offer an accurate, acceptable and simple alternative to facility-based HIV testing for key populations, particularly when impacted by COVID-19 pandemic restrictions. Efforts to mitigate the frequency of invalid results, such as improved instructions for collection of the fingerstick blood sample, should be considered.