Low validity of self-report in identifying recent mental health diagnosis among U.S. service members completing Pre-Deployment Health Assessment (PreDHA) and deployed to Afghanistan, 2007: a retrospective cohort study

Background Since 1998, the U.S. Armed Forces has used the mandatory Pre-Deployment Health Assessment (PreDHA) screening questionnaire as a means of assessing the health and suitability of U.S. service members for deployment. Limited data exists to quantify the validity of the self-reported PreDHA. This study was conducted to assess the validity of self-reporting in PreDHA to identify deployed service members who have had a recent mental health disorder diagnosis. Methods A retrospective cohort study was conducted on 15,195 U.S. service members deployed in support of combat and reconstruction operations in Afghanistan. The Defense Medical Surveillance System (DMSS), the DoD's longitudinal medical surveillance database, was queried to identify cases among the cohort with a recent diagnosis of a pertinent mental health disorder and to obtain those subjects' responses to the PreDHA. Results Of the study cohort, 11,179 (73.6%) subjects had a PreDHA available within the DMSS at the time of analysis. A total of 615 subjects (4.0%) had one or more mental health disorder diagnoses during the pre-deployment period. Out the 615 subjects with diagnosed mental health disorders, 465 had a PreDHA. Among these, only 224, not quite half, answered in the affirmative to the PreDHA question: "During the past year, have you sought counseling or care for your mental health?" Conclusion This study demonstrates that the self-reported PreDHA has low validity for identifying service members with diagnosed mental health disorders. The development of electronic decision-support systems which automatically screen electronic health records to identify high-risk service members may prove a valuable component of improved pre-deployment screening processes.


Background
Within the U.S. military, commanders and medical staff have a shared obligation to ensure that service members under their command or care are free of potentially dis-qualifying or significant mental health disorders that might affect their suitability for and stability during prolonged deployments. Pre-deployment screening programs that identify service members with a recent history of mental health disorder might therefore be a useful adjunct to the existing multilayered process of screening and selection that occurs during accession into the military [1] and during training and service prior to deployment [2].
The Pre-Deployment Health Assessment (PreDHA) screening questionnaire was introduced by the U.S. Department of Defense (DoD) in 1998 as a means of assessing the health and suitability of service members for deployment [3][4][5]. According to current DoD policies, PreDHAs are to be completed by all service members deploying to land-based operations for 30 days or longer [6], and are to be completed within the 60 days prior to the expected date of deployment [7]. Once completed, PreDHAs are forwarded to the Armed Forces Health Surveillance Center (AFHSC) for inclusion in the Defense Medical Surveillance System (DMSS), the DoD's longitudinal medical surveillance database [8]. The DMSS integrates information on pertinent service member demographics, inpatient and outpatient medical diagnoses at military and civilian medical facilities, and other pertinent health data [5,8], which permits robust epidemiological analyses of health conditions associated with military service.
Since the development of the PreDHA, concerns over the rising mental health consequences of the conflicts in Iraq and Afghanistan have been well described [9][10][11][12][13][14][15][16][17]. Lowered standards for acceptance of recruits into the military and repeated deployments have combined to significantly increase the prevalence of mental health disorders among U.S. military personnel, including ADHD [18], PTSD [9,14,15,17], anxiety [9], and depression [9,17,19]. As a result, the risk is increased that service members with disqualifying mental health disorders will be inappropriately deployed, and that service members will be prescribed and exposed to medications such as mefloquine (a commonly used anti-malarial), whose contraindications include a history of certain mental health disorders [20].
It has been noted that screening programs are often developed and implemented with minimal evidence as to their efficacy [21]. Between January 2003 and July 2007, over 1,705,787 PreDHA had been administered to deploying U.S. military personnel [5], but to date no published studies have been performed to quantify either the validity or utility of their administration, or to determine how the potential benefits of administering the PreDHA compare to the possible psychological, financial and opportunity costs of implementing the PreDHA screening program [4].
To assess the validity of self-report among service members completing PreDHA, and to quantify the test characteristics of selected questions on the PreDHA for identifying deployed service members with a recent mental health disorder diagnosis, a retrospective cohort study was performed on a large group of U.S. service members deployed to Afghanistan in 2007.

Data Sources, Inclusion and Exclusion Criteria
The U.S. military collects and records pertinent demographic information on service members deploying to combat zones and to other areas of the world in support of contingency operations in a database known as the Defense Theater Accountability System (DTAS). In support of this study, DTAS was queried for a listing of all U.S. military personnel deployed in support of combat and reconstruction operations in Afghanistan as members of the North Atlantic Treaty Organization (NATO) International Security and Assistance Force (ISAF) [22] as of a reference date in early 2007, and this listing was provided to staff at AFHSC for further data queries using the DMSS.
To ensure consistency of demographic data, service members who did not have demographic records available in the DMSS were excluded from analysis. To improve the completeness of data available for analysis, service members with fewer than 30 days of continuous deployment as of the reference date were also excluded. Among remaining service members, the date of the last PreDHA completed and available for analysis within the DMSS was determined, and if there was a record of a PreDHA completed after the DTAS date of deployment, the DTAS deployment date was deemed questionable and the service member was excluded from analysis. All remaining service members comprised the study cohort.

Demographic and Military Service Data
For each member of the study cohort, the DMSS was queried to determine basic demographic and military service data, including gender, number of prior deployments, and military occupational specialty (combat vs. non-combat). Age was calculated as of the reference date.

Pre-Deployment Health Assessments
To permit analysis of data from study cohort subjects who might have experienced delays in deployment, all PreD-HAs within the 90 days prior to the DTAS deployment date and available in DMSS at the time of analysis were evaluated for pertinent subject and health care provider responses.
Pertinent subject responses included those to three questions: question #7 "During the past year, have you sought counseling or care for your mental health?"; question #2 "Do you have any medical or dental problems?"; and question #8 "Do you currently have any questions or concerns about your health?". Pertinent health care provider responses included those to two questions: "referral indicated" and "final medical disposition". Responses considered pertinent included a referral for "combat/operational stress reaction", "family problems", "fatigue, malaise, multisystem complaint", "mental health", and "neurologic" problems, or a final medical disposition of "not deployable". Pages 1 and 2 of the current two-page PreDHA instrument are enclosed as

Statistical Analysis and Methods
The demographic characteristics of the study cohort and those with a pre-deployment mental health disorder diagnosis were compared by Chi-square test across demographic strata, and odds ratios (OR) and 95% confidence intervals (CI) were calculated for dichotomous strata. Numbers of subjects with recent mental health disorder diagnosis by ICD-9CM coded diagnosis category were determined. Numbers of subjects in the study cohort and with a recent mental health disorder diagnosis with available PreDHAs within 90 days prior to the DTAS deployment date were compared by Chi-square test. Among those with a recent mental health disorder diagnosis, the OR for the presence of multiple available PreDHAs within 90 days of deployment was assessed.
To evaluate the test characteristics of the PreDHA, subjects with one or more PreDHA completed in the 90 days prior to deployment were stratified by presence of recent mental health disorder diagnosis, and the presence of pertinent responses to questions from the most recent PreDHA were assessed to calculate the odds ratio (OR) and "number needed to screen" (NNS) for each. In contrast to other uses of this statistic [21], for this study the NNS was defined as the number of subjects needed to be screened with the PreDHA in the 90 days prior to deployment to obtain a positive response to one or more pertinent questions on the most recent PreDHA in the presence of a recent mental health disorder diagnosis. Additional odds ratios were assessed comparing presence of referral or non-deployable status and positive responses to subjectreported data. Data manipulation was performed using SAS 9.1 [24]. Odds ratios and 95% confidence intervals were obtained via the Cornfield approximation [25] using Stata 8.0.

Ethical Considerations
This analysis was conducted by a duly constituted public health authority to evaluate the effectiveness of military medical screening programs and policies. Data were requested from AFHSC under military command approval, with analysis performed consistent with Department of Defense Directives (DoDD) 6590.2 and DoDD 6490.02E and as non-research exempt from Institutional Review Board (IRB) requirements in accordance with United States Code of Federal Regulations (CFR) section 38 CFR 16 ("The Common Rule"). Identified health data were securely transmitted, processed and retained on data systems compliant with DoD information assurance policies.

Subject Demographics and Mental Health History
A list of 18,733 U.S. service members was provided to AFHSC, and after application of exclusion criteria, the final study cohort comprised 15,195 subjects. A total of 615 subjects (4.0%) had one or more mental health disorder diagnoses in the year prior to deployment. Females had over twice the odds of having a recent mental health disorder diagnosis (OR = 2.26, 95% CI 1.82-2.80) as males (P < 0.001, χ 2 = 58.09). Subjects with one or more prior deployments had significantly greater odds of having a recent mental health disorder diagnosis (OR = 1.44, 95% CI 1.22-1.70) as those without (P < 0.001, χ 2 = 19.18). Those in non-combat occupations had significantly greater odds of having a recent mental health disorder diagnosis (OR = 1.33, 95% CI 1.09-1.62) as those in combat occupations (P = 0.006, χ 2 = 7.69). There were no significant differences between the study cohort and those with a recent mental health disorder diagnosis by age cat-  egory (P = 0.273, χ 2 = 5.14). Demographic data is shown as Table 1.
Of the 615 deployed subjects with a recent mental health disorder diagnosis, 17 (2.3%) had one or more inpatient diagnoses, while 8 (1.3%) had one or more primary inpatient diagnoses. All 615 had one or more outpatient diagnoses, while 531 (86.3%) had one or more primary outpatient diagnoses. The number of subjects with specific diagnoses, by ICD-9CM codes and type of diagnosis are shown as Table 2.
Of the 8 deployed subjects with one or more primary inpatient diagnoses, 3 were diagnosed with major depressive disorders, 4 with adjustment disorder with depressed mood, and 1 with post-traumatic stress disorder. Of the 8, all (100%) had no prior deployments, and 5 (62.5%) were female.

Pre-Deployment Health Assessments
Of the study cohort 11,179 (73.6%) subjects had at least one PreDHA available for analysis that was completed within the 90 days prior to deployment. Of these, a total of 1,633 (14.6%) subjects completed multiple PreDHAs during this period. The odds of a recent mental health disorder diagnosis were greater among those with one or more PreDHA available as compared to those without a PreDHA (OR = 1.12, 95% CI 0.93 -1.35), although this difference was not statistically significant (P = 0.242, χ 2 = 1.37). Among subjects with at least one PreDHA available for analysis and completed within the 90 days prior to deployment, those with a recent mental health disorder diagnosis also had greater odds of completing multiple PreDHA in the 90 days prior to deployment (OR = 1.19, 95% CI 0.93 -1.52) than of completing only one, but this also did not reach statistical significance (P = 0.177, χ 2 = 1.83).
Of the 8 deployed subjects with one or more primary inpatient diagnoses, 7 had PreDHAs available, of which 6 (85.7%) answered in the affirmative to question #7, 2 (28.6%) in the affirmative to question #2, and none (0%) in the affirmative to question #8.

PreDHA Responses and Test Characteristics
Among the 11,179 with one or more PreDHA completed in the 90 days prior to deployment, 465 (4.2%) had one or more recent mental health disorder diagnoses. Of the 465, 224 (48.2%) responded in the affirmative to question #7.

Non-Deployable Final Medical Disposition
Among those who answered question #7 in the negative, those with a recent mental health disorder diagnosis were more likely to receive a final medical disposition of "not  1. Record of one or more pertinent ICD-9CM diagnosis codes in the full year prior to date of deployment.

Discussion
This is the first study to assess the sensitivity and other test characteristics of responses to PreDHA questions for identifying a recent mental health disorder diagnosis among deployed U.S. service members. This study found a prevalence of recent mental health disorder diagnosis of 4.2% among those with one or more PreDHA completed in the 90 days prior to deployment and available for analysis in DMSS. Although this study found that subjects with a recent mental health disorder diagnosis were significantly more likely to answer pertinent questions on the PreDHA in the affirmative, more than half of such subjects provided no indication of their diagnosis on their most recent PreDHA.
This study found that the sensitivity of the question "during the past year, have you sought counseling or care for your mental health?" in identifying a recent mental health disorder diagnosis was only 48.2%. Subjects with a recent mental health disorder diagnosis were over 41 times as likely to answer this question in the affirmative as those without, but this study also found that almost 50 subjects needed to be screened with PreDHA to identify one subject with a recent mental health disorder diagnosis who indicated such a response. Including affirmative responses to additional relevant questions, including those inquiring about current health concerns or problems did not appreciably increase the sensitivity of self-reported data. Furthermore, although subjects with a recent mental health disorder diagnosis were over twice as likely to endorse current medical or dental problems (OR = 2.88), or to have questions or concerns about their health as those without (OR = 2.28), only 23.4% of those with a mental health disorder diagnosis endorsed current medical or dental problems, and only 6.5% endorsed any current questions or concerns about their health.
These findings fit with those of other researchers, who have noted among deploying and redeploying U.S. service members perceptions of significant stigma associated with mental health conditions [9][10][11], and a tendency among U.S. service members towards underreporting as well as a preference towards declining treatment. A major study of U.S. service members preparing for deployment to Iraq found among those who screened positive for major depression, generalized anxiety, or PTSD, 86% acknowledged a problem, but only 40% were interested in receiving help [9]. Among a larger sample of service members screening positive, a majority indicated perceived barriers to receiving mental health care, including concerns that others in their unit would lose confidence in them, view them as being weak, or blame them for their mental health problems [9]. The expression of these concerns might be expected to increase in the context of massscreening programs such as PreDHA which are often conducted at unit-level.
Perhaps not surprisingly, therefore, this study found a very low rate of pertinent health care provider referrals. Less than one-third of 1% of all subjects with an available PreDHA (0.3%) received such a referral. Almost 860 subjects had to be screened with PreDHA to result in a single pertinent referral for a subject with a recent mental health disorder diagnosis. Of subjects receiving such a referral, approximately 6 in 10 (61.2%) had answered in the affirmative to one or more pertinent question on their most recent PreDHA, and those indicating a positive response to such questions were over nine times (OR = 9.34) as likely to be referred as those who did not. According to current guidance [7], upon completion by the service member, responses to PreDHA are to be immediately reviewed by a "medic, nurse, medical technician or corpsman", and any positive responses to (among others) questions 7, 2 and 8 "requires referral to a trained health care provider (physician, physician assistant, nurse practitioner, advanced practice nurse, independent duty corpsman, independent duty medical technician, or Special Forces medical sergeant)" [7]. Existing DoD guidance is not clear as to whether this referral requires appropriate additional annotation by the trained health care provider on the PreDHA, or merely requires the PreDHA to be reviewed and certified by such a provider.
Of note in this analysis, subjects who had a recent mental health disorder diagnosis but denied having such a history were not statistically more likely to be identified by the provider as "not deployable" than those without such a history, which suggests that these subjects might have been able to successfully conceal or minimize a potentially disqualifying condition. Whether the 21 "not deployable" subjects with a mental health disorder diagnosis were appropriately deployed in accordance with current DoD guidance and regulations is not clear by this study's methodology. Existing DoD instructions [7] describe only the manner in which PreDHAs are to be completed, but do not provide guidance as to what patterns of subjectprovided responses constitutes a disqualification for deployment; nor do they formalize a mechanism for ensuring that service members flagged by the provider as "not deployable", in fact, are not deployed. A recent policy memorandum drafted in response to Congressional concerns [26]"provides guidance on deployment... for military personnel who experience psychiatric disorders" and states that "diagnosed conditions that are not amenable or anticipated not amenable to treatment and restoration to full functioning within one year of treatment should generally be considered unfitting...." The memorandum specifically states that "psychotic and bipolar disorders are considered disqualifying for deployment". This study found evidence of 5 subjects with diagnoses of psychoses and 14 subjects with diagnoses of bipolar and manic disorder. Whether these subjects were appropriately deployed is also not clear by this study's methodology.
This study has a number of important limitations that require the results to be interpreted in context. Most significantly, this study included only service members who were identified on official rosters as being deployed; not the larger cohort of service members who were administered PreDHA in anticipation of deployment. Limitations in data available within the DMSS precluded accurately identifying this cohort; future studies using alternate methodologies might assess the number of service members who may have been identified as "not deployable" and who did not subsequently deploy.
This study, as with previous research on deploymentrelated mental health issues [12,27], relied upon medical surveillance data in the DMSS to identify the presence of pertinent mental health disorder diagnoses. The strengths and limitations of this data have been previously described [12], although research has demonstrated good correlation between ICD-9CM coded mental health disorder diagnoses and evidence of psychoactive pharmacotherapy [28], confirming the utility of surveillance databases in identifying functional psychiatric morbidity.
This study examined only records of ICD-9CM diagnostic codes corresponding to pertinent mental health disorders, and did not assess ICD-9CM "V-coded" visits for counseling not associated with a formal diagnosis [29], nor visits to social workers, chaplains, and other counselors practicing outside of administrative systems whose data is captured by the DMSS. Independent of these limitations, this study demonstrates that the subject-reported data collected during the PreDHA process is of questionable validity. Existing policy [26] states that "it is the responsibility of the Service member to report past or current physical or mental health conditions or concerns and associated treatments". The results of this study provide strong evidence that relying on selfreport alone may be insufficient policy for screening for disqualifying or significant mental health conditions. These results support the recent conclusions of the Department of Defense Task Force on Mental Health, a special body established at the direction of Congress to "examine matters relating to mental health and the Armed Forces". In the Task Force's final report, it was noted that its members "were told on multiple site visits that the validity of the Pre-Deployment Health Assessment suffers because service members underreport their mental health concerns..." [31] In the policy memorandum announcing the introduction of the PreDHA in 1998 [3], it was noted that the objective of the assessments was merely to provide "quick confirmation and documentation of a service member's health readiness for deployment or redeployment and to determine if there is a need for a clinician's evaluation before deployment or redeployment. Future revisions of deployment health assessments shall require pilot testing and question validation before being put into use" [3]. Furthermore, this memorandum stated that "deployment-related mental health screening will be addressed in a separate policy memorandum". Since this original announcement released over ten years ago, no such policy memorandum has been issued, and no formal validation of the PreDHA has been published or undertaken.

Conclusion
This study demonstrates that self-report by PreDHA has low validity in identifying members of the U.S. military with a recent mental health disorder diagnosis. In light of this study's findings and in consonance with prior recommendations to improve the quality of mental health care provided to U.S. military service members, consideration should be given to complementing the PreDHA with a validated mental health screening instrument of higher sensitivity [32]. The development of electronic decisionsupport systems which automatically screen electronic health records, including ICD-9CM diagnosis and pharmacy data, to identify high-risk service members may prove a valuable component of improved pre-deployment processing [33]. Such systems have been proposed previously as a means of improving the safety of prescribing and dispensing of mefloquine, which shares with predeployment screening the requirement to accurately identify a history of recent mental health diagnosis [20].