Introduction

Osteoporosis is a disease characterized by low bone mass and micro-architectural deterioration of bone tissue leading to enhanced bone fragility and a consequent increase in fracture risk [1]. The clinical diagnosis is based on bone densitometry and a bone mineral density (BMD) measurement of 2.5 standard deviations (SD) or more below the mean for a young adult population [1]. However, as it is not feasible to expose all participants to BMD measurement, case definitions of osteoporosis in large scale epidemiological studies and national health surveys often rely on self-report of diagnosis, medication or fractures (e.g., [2–4]).

Sparse evidence suggests moderate agreement between self-report of doctor diagnosed osteoporosis and a dual-energy X-ray absorptiometry (DXA) documented osteoporosis (kappa = 0.43, 95% confidence interval [CI] = 0.33–0.53) [5], or a physician’s interpretation of medical record information and physical examination (kappa = 0.33, 95% CI = 0.27–0.39) [6]. Both studies were done among adults aged 65 and over. As both under-reporting and over-reporting of chronic conditions have been associated with increased age [6, 7], results may be different for mid-age adults. Although there is some evidence that the validity of self-reported diagnosis is poorer for incident than for prevalent diseases [8], validity of self-reported incidence of osteoporosis has not been examined before.

The aim of this study was to assess the concurrent and construct validity of self-report of prevalent and incident osteoporosis in mid-age and older women. Self-reported diagnosis was compared to information about medication use available from self-report as well as the Australian pharmaceutical database for prescription reimbursement claims (Pharmaceutical Benefits Scheme [PBS]) [9]. The main indications for reimbursement of osteoporosis medications include a fragility fracture in people above the age of 70, and BMD T-scores ≤ −3 measured at two sites, resulting in a high likelihood of prevalent BMD-defined osteoporosis in participants with claims. Medication data were used as a reference standard rather than medical records or physical examination because of the lack of a centralized medical records system in Australia and the remoteness of many of the participants.

Methods

Study sample

The Australian Longitudinal Study on Women’s Health (ALSWH) is an ongoing study of factors affecting the health and well-being of three population-based cohorts of women born in 1973–1978, 1946–1951, and 1921–1926. Recruitment, data collection procedures and attrition have been described in more detail elsewhere [10]. In short, women were selected randomly from the national Medicare health insurance database which includes all citizens and permanent residents of Australia, with oversampling of women living in rural and remote areas to capture the heterogeneity in health experiences of women living outside metropolitan areas [10]. Baseline characteristics indicated that the sample was reasonably representative of the general population of Australian women, although there was overrepresentation of Australia-born and university educated women [10]. Ethical clearance was obtained from the Universities of Newcastle and Queensland and all participants gave signed informed consent. Since 1996, mailed surveys have been sent out on a rolling basis generally at 3-year intervals. For the current analyses, data were used from surveys 5 and 6 (conducted in 2007 and 2010 with response rates of 83.8% and 81.6%, respectively) for the mid-age cohort and surveys 4 and 5 (conducted in 2005 and 2008, with response rates of 83.9% and 79.4%, respectively) for the older cohort, as these surveys included self-report of medication. To validate self-reported prevalent osteoporosis, data were included from the 10,509 mid-age and 7,072 older women who provided self-reported information on osteoporosis and medication at surveys 5 and 4, respectively. To validate self-reported incident osteoporosis, data were included from the 7,118 mid-age women and 2,516 older women who additionally provided self-reported information on osteoporosis and medication at surveys 6 and 5, respectively, and who did not report having osteoporosis at the previous survey.

Prevalent and incident osteoporosis

Self-reported osteoporosis

Participants were classified as having prevalent osteoporosis if they answered “yes” to the question “In the past three years, have you been diagnosed or treated for osteoporosis?” at survey 5 for the mid-age women and survey 4 for the older women. Participants without self-reported osteoporosis at survey 5 (mid-age) or 4 (older) and who answered “yes” to the question at the subsequent survey three years later, were classified as having incident osteoporosis.

Self-reported osteoporosis medication

In surveys 5 and 6 for the mid-age women and survey 4 for the older women, participants were asked to copy the names from the packets of all their medications, vitamins, supplements or herbal therapies taken in the past 4 weeks. Participants were classified as having prevalent osteoporosis if, they reported at least one of the osteoporosis medications listed in Appendix 1 in survey 5 for the mid-age women and survey 4 for the older women. Use of Vitamin D however, was not included in this definition as supplementation without additional use of other anti-osteoporosis medication is used mainly for prevention and is thus less indicative of osteoporosis diagnosis [11]. Participants from the mid-age cohort were classified as having incident osteoporosis if they did not report medication for osteoporosis at survey 5, but reported at least one of the osteoporosis medications listed in Appendix 1 at survey 6. As self-report of medication use was not included in any of the subsequent surveys for the older women, self-reported incident osteoporosis medication could not be determined for this cohort.

Pharmaceutical benefits scheme recorded osteoporosis medication

PBS is part of the Australian government’s broader National Medicines Policy which includes subsidizing the costs of medication [9]. PBS records all the prescription medicines that were dispensed with Government subsidy. The Scheme is available to all Australian citizens and permanent residents. Appendix 1 provides a summary of the osteoporosis medications and the indications for which they are subsidized by the PBS. PBS records were linked to ALSWH survey data, but only for those participants who consented to linkage (63% and 66% of the mid-age and older women, respectively). Participants were classified as having prevalent osteoporosis based on PBS data if, in the 3 years preceding the return date of the relevant survey (survey 5 for mid-age women, survey 4 for older women), at least one claim for osteoporosis medication was recorded in PBS. Participants were classified as having incident osteoporosis if they had not received an osteoporosis medication recorded on the PBS database during the 3 years preceding survey 5 for mid-age women and survey 4 for the older women, but did have at least one claim for osteoporosis medication in the PBS database in the 3 years preceding survey 6 for the mid-age women and survey 5 for the older women.

Osteoporosis-related characteristics

The following characteristics were measured using the same methods in all surveys: bodily pain and physical functioning were measured as sub scales from the Short Form Health Survey 36 (SF-36) [12, 13]. Scores ranged from 0 to 100, with higher scores indicating less pain and better physical functioning. Fractures were assessed from responses to the question “In the last 12 months, have you broken or fractured any bones?” (yes/no). Back pain was assessed with the question “In the last 12 months, have you had back pain: never, rarely, sometimes, or often?” The response categories were collapsed into never/rarely vs. sometimes/often. Body weight was preferred over body mass index (BMI) as a risk factor for osteoporosis, because BMI may not be a reliable indicator of body composition in women with osteoporosis who lose height as a result of vertebral deformities and fractures [14]. Furthermore, low BMI is believed to be a risk factor for osteoporosis due to reduced loading on the bone, which suggests that weight is important rather than the weight to height ratio. Body weight was measured with the question “How much do you weigh without clothes or shoes?” (kilograms).

Confounders and effect modifiers

The associations between self-report of osteoporosis and osteoporosis-related characteristics could be confounded by age, level of education, height, chronic conditions, and depressive symptoms as these variables have all been found to be associated with either accuracy of self-report [7, 15] or osteoporosis [16–19], and the osteoporosis-related characteristics [16, 19–23]. As memory problems may affect self-report of both osteoporosis diagnosis and medication, the validity of self-reported diagnosis of osteoporosis may be different in older women with and without memory problems [15]. Memory problems are therefore included as a potential effect modifier. All variables were measured using the same methods across surveys unless stated otherwise. Height was based on self-report and recorded in centimetres. Education was assessed as the highest qualification completed, ranging from “no formal qualification” to “university degree or higher”. Chronic conditions were assessed by summing the number of reported conditions from the list: diabetes, arthritis, heart disease, stroke, lung disease and cancer (range 0–6). In the mid-age cohort, depressive symptoms were assessed using the ten-item Center for Epidemiologic Studies Depression Scale (CES-D); scores range from 0 to 30, with higher scores indicating more depressive symptoms [24, 25]. In the older cohort, the Goldberg Anxiety and Depression Scale (GAD) was used [26]; scores range from 0 to 18 with higher scores indicating more anxiety and depressive symptoms. Memory problems were assessed in the older cohort only with the question “In the last 12 months, have you had problems with poor memory: never, rarely, sometimes or often?” The categories were collapsed into never/rarely vs. sometimes/often. In the older cohort, participants were asked to indicate their housing. Response options were collapsed into community-dwelling (i.e., house, flat/unit/apartment, mobile/caravan, and retirement village) and institutionalized (i.e., nursing home, hostel, and other).

Statistical analysis

Sample characteristics were given for both cohorts as means and standard deviations for approximately normally distributed continuous variables, as medians and interquartile ranges for not normally distributed continuous variables, and as percentages for categorical variables. The characteristics of women who consented to data linkage were compared to those who did not. Concurrent validity of self-reported prevalent and incident osteoporosis was examined by comparing self-reported diagnosis against self-reported medication and PBS medication and calculating percentage agreement, bias corrected kappa for chance adjusted agreement [27], sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). To examine the effect of poor memory in the older cohort, the analyses were conducted for the total sample was well as for women with and without memory problems separately. Kappa values were interpreted as: poor if <0.40, moderate if 0.40–0.60, substantial/good if 0.60–0.80, and almost perfect if >0.80 [28]. Construct validity of self-reported prevalent and incident diagnosis was examined by testing associations with known osteoporosis-related characteristics. The following hypotheses were tested: women with osteoporosis have (1) higher fracture risks [29], (2) scores indicating more bodily pain [30], (3) lower weight [29, 31, 32], (4) lower scores for physical functioning [33], and (5) more back pain [34] than women without osteoporosis. Logistic regression was used for dichotomous characteristics (fractures and back pain), whereas linear regression was used for continuous characteristics (weight, bodily pain and physical functioning). Cross-sectional models were run for self-reported prevalent diagnosis. For incident diagnosis, associations were examined between self-report and change in bodily pain and physical function, and incident back pain between the first and follow-up surveys. As weight is a risk factor of osteoporosis rather than an outcome [35], associations with this characteristic were assessed with weight (dichotomized using <59 kg as the cut-off [35, 36]) as the predictor and self-reported diagnosis as the outcome. All associations were adjusted for age, education, chronic conditions, and depression; the association with weight was additionally adjusted for height. Construct validity was considered acceptable if at least four out of five of the hypotheses were not rejected [37]. All analyses were done for the two cohorts separately, using STATA 11.1 (StataCorp LP, College Station, TX, USA). The p values were based on two-sided tests and were considered statistically significant at p < 0.05.

Results

The 10,509 and 7,072 women in the mid-age and older cohorts were on average 58.5 (SD 1.5, range 53.6–61.9) and 81.2 (SD 1.4, range 77.3–85.5) years old in 2007 and 2005, respectively (Table 1). In both cohorts, the majority of the women lived in rural areas and had no or one chronic conditions. The proportions of women with prevalent osteoporosis defined from self-reported diagnosis, self-reported medication and PBS medication were 6.0%, 2.3%, and 3.5% in mid-age women, and 22.2%, 16.7%, and 26.0% in older women, respectively. The proportions of women with incident osteoporosis over the 3 years of follow-up defined from self-reported diagnosis, self-reported medication and PBS medication were 3.8%, 2.2%, and 1.8% in mid-age women, and 13.4%, (n/a) and 11.5% in older women, respectively.

Table 1 Characteristics for the total sample, and for women who did and did not consent to data linkage separately
Full size table

The prevalent osteoporosis definition based on PBS medication could be derived for 6,632 mid-age and 4,668 older women who consented to data linkage (Table 1). Among mid-age women, those who consented to data linkage were less likely to live in remote areas (p < 0.001), had higher education levels (p < 0.001), and had fewer chronic conditions (p = 0.003) and depressive symptoms (p < 0.001) than those who did not. Among older women, those who consented to data linkage were slightly younger (p < 0.001), achieved higher education levels (p < 0.001) and had fewer co-morbidities (p = 0.007) than those who did not. No statistically significant differences were found in proportions of self-reported diagnosis between those who consented to data linkage and those who did not.

Concurrent validity of self-reported diagnosis

Prevalent osteoporosis

Table 2 shows the cross-tabulations for self-reported prevalent diagnosis against the self-reported medication and PBS medication definitions for both cohorts. In mid-age women, high percentage agreement was found between self-reported diagnosis and both medication-based definitions. The chance-adjusted agreement was moderate when self-reported diagnosis was compared with self-reported medication (kappa = 0.41, 95% CI = 0.37–0.45), but higher when compared with PBS medication (kappa = 0.51, 95% CI = 0.46–0.56). Sensitivity, specificity, and NPV were good, whereas the PPV was only poor.

Table 2 Concurrent validity: self-reported prevalent osteoporosis compared with osteoporosis definitions based on self-reported medication and Pharmaceutical Benefits Scheme (PBS) prescription reimbursement claims
Full size table

In older women, high percentage agreement was found between self-reported diagnosis and both medication-based definitions. The chance-adjusted agreement was moderate when self-reported diagnosis was compared with self-reported medication (kappa = 0.57, 95% CI = 0.55–0.59), but higher when compared with PBS medication (kappa = 0.65, 95% CI = 0.63–0.68). Sensitivity and PPV were moderate to good, whereas specificity and NPV were good. When sensitivity analysis were done for women with and without memory problems, the results were similar to those of the total sample with slightly better values in women who reported memory problems (self-reported diagnosis against PBS medication: kappa = 0.68, 95% CI = 0.65–0.71, sensitivity = 71.8, specificity = 93.9, PPV = 81.6, NPV = 89.7) than in women without memory problems (kappa = 0.62, 95% CI = 0.58–0.66, sensitivity = 65.5, specificity = 93.3, PPV = 76.6, NPV = 89.0).

Incident osteoporosis

Table 3 shows the cross-tabulations for self-reported incident diagnosis against the self-reported medication and PBS medication definitions of osteoporosis for both cohorts. In the mid-age cohort, chance-adjusted agreement was poor when self-report was compared with self-reported medication (kappa = 0.35, 95% CI = 0.30–0.45) and PBS medication (kappa = 0.39, 95% CI = 0.32–0.47). As for self-reported prevalent osteoporosis, sensitivity, specificity, and NPV were good, but the PPV was only poor.

Table 3 Concurrent validity: self-reported incident osteoporosis compared with self-reported medication and Pharmaceutical Benefits Scheme (PBS) subsidised OP medication
Full size table

In the older cohort, percentage agreement between self-reported incident diagnosis and PBS medication was high, while the change-adjusted agreement was moderate (kappa = 0.55, 95% CI = 0.51–0.61). Sensitivity and PPV were moderate, but specificity and NPV were good.

Construct validity of self-reported diagnosis

Prevalent osteoporosis

In both cohorts, self-reported prevalent diagnosis was statistically significantly associated with all five characteristics (Table 4). Women with self-reported osteoporosis had higher risks of fracture and back pain, and lower pain and physical functioning related quality of life than women without self-reported osteoporosis. In addition, women with body weight <59 kg had higher risks of reporting osteoporosis, compared with those with body weight ≥59 kg.

Table 4 Construct validity: associations between self-reported incident osteoporosis (OP) and osteoporosis-related characteristics
Full size table

Incident osteoporosis

Self-reported incident diagnosis was significantly associated with three out of five characteristics in the mid-age women and four out of five characteristics in the older women. Women with self-reported osteoporosis had higher fracture risks, higher risk of back pain (in older women only), and greater loss of physical functioning related quality of life than women without self-reported osteoporosis. In addition, women with body weight <59 kg had higher risks of reporting osteoporosis, compared with those with body weight ≥59 kg.

Discussion

In this study, we examined the validity of self-reported osteoporosis in both mid-age and older women against self-reported medication and medication information from the Australian pharmaceutical database for prescription reimbursement claims. In both cohorts, the chance-adjusted agreement of self-reported prevalent diagnosis with the two medication-based definitions was moderate to good. Sensitivity, specificity and NPV were good, but PPV varied from poor to good depending on the cohort and reference definition used. Sensitivity, PPV and chance-adjusted agreement for self-reported incident diagnosis were somewhat lower than for self-reported prevalent diagnosis. Construct validity of self-reported prevalent diagnoses was acceptable in both the mid-age and older women, whereas construct validity of self-reported incident diagnosis was acceptable in the older women only.

Concurrent validity of self-reported diagnosis

The main limitation of this study is lack of the true gold standard for the diagnosis of osteoporosis, bone densitometry. However, given the population-based design of the study and the fact that over half the sample lived in rural and remote areas where availability of DXA scans is scarce [38], exposing all women to a DXA scan is not feasible nor a cost-effective use of resources. With the linkage of our ALSWH data to the national pharmaceutical database, and the strict indications for government subsidy for osteoporosis medication (see Appendix 1), we were, in part, able to circumvent this limitation. The main indications for PBS subsidy for osteoporosis medications are a fragility fracture in people above the age of 70, and BMD T-scores ≤ −3 measured at two sites [9]. Subsidised medication would not be available to women who did not meet these criteria. It is therefore highly likely that women with claims for osteoporosis medication prescription, subsidised by the PBS, indeed had been diagnosed with osteoporosis by a physician based on their BMD, particularly if the women were younger than 70. It is well known however, that osteoporosis is an underdiagnosed and undertreated condition [34, 38, 39]. The current reference standards were based on medication use and thus could not identify women with osteoporosis who did not use anti-osteoporosis medication. This may have resulted in underestimation of the sensitivity and PPV.

When self-reported prevalent diagnosis was compared with either the self-reported medication or the PBS medication definitions, the sensitivity, specificity, and NPV were generally good in both cohorts. The low sensitivity in the mid-age women may be explained by the strict indications for OP medications in PBS. Women under 70 with BMD T-scores between −2.5 and −3 have osteoporosis according to clinical guidelines, but do not meet indications for PBS subsidy benefits and may thus be misclassified in the PBS definition (false negatives). In the self-reported medication definition, preventive use of osteoporosis medication potentially may have resulted in misclassification in the self-reported medication definition (false positives) and subsequently overestimation of the sensitivity. Sensitivity was indeed higher when self-reported diagnosis was compared against self-reported medication than when compared against PBS medication. In sum, the concurrent validity of self-reported diagnosis generally is good, but in mid-age women, sensitivity may be somewhat overestimated when compared with the self-reported medication definition and underestimated when compared with the PBS medication definition.

The chance adjusted agreement (kappa) was higher when self-reported diagnosis was compared with PBS medication than with self-reported medication. This may reflect the fact that the PBS medication definition is highly specific (i.e., participants with medication claims are highly likely to have been diagnosed with osteoporosis) whereas the self-reported medication definition is more inclusive (e.g., includes women in the osteopenic range). Values for agreement of self-reported diagnosis with medication definitions were higher in the current study than in other studies that compared self-report with information from medical records [5, 6]. The only study that compared self-report with DXA results found a kappa of 0.43 (95% CI = 0.33–0.53) in a sample of 332 women aged 65–90 years, of whom 32% reported having osteoporosis [5]. The participants were selected from a larger sample based on the availability of DXA results in their medical records. Although other diagnostic values were not reported, information was available to calculate sensitivity (62%), specificity (76%), PPV (75%) and NPV (80%). Differences in age range and osteoporosis prevalence may explain the slightly lower values compared with the results in our older cohort. In addition, the higher diagnostic values in our study may result from the fact that the PBS definition captures the more severe cases, who are more likely to be aware of the diagnosis.

In contrast with our expectations, we found slightly better diagnostic values in older women with memory problems than in those without memory problems. Similar results have been reported in a study that validated self-reported stroke [15]. A potential explanation could be that women with memory problems more often received help from a relative or caregiver with the completion of the survey (help received by women with [11%] vs. without [9%] memory problems; p < 0.001). Although statistically significant, differences in diagnostic values were only small (≤6 percentage points), suggesting that memory problems minimally influenced the concurrent validity of self-reported osteoporosis in older women.

Comparison of the proportions of women with osteoporosis according to the three definitions in our study with prevalence rates reported the literature shows that proportions for self-reported diagnosis were in the same range as those reported in the 2007–2008 Australian National Health Survey (2.5% in women aged 55–64 years) [2] and the Australian Bone Care Study (25% in 60 + -year-old women) [34]. However, proportions in our study were about half of the BMD measured prevalence rates in the Geelong Osteoporosis Study (2001–2006): 8.9% in 55- to 59-year-old women, and 51.0% in 80 + -year-old women [40]. The great differences between self-reported and BMD measured prevalence rates are likely to be explained by underdiagnosis of osteoporosis, particularly in more remote areas of Australia [38].

In line with results published by Oksanen et al (2011) [8], we found that sensitivity, PPV and chance-adjusted agreement were lower for self-reported incident than prevalent diagnosis. The agreement was only poor in mid-age women and moderate in older women. Oksanen et al. argue that low sensitivity of self-report potentially leads to underestimation and overestimation of associations between risk factors and incident health conditions. Therefore, results based on self-reported incidence of diagnosis should be interpreted with care.

Construct validity of self-reported diagnosis

Self-reported prevalent diagnosis was associated with all five characteristics examined in both cohorts, suggesting acceptable construct validity. Construct validity is further supported by the finding that the effect sizes reported in this study were similar to those reported in the literature. The odds ratios for a fracture in women with self-reported osteoporosis relative to those without (mid-age: OR = 3.67, older: OR = 2.80) were comparable to the relative risks for a fracture reported for DXA-determined osteoporosis in the women of the Geelong Osteoporosis Study (RR = 3.72, 95% CI = 2.0–6.9) [41]. In the current study, the odds for reporting osteoporosis in women with low body weight was 2.34 (95% CI = 1.92–2.89) in mid-age women and 1.51 (95% CI = 1.32–1.73) in older women compared with normal to high body weight women. These odds ratios were somewhat higher and lower, respectively, than reported in the Study of Osteoporotic Fractures (SOF) including 7,782 women aged 65 years and over (odds ratio [OR] = 2.0, 95% CI = 1.5–2.7) [36]. These differences in odds ratios are likely to be explained by differences in age range of the three cohorts.

Women with self-reported osteoporosis had lower pain and physical functioning related quality of life than women without osteoporosis. Studies that compared women with and without BMD-measured osteoporosis found no significant differences in these outcomes [42, 43]. However, these studies compared normal, osteopenic and osteoporotic women, and while scores for osteoporotic women did seem worse, differences between normal and osteopenic women were minimal. This in addition to the small number of women with osteoporosis may have caused lack of statistically significant overall group differences. Alternatively, pain and physical functioning seem to be associated with fractures rather than osteoporosis per se [30]. This is particularly true for back pain and vertebral fractures [43].

Self-reported incident diagnosis was associated with three of the five characteristics in mid-age women and four of the five outcomes in older women. However, for all characteristics the effects were in the expected direction. Lack of statistically significant differences in the pain outcomes may have been caused by the low number of incident cases and consequently wide confidence intervals. Overall, the results suggest acceptable construct validity of self-reported incident osteoporosis in both the mid-age and older women.

A strength of the current study is the large sample size. Even though data from only 63% and 66% of the mid-age and older women could be included for the comparison with PBS medication data, our sample sizes were much greater than those reported in similar validation studies [5, 6]. Comparison of women who did and those who did not consent to data linkage showed that consenters were younger, higher educated and less often lived in remote areas, but such differences were only small and would be expected to minimally alter the results. Importantly, no differences were found in proportions of self-reported diagnosis between those who consented to data linkage and those who did not. The large population-based samples, and small differences in characteristics of women who were excluded from the current analyses, support the generalizability of the results.

In conclusion, the concurrent validity of self-reported prevalent osteoporosis compared with medication data from self-report and the Australian pharmaceutical database for prescription reimbursement claims was moderate to good in mid-age and older women. The concurrent validity of self-reported incident osteoporosis was only poor in mid-age women and moderate in older women. Construct validity was acceptable for self-reported prevalent but not for incident osteoporosis.