Validation of the advance care planning engagement survey in Singapore

Abstract

Background

Singapore has an ageing population. End-of-life care and advance care planning are becoming increasingly important. To assess advance care planning engagement, valid tools are required. The primary objective of the study is to validate the 15-, 9- and 4-item versions of the ACP Engagement Survey in Singapore.

Methods

10 inpatients in a Singapore community hospital were purposively sampled for a cognitive debriefing interview on the ACP Engagement Survey. We recruited patients 21 years and older, who were able to understand and speak English, without a diagnosis of dementia, and who were not admitted under the palliative care service. Next, 150 inpatients and caregivers were recruited using convenience sampling across 2 Singapore community hospitals to complete the 15-item ACP Engagement Survey. We assessed content validity, internal consistency with Cronbach’s alpha, construct validity with hypotheses testing and test-retest reliability using intraclass correlation coefficients.

Results

The ACPES scores were significantly higher for those who reported yes for pre-planning activities such as making a will, making a lasting power of attorney, telling one’s doctor about end-of-life care preferences, and telling family or loved ones about end-of-life care preferences. Cronbach’s alpha was 0.945 for the 15-item version, 0.915 for the 9-item version, and 0.912 for the 4-item version. Intraclass correlation coefficient was 0.933 for the 15-item version, 0.938 for the 9-item version and 0.865 for the 4-item version.

Conclusions

This study provided good psychometric support for the validity of the 15-item, 9-item and 4-item versions of the ACP Engagement Survey in Singapore.

Trial registration

SingHealth Centralised Institutional Review Board (CIRB) approved this study (reference 2022/2025).

Background

Singapore, like many developed nations, is facing an ageing population. The number of seniors aged 65 years or older grew from 338,000 in 2010 to 614,000 in 2020. As of 2020, seniors formed 15.2% of the resident population, an increase from 9.0% in 2010 [1]. As the population ages, there is a growing need for healthcare services, including end-of-life care.

Advance care planning (ACP) is a process that supports adults at any age or stage of health in understanding and sharing their personal values, life goals, and preferences regarding future medical care [2]. The goal of ACP is to help ensure that people receive medical care that is consistent with their values, goals and preferences during serious and chronic illness. These discussions usually involve patients, their family and healthcare professionals. ACP can improve end-of-life experiences for patients as their wishes are more likely to be known and followed [3, 4]. Family members of patients who had done ACP before death had fewer symptoms of post-traumatic stress, depression, and anxiety, and were also more likely to be satisfied with the quality of the patient’s death [3].

In Singapore, ACP uptake remains low, with 5,100 ACPs completed between 2011 and 2015 [5]. A 2017 cross-sectional survey in community-dwelling individuals in Singapore found that out of 406 respondents, only 7 people had completed an ACP discussion [6]. In a study in a geriatric medicine department of a Singapore hospital, out of 311 eligible patients offered ACP, 195 (62.7%) refused or did not proceed with ACP [7]. As a consequence, there was less concordance among patient’s wishes and actual end-of-life care. Family members were more likely to opt for CPR, intubation, nasogastric tube feeding and antibiotic use on the patient’s behalf than the patients themselves [8]. 77% of people preferred to die at home, but only 24% achieved this [9].

In a qualitative study exploring ACP experiences in Singapore, respondents who were approached for ACP when they were not ready described feeling ‘angry’ and ‘upset’ [10]. Assessing one’s ACP engagement may be useful in identifying patients who are ready to discuss ACP, and may lead to increased ACP uptake.

To evaluate ACP behaviours and engagement, valid measurement instruments are required. To our knowledge, there are few scales that measure ACP engagement. The Readiness for End-of-Life Conversations Scale was developed for use in adult cancer patients [11]. The Decisional Balance Scale based on the Transtheoretical Model was used to assess stage of change specifically regarding signing an advance directive [12]. The ‘Stages of Change for the Component Behaviours of Advance Care Planning’ scale assesses stages of behaviour change [13]. We chose the ACP Engagement Survey (ACPES) [14] as it has been validated and used internationally, and as it also has the potential to measure change in ACP behaviours in response to interventions.

The ACPES is a tool used to measure an individual’s engagement in ACP. It was developed by researchers at the University of California, San Francisco and is designed to be used in a clinical setting to help healthcare providers assess an individual’s level of engagement in ACP discussions. It is an 82-item survey for which shorter versions (55, 34, 15, 9, 4 items) have also been validated [15].

The ACPES is based on Social Cognitive Theory and Behaviour Change Theory [14]. These theories posit that behaviour change requires factors such as knowledge, contemplation, self-efficacy, and readiness. Based on these factors, individuals then proceed through the stages of change including pre-contemplation, contemplation, preparation, action, and maintenance. Using the stages of change model, we can assess if interventions are moving people along the behavior change pathway towards action. By determining deficits in behaviour change factors, interventions could be tailored to facilitate completion of each ACP behaviour.

The conceptual framework of the ACPES was developed by the authors of the original study based on focus groups with patients and surrogate decision makers, and input from content experts. It includes 4 domains: Decision Makers: identifying a surrogate decision maker (SDM), Quality of Life (QOL): discussing goals and values with clinicians and SDM, Flexibility: deciding how much flexibility to grant SDM in making decisions, and Asking Questions: asking clinicians questions to make informed medical decisions.

The ACPES has been validated in the United States in English and Spanish, and translated and validated in the Netherlands [16], Japan [17], Taiwan [18], and China [19]. The original 82-item ACPES was conducted in adults aged 55 years old and above, and comprised of a mix of inpatients, outpatients and nursing home residents. Cronbach’s alpha was 0.94, intraclass correlation coefficient (ICC) was 0.7 [14]. The Dutch study translated and validated the 34-item ACPES in community-dwelling adults aged 18–93 years old; Cronbach’s alpha was 0.97, ICC was 0.88, with good construct validity with > 75% of hypotheses supported [16]. The Japanese study translated and validated the 15, 9 and 4-item versions in adults with chronic disease aged 65 years and above; Cronbach’s alphas were 0.94 (15-item version), 0.91 (9-item version), and 0.95 (4-item version), and ICCs were 0.88 (15-item version), 0.9 (9-item version), and 0.84 (4-item version) [18]. The Taiwan study translated and validated the 4-item ACPES in community-dwelling adults aged 20 and above; Cronbach’s alpha was 0.97, ICC was 0.86 [17]. The China study translated and validated the 34-item ACPES in community-dwelling adults with chronic disease; Cronbach’s alpha was 0.82 [19].

The ACPES has been found to be a reliable and valid tool for measuring ACP engagement, and is sensitive to change in response to ACP interventions, such as the Canadian Speak Up Campaign, and the web-based programs ‘PREPARE’, ‘Making Your Wishes Known’, and ‘MyDirectives’ from the USA [20]. Currently, there are no validated tools to assess ACP engagement in Singapore.

The primary objective of the study is to validate the 15-, 9- and 4-item versions of the ACPES in Singapore. The original 82-item survey has a mean administration time of 49 min, which is unlikely to be feasible in clinical practice [21]. Shorter versions of the survey have been validated and are also responsive to interventions [15].

Methods

We applied international validation guidelines, namely the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative [22, 23]. ACP behaviour change in the 15-item survey is measured with two subscales: self-efficacy and readiness for ACP. Response options range from 1 = not at all to 5 = extremely for the self-efficacy subscale. The response options to the readiness subscale range from 1 = I have never thought about it to 5 = I have already done it. The response options “not sure” and “refuse to answer” are also available to the participants. The ACP Engagement score is the mean score of all responses in the survey. The higher the score, the more engaged the participant is in the ACP process. The response options “I don’t know” or “refused” is coded as missing and the scale mean was computed ignoring the missing data. We chose the 15-item version as it assesses ACP behaviours in the realms of self-efficacy and readiness, and is the shortest survey to cover all 4 domains of SDM, QOL, Flexibility and Asking Questions.

Ethics

SingHealth Centralised Institutional Review Board (CIRB) approved this study (reference 2022/2025). All methods were performed in accordance with the relevant guidelines and regulations. In the first phase, written informed consent was obtained from participants whose interviews were audio-recorded. In the second phase, verbal consent was taken.

Data collection

We conducted a two-phase study. The first phase involved cognitive interviews to assess content validity of the ACPES. For the first phase, we conducted cognitive interviews by asking participants about the comprehensiveness, comprehensibility, ambiguity, and relevance of the 15-item ACPES. We purposively sampled 10 inpatients in Outram Community Hospital. Community hospitals in Singapore look after patients who require a period of continuation of care after discharge from acute hospitals. Patients receive medical and nursing care, as well as rehabilitation services. There are also palliative services for specific wards in community hospitals. We approached participants aged 21 years and older who were able to understand and speak English, without a diagnosis of dementia, and who were not admitted under the palliative care service.

The second phase from June to December 2022 involved a cross-sectional study, whereby all participants responded to the ACPES at baseline, and a subset of those participants repeated the ACPES two weeks later. We recruited a new sample of 150 patients and caregivers across two community hospitals, Outram Community Hospital and Sengkang Community Hospital. A sample size of ≥ 100 is considered very good as per COSMIN recommendations [23]. Participants were recruited by convenience sampling. Participants who fulfilled the inclusion criteria were approached inpatient and recruited into the study. We also approached caregivers by patients’ bedsides during visiting hours. Participants who were at least 21 years old, able to understand and speak English, and agreeable to participate were recruited. Those who had a diagnosis of dementia and patients admitted under palliative care were excluded from the study.

Content validity

Content validity examines the extent to which the concepts of interest are comprehensively represented by the items in the questionnaire [24]. It is defined as the degree to which the content of an instrument is an adequate reflection of the construct to be measured [23]. COSMIN assesses three aspects of content validity: relevance, comprehensiveness and comprehensibility [22]. Professionals are asked about relevance and comprehensiveness. Patients are asked about relevance, comprehensiveness and comprehensibility.

To assess the content and face validity of the ACPES, we invited four reviewers who are experts in palliative medicine and ACP to assess the relevance and comprehensiveness of the questions independently. The 15-item ACPES was presented to a group of four experts. One is a palliative medicine specialist with over two decades of experience, one is a family physician with special training in palliative care who has over a decade of experience in looking after palliative care patient, one is a medically-trained researcher with extensive experience and publications in palliative care research, and one is a master medical social worker involved in palliative care education and research.

Each item was analysed by the experts for relevance to the construct of interest, relevance to target population of interest, relevance to context of use of interest, if the response options were appropriate, if the recall period was appropriate, and if key concepts were missing. Each item was scored as either accepted, rejected or accepted with modification. Based on the reviewers’ analysis, the items in the survey were accepted without modification.

We purposively sampled 10 patients of different ages, races, religions and education levels, who participated in cognitive interviews. The cognitive interviews were carried out to explore if participants found the ACPES easy to understand, relevant, easy to complete and comprehensive. They were also asked if there were missing, ambiguous or inappropriate items. Based on these interviews, participants found the questions relevant, comprehensive and comprehensible, and no modifications were required.

Internal consistency

Internal consistency is a measure of the extent to which items in a questionnaire are correlated, thus measuring the same concept [24].

We performed all statistical analyses using IBM SPSS version 26.0. We calculated Cronbach’s Alphas to examine if the responses to the items were inter-correlated, which are considered sufficient when above 0.70, and preferably below 0.95. If Cronbach’s Alpha > 0.95, it may indicate redundancy of items. We assessed per subscale of self-efficacy and readiness to see if they measured the same underlying construct.

Test-retest reliability

We assessed test-retest reliability by asking 25% of the respondents to complete the ACPES twice, first at baseline and then after two weeks. The intraclass correlation coefficients (ICC) were calculated using a two-way mixed effects model. ICC ranges from 0 to 1. An ICC ≥ 0.9 indicates excellent reliability, ≥ 0.8 indicates good reliability, and ≥ 0.7 indicates moderate reliability.

Construct validity

Construct validity is the degree to which the scores of an instrument are consistent with hypotheses, based on the assumption that the instrument validly measures the construct to be measured [23]. Convergent validity is a subtype of construct validity that can be assessed by hypotheses testing, determining whether the scores of the instrument are consistent with that of other related instruments. The hypotheses are formulated a priori, based on existing knowledge about the construct. The construct validity of the ACPES is considered adequate when 75% of hypotheses are supported [24]. We had the following hypotheses:

• 1) Higher ACP engagement scores are associated with having made a will.

• 2) Higher ACP engagement scores are associated with having a lasting power of attorney (LPA).

• 3) Higher ACP engagement scores are associated with having told one’s family about end-of-life preferences.

• 4) Higher ACP engagement scores are associated with having told one’s doctor about end-of-life preferences.

• 5) Higher ACP engagement scores are positively correlated with increasing age.

Survey administration

The ACPES was developed to be self-administered or interviewer-administered. Most surveys were self-administered, unless participants had difficulty with reading, in which case trained study members administered the survey. We also collected data on basic demographic information and self-reported pre-planning activities. Participants were asked if they had made a will, had made a LPA, had told their doctor about end-of-life care preferences, and had told their family about end-of-life care preferences. For the test-retest reliability, we had 25% of participants retake the ACPES after two weeks. Inpatients who were still admitted two weeks later were approached to repeat the survey.

Results

150 patients and caregivers completed the survey. The participants’ characteristics are shown in Table 1. Half the participants were male. The mean age was 59.1 years old, with a range from 23 to 92 years old. 50% were aged above 60 years old. The participants were predominantly Chinese, with 58.7% completing tertiary education. There were a mix of religions with Buddhism 29.3%, Christianity 28.7%, and Islam 9.3%. This sample reflects the diversity of the Singapore population, and is similar to census data [1].

Table 1 Demographic characteristics of participants (n = 150)

Table 2 shows the mean scores for the ACPES. The mean ACPES score in this study was 2.84 (SD 1.04) for the 15-item version, 2.78 (SD 1.08) for the 9-item version, and 2.31 (SD 1.26) for the 4-item version. The mean self-efficacy score was 3.64 (SD 1.15). The mean readiness score was 2.33 (SD 1.21). The item-total correlations were 0.47–0.83 which suggests that each item correlates well to the survey (Table 2).

Table 2 Descriptive statistics: mean, SD and item-total correlation

Reliability

As shown in Table 3, the Cronbach’s alpha was 0.945 for the 15-item version, 0.915 for the 9-item version, and 0.912 for the 4-item version.

The ICC was calculated for the 38 participants who responded to the repeat survey two weeks later. The ICC was 0.933 for the 15-item version, 0.938 for the 9-item version and 0.865 for the 4-item version.

Table 3 Reliability (internal consistency and test-retest reliability) of the ACPES

Construct validity

For associations of the ACPES score with pre-planning activities, we used point-biserial correlation. There was a positive correlation between ACPES scores and pre-planning activities such as making a will, making a LPA, telling one’s doctor about end-of-life care preferences, and telling family or loved ones about end-of-life care preferences (Table 4). This supports our first four hypotheses.

Table 4 Association between pre-planning activities and ACPES score, point-biserial correlation rpb

We used Pearson correlation coefficient for the fifth hypothesis. The ACPES scores were significantly positively correlated with increasing age only in the 4-item version, with Pearson correlation coefficient r = 0.17 (Table 5).

Table 5 Association between age and ACPES score, pearson correlation coefficient

Item score distribution

There were 10 items in which > 15% of participants chose the lowest score. There were 11 items in which > 15% of participants chose the highest score (Table 6).

Table 6 Item score distribution, reported if > 15%

Discussion

In this study of 150 patients and caregivers in 2 community hospitals, we found that the 15-item, 9-item and 4-item versions of the ACPES have adequate content validity, excellent internal consistency, test-retest reliability and construct validity.

Based on the Cronbach’s alphas, all three versions of the ACPES (15-item, 9-item, and 4-item) demonstrate excellent internal consistency, with values above the commonly accepted threshold of 0.70, suggesting that the survey items are highly correlated and measure the same construct of ACP engagement. These values are similar to the original validation study by Sudore et al., where the Cronbach’s alpha values were 0.92, 0.89 and 0.84 for the 15-, 9- and 4-item versions respectively [19].

The intraclass correlation coefficient (ICC) values also suggest that the three versions of the ACPES have excellent test-retest reliability, and that individuals will have consistent results over time.

For construct validity, the four hypotheses that were supported were that higher ACP engagement scores are significantly associated with having made a will, a LPA, having told one’s family about end-of-life preferences and having told one’s doctor about end-of-life preferences. The ACPES scores on the 15-, 9- and 4-item versions of the survey were positively correlated and statistically significant for people who engaged in pre-planning activities (Table 7), which is also consistent with other studies. In the Japanese study, scores were significantly higher for those who agreed with “I have filled out a living will or advance directive,” “I have told my doctor about my preferences for end-of-life care,” and “I have told my friends or family about my preferences for end-of-life care” [17]. In Sudore’s study, ACPES scores were also significantly higher for those who had made a will, made funeral plans, or made an advance directive [15]. This was applicable to the original 82-item and shorter versions of the survey, including the 15-, 9- and 4-item versions.

Table 7 Comparing means for ACPES score with pre-planning activities, mean (SD)

The fifth hypothesis was that higher ACP engagement scores are positively correlated with increasing age. Older age has been previously associated with greater ACP engagement and completion [14, 25,26,27,28]. It was hypothesised that older adults may be more likely to engage in ACP as they may be more aware of their mortality and may have experienced the death of friends or family members. As a result, they may be more motivated to make sure their end-of-life wishes are known. In addition, older adults may be more likely to have chronic health conditions that increase their risk of becoming seriously ill or incapacitated. As a result, they may be more involved in discussing their end-of-life wishes. This study showed that survey scores were positively correlated with increasing age only in the 4-item survey, but not for the 9-item or 15-item versions.

There are a few possible explanations why increasing age was not correlated to higher ACPES scores. Asian cultures have a taboo surrounding the open discussion of end-of-life issues, believing that it would bring bad luck [29, 30]. Patients may also wish to entrust decision-making to family members or doctors [29]. A qualitative study in Singapore revealed that patients were comfortable to leave the future to their family, particularly their children, as they assumed they would do what was appropriate [30]. Patients may also be concerned that ACP would cause distress to themselves or to family members, feel that ACP is irrelevant or not important, or are uncomfortable talking about death [29, 31].

The mean ACPES scores in this study was 2.84 for the 15-item version, 2.78 for the 9-item version, and 2.31 for the 4-item version. These scores are lower than those in the original study, which were 3.16, 3.11, and 2.7 for the 15-, 9- and 4-item versions respectively [15]. This could be due to higher exposure to ACP in the participants in the original study, where 48% had completed an advance directive and 44% had made life or death decisions for themselves [14]. Furthermore, white race is associated with higher ACP uptake and completion compared to other races [25, 27, 28]. The scoring thresholds associated with behaviour change leading to improved ACP uptake are not known; further studies will be needed to determine the thresholds for a full range of ACP behaviours that lead to action taken on ACP completion.

There are limitations to this study. Firstly, our study recruited only participants who were able to understand and speak English. Singapore is a multicultural society and for the older generation, English is not often their first language. Hence, this may limit the feasibility of use in Singapore, especially with the geriatric population. Secondly, floor and ceiling effects can occur when existing measurements are applied to a population which is less or more severely diseased than the population for which the instrument was originally developed for [32]. Our study recruited patients and caregivers in community hospitals, whereas the original study recruited a mix of adults in inpatient and outpatient settings, including primary care, dialysis centres and cancer centres.

Overall, these results suggest that the 15-, 9- and 4-item versions of the ACPES are reliable and consistent measures of assessing ACP engagement in the general population in Singapore. Clinicians can choose between the 3 versions depending on their goal. The 4-item version is valid and useful as a quick screening tool to assess readiness for ACP. The longer versions, which covers questions relating to readiness and self-efficacy, may be more useful when trying to evaluate slight differences in ACP engagement. The 15-item version additionally assesses readiness and self-efficacy in the domain of flexibility when it comes to decision making, hence clinicians interested in finding out patients’ thoughts on this can also opt for this version.

The results of the survey can be used to guide ACP discussions and help healthcare providers tailor their communication and education efforts to the individual’s needs and preferences. Further studies to translate and validate the ACPES into other languages such as Chinese, Malay and Tamil may be considered for use in Singapore and the region.

For future research, we recommend to study the effects of the ACPES in longitudinal research, perform ACP interventions to assess the responsiveness of the ACPES, and to determine the score thresholds that lead to improved ACP uptake. After responsiveness has been assessed, the ACPES could be used to measure the effect of interventions to increase ACP engagement.

Conclusion

This study provided good psychometric support for the validity of the 15-item, 9-item and 4-item versions of the ACPES in Singapore. This instrument would be helpful to clinicians in Singapore, who could use it to assess patients’ readiness to discuss ACP.