Introduction

Resident food intake in long-term care (LTC) is influenced by a wide variety of factors, including the mealtime experience (1). Various physical and psychosocial attributes of the dining environment contribute to the overall mealtime experience and they can also influence residents’ food intake, satisfaction with their meals, and enjoyment (2–5). The Mealtime Scan (MTS) (2, 6) is a tool that was designed to evaluate in a standardized manner the physical and psychosocial aspects of the mealtime environment in dining areas in LTC homes. The MTS is based on best-practices of environmental design in dementia care facilities (7) and grounded in prior work on mealtime expectations of persons living with dementia (4, 6, 8) as well as best practices in mealtime care (8). The scan has demonstrated inter-rater reliability (6) and construct validity (2).

MTS was used in a large multi-site study (2, 3) and although found to be valuable for differentiating mealtime experiences and dining rooms (2, 10), improvements were needed to expand on the rating of social interactions observed and to improve scaling so that responsiveness as a result of targeted interventions would be enhanced (9). The objectives of this paper are to: a) provide a detailed explanation of the specific revisions, resulting in the MTS+ tool and b) describe the interrater reliability of this revision.

Design and Methods

Revisions Resulting in MTS+

The investigator team has been using the MTS for a variety of practice-focused opportunities since its development and original testing (6). Variations on scales and scoring, resulting in the new MTS+ were trialed in an iterative manner until the resulting tool was found to be sufficiently standardized, could discriminate various environments, and could also be responsive to changes in the mealtime environment (9). The following specific revisions were made:

Physical Environment

The original MTS tool included several subcomponents in the physical environment: number and type of persons (e.g., staff member, resident, student, volunteer) in the dining room; environmental metre readings; presence of orientation cues; observed sounds and use of radio/TV and music. Measurements from an environmental metre to assess luminescence, humidity, noise and temperature were conducted at the beginning and a mid-point in the meal, at two or more locations in the dining room. These measures were not found to be useful in differentiating dining rooms (2) and also found to be disruptive to the mealtime, and were recommended for removal (2). To capture key components, this metre was substituted with a subjective rating for lighting sufficiency and comfortable temperature, scored as none (0), some (1), most (2) or all (3) residents experiencing. Disruptive noise (e.g. dish clearing, staff calling) was separated out from sound that resulted from social interaction on this revision. Using a list of 10 potentially disruptive sounds (e.g. vacuum cleaning during the meal, alarm bells, staff calling out, as well as an ‘other’ category), excess noise was rated as low (0) to high (3) with a maximum score of 30. Coding rules were provided for each rating (e.g., 2= noise is minimal to moderate, interferes with some mealtime experiences/processes). Other additions to the physical environment component of this tool included noting the types and frequency of table arrangements (i.e., number of seats per table, and number of residents seated at each table configuration). Number of persons in the dining room, orientation cues, music and television use during the meal were unchanged from the original version. The physical environment overall rating of 1 (low) to 8 (high) did not change, but detail for coding each component of the physical environment (e.g. lighting, orientation cues) was provided for each point on this scale. Excess noise was considered the most important aspect of the dining environment and is used to drive the overall physical environment rating.

Social Environment

Although the original MTS included an overall social rating scale, it did not capture specific types of social interactions. The MTS+ includes a more elaborate section specifically aimed at rating various social interactions. Five types of interactions, some of which are positive (e.g., social talk and affection) and others are negative (e.g., chastising residents) are now included in the MTS+. Frequency of occurrence of each interaction between various actors (e.g., staff with residents, residents with residents) is captured based on a ranking of 0 (never occurred) to 4 (occurred often). The overall social environment rating from 1 (low) to 8 (high) is consistent with the original MTS, but a detailed coding scheme is provided based on the types of social interactions observed. Frequent negative social interactions result in a lower score. Social interactions involving residents are used to drive the overall social interaction rating.

Relationship-Centered Environment

The relationship-centred environment is based on three subcomponents. The items on the Mealtime Relational-Care Checklist (M-RCC; 17 items), Relational Care for Residents Requiring Eating Assistance (9 items), and Mealtime Clean-up (two items) included in the original MTS were unaltered (6, 10). However, scoring of these tasks and relational practices was changed from present/absent to 0 (no incidents of relational practice observed), 1 (more task focused than relational), 2 (equal amount of task focused and relational), 3 (more relational than task focused) and 4 (relational care practices consistently observed). The maximum sum of the M-RCC checklist is 60 points, an additional 32 points for those residents requiring eating assistance and 8 more for the practices involved in meal clean-up. A tally sheet is also included for describing any resident expressions of distress (e.g. anger, wandering) or well-being (e.g., laughing, singing) so that they can be captured; however, these are not used to drive the Relationship-Centred rating. Similar to the overall physical and social environment ratings and consistent with the original MTS, the overall relationship-centred environment is rated from 1 (low) to 8 (high), with descriptors and cut-points provided for each scaled component (e.g., M-RCC total < 30). These cut-points were based on use of the tool in 45 mealtime observations with one rater to determine logical groupings that discriminated mealtime environments.

Overall Quality of Dining Environment

The original MTS did not have an overall quality rating for the dining environment. It was considered important going forward to develop such an overall rating that encapsulated the physical, social and relationship-centred environments. The overall quality of dining environment rating had a scale consistent with its three subcomponents (e.g. 1=low and 8=high) and detailed descriptors support consistency in rating. Coding rules are provided (e.g. use average of physical, social and relationship-centred ratings) to also promote consistency. A detailed protocol for training raters was also developed iteratively and is used for training assessors.

Testing Inter-Rater Reliability of MTS+

To promote consistency, two trainees with a similar level of experience in LTC or dementia were trained to conduct the MTS+. They were trained by the lead authors, who have extensive experience in LTC, ageing, nutrition, and dementia. The training consisted of a standard protocol (6); readings on mealtime environments and person- and relationship-centred care; YouTube video demonstrations of mealtimes in various contexts; in-person discussions; and three practice observations using the MTS+ and discussing ratings with the lead authors who observed the same meal. These are recommended methods for training in observational data collection (11).

Three diverse residential care homes were included in this study; two of which were not-for-profit charitable organizations, while one was owned by a multi-site, for-profit, long-term provider. A total of nine dining rooms, three from each home, were included in this inter-rater reliability study; one dining room chosen from each home was a designated dementia-care area. Each dining room was observed at the same time by the two assessors for five meals, resulting in a total of 45 meals observed. Ethics review did not require individual consent from those present in the dining rooms during observations. Newsletters, staff and family/resident written communications were used to raise awareness of the study. A notice before each observed meal was posted near the dining room and staff, residents, family and volunteers in the dining room had the opportunity to ask to be excluded from the observation. Most of the dining rooms had a smaller area that was not included in the study observations, where residents could eat if they did not want to be observed. In this study there were no instances when a resident or team member requested that they not be observed. The assessors were instructed to maintain their status of unobtrusive observation while standing in close proximity to each other; they were encouraged to move around the dining room discreetly as needed (e.g., for visibility and audibility purposes). Observations started when most residents were in the dining room for the meal and continued until most residents had left the dining room. The assessors did not share their results and independently entered their results into an Excel data base, which was then amalgamated by the researchers for analysis. An ethics board at the University of Waterloo provided clearance for this study (ORE# 31001).

Statistical Analyses

Descriptive statistics (i.e., mean, range, standard deviation, frequency, proportions) from components of the MTS+ were calculated. Interclass Correlation Coefficient (ICC) estimates and their 95% confidence intervals based on a mean rating (k=2), absolute agreement, and 2 way mixed effect model were determined (12). An ICC value of >0.9 was considered excellent, 0.75–0.9 “good”, 0.5–0.75 “moderate”, and less than 0.5 “poor” (13). All analyses were completed using SPSS statistical package version x (SPSS Inc, Chicago, IL).

Results

Table 1 provides descriptive characteristics for the dining rooms assessed. Most residents (mean = 23, SD 2.6) ate in the dining rooms during observed mealtimes, while very few ate in an adjacent area at each meal (mean = 1, SD 1.0). The number of residents requiring eating assistance per meal was somewhat variable (mean = 6, SD 3.0) and a mean of 3 staff members (SD 1.4) were involved in eating assistance. Overall, the mean number of people in the dining rooms was 32 (SD 3.9), including three family members and/or volunteers (SD 2.1). The television could be heard/viewed at only 11% of observed meals, while music was noted at 40% of the meals. Orientation cues including table settings and food aroma, among others, averaged 5.3 points out of a maximum of 14 points (SD 1.3) while excess noise had a mean score of 8.2 (SD 2.6) out a maximum score of 30. The average M-RCC score was 35.3 (SD 3.9) out of 60, indicating a moderate level of relationship-centred practices. The global score for the physical, social and relationship centred environments, as well as the overall quality of the dining environment were moderate with average scores ranging between 3.1 and 4.2 out of 8.

Table 1 Descriptive Characteristics of Dining Areas Assessed by MTS+ (n=45)
Full size table

Table 2 presents the inter-rater reliability for six components of the MTS+. The overall quality of dining environment rating had moderate reliability (ICC= 0.67) indicating that this summative rating for the mealtime environment is reliable among raters. Global physical environment rating had the lowest reliability (ICC= 0.48) and was considered poor, despite excess noise (ICC= 0.65) and orientation cues (ICC= 0.75) having moderate and good reliability respectively. The global social environment (ICC= 0.78) and relationship-centred environment (ICC= 0.78) demonstrated good reliability. There was variability in the reliability of ICC as noted by the relatively large 95% confidence intervals, especially for the physical components and overall physical environment score.

Table 2 Inter-Rater Reliability of Key Features of MTS+ (n=45)
Full size table

Discussion

Compared to its original version, the MTS+ was designed to capture more social aspects of the mealtime environment in LTC homes and provide more detailed scaling so that the tool would be responsive to interventions. These modifications required the re-evaluation of the reliability of this tool. The findings of this study reveal that the inter-rater reliability of MTS+ using the overall dining quality environment rating is moderate, but adequate for use in research.

In comparison to the original MTS, the new MTS+ demonstrated similar ICC estimates on social and relationship-centred environment ratings as well as the excess noise score and all had moderate (ICC >0.6) or good (ICC > 0.75) reliability. However the physical environment had lower reliability (MTS ICC=0.73 vs. MTS+ ICC=0.48). This decline may be related to the changes made to the procedures used to rate various aspects of the physical environment. Specifically, MTS+ measures some environmental conditions through observer perception rather than an objective scale based on an environmental meter. It is however surprising that the excess noise score, which was based on observer perception in both versions, had exactly the same ICC (ICC 0.65) and raters were trained to use this component to drive the physical environment rating. This suggests a potential training deficit with determining the physical environment rating. It is recommended to maintain the MTS+ modifications for the physical environment as they are more feasible than the environment metre, but that greater training be provided, especially by practicing the use of the MTS+ in diverse physical environments.

Despite the other ICC values for the MTS+ items being in the moderate or good range, a note of caution on reliability must be made due to the size of some of the 95% confidence intervals. These variations suggest some unpredictability of the reliability of scoring and reinforces the approach taken by the creators in advocating that the same trained rater be used for observing an environment that is involved in repeat measurements over time to determine changes as a result of improvement plans or interventions. This will ensure that if a change is noted, it is due to an actual change in the dining environment and not a result of questionable reliability among assessors. For multi-site studies where the description of the dining environment is the goal and there are many raters involved, the original MTS with its higher reliability (and less complicated rating system) is potentially preferable. Construct validity of MTS and the M-RCC checklist within the MTS have been demonstrated (2, 10). Despite the modifications made with MTS+, it is anticipated that the tool continues to be construct valid with respect to food intake and other measures of the dining environment, as the concepts being assessed are consistent with the original version.

There are some limitations to this study. Three or more assessors are recommended for robust determination of interrater reliability (13). Due to the nature of the extensive training required for MTS+, limited resources, including access to homes, and in particular the disruption that three or more assessors in a single dining room would cause, we were unable to include more than two assessors in this reliability testing. It is noted that this limitation leads to lower ICC estimates (13) and may explain the poor value for the physical environment with its relatively large 95% confidence intervals. Although the authors hypothesized that intra-rater reliability of the tool is likely to be higher than inter-rater reliability for MTS+, it is not feasible for an individual observer to assess the same meal without multi-camera video recording of an entire dining room. At the time of this study, such recording of the entire dining area was not available, nor would it have been viewed by the homes, staff and residents or their families, as appropriate.

In conclusion, MTS+ is a reliable tool for assessing the overall quality of the dining environment, as well as the social and relationship-centred environments and the rating of excessive noise. At this time, it is the only comprehensive and sufficiently detailed tool for use in research to examine the mealtime environment from the physical, social and relationship-centred aspects. As noted in this study, detailed training is needed, especially for the physical environment, and this tool should be considered as a research-level instrument, rather than used by LTC providers to describe their dining rooms. The overall quality of the dining environment rating provides a summary variable for intervention research and can be used with confidence, especially when the same rater is used in a time series or pre/post intervention assessment to determine changes in the mealtime environment.

Conflict of Interest: HK and HC are originators of the MTS and MTS+. There is no financial conflict of interest. SA and JM declare no conflicts of interest.

Ethical standard: A University of Waterloo Research Ethics Board provided clearance for the conduct of this study.