INTRODUCTION | PARTICIPANT DEMOGRAPHICS FORM
SCORING | RELIABILITY AND VALIDITY | DOWNLOAD THE POSTER

Reliability and Validity of Assistive Technology Outcome Measure (ATOM), Version 1.0 for Adults with Physical Disability using Wheelchairs

Dharne, Medha, G., MS, Lenker, James, PhD, Harris, Frances PhD, and Sprigle, Stephen, PhD.

ABSTRACT

The assistive technology field needs device-specific outcome instruments to measure the usability, effectiveness, and impact of particular classes of assistive devices. The Assistive Technology Outcome Measure (ATOM) is a new tool that seeks to measure the impact of wheeled mobility systems. This study establishes the internal consistency, test-retest reliability and convergent validity of ATOM, version 1.0. Convenience sampling technique was used to recruit 54 adults using wheelchairs. The results indicate that ATOM has strong internal consistency, good test-retest reliability and fair convergent validity. Future research should aim to establish a standardized version of ATOM that can be used clinically with a variety of population in a variety of clinical settings.

KEYWORDS: wheelchair, wheeled mobility device, outcome measurement, assistive technology

INTRODUCTION

By measuring the impact of assistive technology devices (ATDs), one can demonstrate the effectiveness of AT intervention strategies and the practitioners and programs that provide AT services [1]. Unfortunately, there are few fully validated AT outcome measurement tools. In particular, the field needs tools capturing the impact of particular classes of devices, so-called device-specific measures [2]. In the specialty area of wheeled mobility and seating, a clinically friendly outcome measurement tool is needed that will respond to increasing calls from third-party funding agencies for evidence of clinical effectiveness.

LITERATURE REVIEW

We reviewed 17 peer-reviewed articles (Appendix A), published between 1991 and 2004, reporting on the outcomes of seating and wheeled mobility device interventions. Among these, there were three approaches to measuring ATD impact: (a) eight authors used a standardized tool; (b) seven used tools that were developed specifically for the study being reported; and (c) two used open-ended interviews as part of a qualitative methodology. Among the eight articles using a standardized tool, no single tool was repeated twice. Clearly, there has yet to be a gold standard tool to emerge for measuring seating and mobility outcomes.

The Assistive Technology Outcome Measure (ATOM) was conceived and originally pilot tested by researchers at Helen Hayes Hospital in West Haverstraw, NY. The goal was to develop an easily administered tool that would capture a range of seating and mobility impacts: usage in different environments, community participation, function, assistance, comfort, and hassle. It was hoped that the ATOM would yield useful data for consumers, clinicians, program administrators, and third-part funding agencies. Pilot data was captured from 56 participants who had received a seating and wheeled mobility intervention. A revised form known as ATOM version 1.0 was subsequently developed to clarify item wording and response options. The content validity index of ATOM v.1.0 was 0.881 [3]. Heretofore, its other psychometric properties have not been evaluated. The purpose of the current study was to establish the internal consistency, test-retest reliability and convergent validity of ATOM v.1.0.

METHOD

Participants

The participants included a convenience sample of 54 adults with physical disability who had used a wheelchair as their primary mobility device for at least six months. Potential participants were identified through 'Buffalo Wheelchairs,' a regional wheelchair vendor. The participants were given a $40 honorarium in consideration for their time.

Instruments

ATOM v.1.0 (see Appendix B) and Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0) were the two instruments used.

Procedure

Initially, ATOM was pilot tested on four participants. Each of the remaining 50 participants was interviewed twice. First interview was in person; both ATOM and QUEST were administered. One week later, during the second interview ATOM was administered over the telephone.

Analysis

Cronbach's coefficient alpha, Intra-class correlation coefficient (ICC 3,1) and Pearson's product moment correlation coefficient were used for the analysis. In addition, descriptive statistics was calculated for the demographic variables. Frequencies of item responses of ATOM v.2.0 at time one were calculated to note any ceiling and floor effect.

RESULTS

Following pilot testing no revisions were made in the tool. Among 50 participants using wheelchairs 50% were males and 50% were females. Participants' ages ranged from 24 to 86 years (Mean 51.3 years, SD 14.22). All demographic information is summarized in Appendix C.A total of 52% participants used manual wheelchairs and the remainder used powered wheelchairs. The demographic information regarding wheelchair and its usage is summarized in Appendix D. Out of the 18 items, 17 items elicited the entire range of the response options i.e. all response options were chosen by at least one participant. The overall ATOM average for all 50 participants at time 1 was 3.16 with a range from 2.11 to 4.0. Cronbach's alpha for all 18 items was calculated to be 0.825 (N=18). The intra-class correlation coefficients (ICC) for all 18 items of ATOM at time 1 and time 2 are summarized in Appendix E. The ICC for the overall ATOM mean at both times was calculated to be 0.869 (N=50). The correlation coefficients between ATOM and QUEST items are summarized in Appendix F. The correlation coefficient for the overall ATOM and QUEST means was 0.303 (p = 0.05, 2 tailed). In addition nine other correlations were statistically significant.

DISCUSSION

The Cronbach's alpha value of 0.825 shows a moderate correlation among the items. Only a scale with strong internal consistency will show a moderate correlation (alpha between 0.70 and 0.90) among its items [4]. Items with a high alpha value (alpha > 0.9) are probably redundant.

The intra-class correlation coefficient value of 0.869 for the overall ATOM means indicates good reliability [4]. A value of at least 0.70 is required for an acceptable degree of reliability [5]. Six items of ATOM have an ICC value lower than 0.70 which indicates poor to moderate reliability. Out of the six items, two items with an ICC value of less than 0.50 will be examined closely for revision. While the other four items with an ICC value of equal to or more than 0.50 but less than 0.70 will require more testing before they could be considered for revision. One reason for obtaining this result could be the use of two different modes of data collection. At time 1 data was collected through an in person interview and at time 2 a phone interview was conducted. At time 2 during the data collection, some of the participants asked that questions be repeated. A few provided responses that were inconsistent with their demographic information. In these instances, the questions were repeated in order to assure that the participants clearly understood the item and its responses. These experiences suggest that some participants may had difficulty understanding the interviewer over the telephone, perhaps due to distractions occurring in their home environment at the time of the interview and perhaps due to the accent of the principal investigator for whom English is a foreign language. Overall a low correlation between the items of QUEST and ATOM was obtained; this suggests that ATOM is not merely a duplication of QUEST. QUEST mainly is an instrument to measure the satisfaction of the client with the assistive technology device. While ATOM is a comprehensive tool that measures not only satisfaction but also usage, function, hassle, comfort, and effectiveness of a wheeled mobility device. Hence the items on both outcome tools that are measuring similar construct have a statistically significant correlation. E.g. Item 7 of ATOM - What is your comfort level while sitting in your wheelchair (or scooter)? , Item 7 of QUEST - How comfortable your assistive device is? Although the low correlation values do not indicate good convergent validity of ATOM, this effort was the first step towards establishing the construct validity of ATOM. In the future, perhaps a more appropriate instrument should be used for this purpose.

Thus it can be concluded that ATOM v.1.0 has a strong internal consistency, good test-retest reliability and fair convergent validity.

LIMITATION

The only limitation of the study that might have affected the scores was the mode of collecting data. There were two interviews and each interview happened in different forms: first interview was in person and second interview was over the telephone. Since the second interview was not face to face, full attention on part of the interviewee was not guaranteed.

FUTURE RESEARCH

Future research should aim to establish a standardized version of ATOM that can be used clinically with a variety of population in a variety of clinical settings. First, there is a need to develop a user's manual. Secondly research should be conducted to establish other psychometric properties of this tool (e.g. inter-rater reliability, construct validity, and predictive validity). Inter-rater reliability should be aimed to establish the reliability of proxy respondents; for cases where end users are unable to respond reliably for themselves (e.g. children, older people with dementia, or people with moderate to profound mental retardation).

ACKNOWLEDGMENTS

This research in this article was supported in part by:
Office of the Dean, School of Public Health & Health Professions, University at Buffalo

REFERENCES

  1. Gelderblom, G. J. and Witte, L. P. (2002). The assessment of assistive technology outcomes, effects and costs. Technology and Disability, 14, 91-94.
  2. Lenker, J., Scherer, M., Fuhrer, M., Jutai, J., and DeRuyter, F. (2005). Psychometric and administrative properties of measures used in assistive technology device outcomes research. Assistive Technology, 17.1, 7-22.
  3. Dharne, M. G., Lenker, J., Harris, F., and Sprigle, S. (2006). Content Validity of Assistive Technology Outcome Measure (ATOM) version 2.0. Conference Proceedings of the Rehabilitation Engineering and Assistive Technology Society of North America (RESNA), USA.
  4. Portney, L.G. and Watkins, M.P. (2000). Foundations of clinical research applications to practice. (2nd ed.). (pp. 724). New Jersey. Prentice-Hall, Inc.
  5. Nunally, J. C. and Bernstein, I. R. (1994). Psychometric theory. (3rd ed.). New York: McGraw-Hill.

Appendix A

Citations

  1. Brandt, A., Iwarsson, S., Stahle, A. (2004). Older people's use of powered wheelchair for activity and participation. Journal of Rehabilitation Medicine, 36, 70-77.
  2. Chaves, E.S., Boninger, M.L., Cooper, R., Fitzgerald, S.G., Gray, D.B., and Cooper, R. A. (2004). Assessing the Influence of Wheelchair Technology on Perception of Participation in Spinal Cord Injury. Archives of Physical Medicine Rehabilitation, 85, 1854-1858.
  3. Crane, B.A., Holm, M.B., Hobson, D., Cooper, R.A., Reed, M.P., and Stadelmeier, S. (2004). Development of a consumer-driven Wheelchair Seating Discomfort assessment Tool (WcS-DAT). International Journal of Rehabilitation Research, 27, (1), 85-90.
  4. Davies, A., Desouza, L. H., and Frank, A. O. (2003). Changes in the quality of life in severely disabled people following provision of powered indoor/outdoor chairs. Disability and Rehabilitation, 25(6), 280-90.
  5. Devitt, R., Chau, B., and Jutai, J. (2003). The effect of wheelchair use on the quality of life of persons with multiple sclerosis. Occupational Therapy in Health Care, 17, (3/4), 63-79.
  6. DiGiovine, M.M., Cooper, R.A., Boninger, M.L., Lawrence, B.M., VanSickle, D.P., and Rentschler, A.J. (2000). User Assessment of Manual Wheelchair Ride Comfort and Ergonomics. Archives of Physical Medicine Rehabilitation, 81, 490-494.
  7. Evans, R. (2000). The effect of electrically powered indoor/outdoor wheelchairs on occupation: a study of user's views. British Journal of Occupational Therapy, 63, (11), 547-53.
  8. Fuchs, R. H., and Gromak, P.A. (2003). Wheelchair Use by Residents of Nursing Homes: Effectiveness in Meeting Positioning and Mobility Needs. Assistive Technology, 15, 151-163.
  9. Graber, S.L., Bunzel, R., and Monga, T. N. (2002). Wheelchair utilization and satisfaction following cerebral vascular accident. Journal of Rehabilitation Research and Development, 39, (4).
  10. Kettle, M., Rowley, C., and Chamberlain, M. A. (1992). A national survey of wheelchair users. Clinical Rehabilitation, 6, 67-73.
  11. Miles-Tapping, C. (1997). Power wheelchairs and independent life styles. Canadian Journal of Rehabilitation, 10, (2), 137-45.
  12. Mills, T., Holm, M.B., Trefler, E., Schmeler, M., Fitzgerald, S., and Boninger, M. (2002). Development and consumer validation of the Functional Evaluation in a Wheelchair (FEW) instrument. Disability and Rehabilitation, 24, 1/2/3, 38-46.
  13. Shaw, G. (1991). Wheelchair Seat Comfort for the Institutionalized Elderly.Assistive Technology, 3, 11-23.
  14. Shaw, G., and Taylor, S. J. (1991). A Survey of Wheelchair Seating Problems of the Institutionalized Elderly. Assistive Technology, 3, 5-10.
  15. Stanley, R.K., Stafford, D.J., Rasch, E., and Rodgers, M.M. (2003). Development of a functional assessment measure for manual wheelchair users. Journal of Rehabilitation Research and Development, 40, (4), 301-308.
  16. Wessels R., De witte L., Andrich R., Ferrario M., Persson J., Oberg B., Oortwjn W., Vanbeekum T., Lorentsen O. (2000). IPPA, a user centered approach to assess effectiveness of AT provision. Technology and Disability, 13(2): 105-15.
  17. Wressle, E. and Samuelsson, K. (2004). User Satisfaction with Mobility Assistive Devices. Scandinavian Journal of Occupational Therapy, 11, 143-150.

Appendix B

Assistive Technology Outcome Measure (ATOM) Version 1.0 (Microsoft Word document).

Appendix C

Demographic Information General (Microsoft Word document)

Appendix D

Demographic Information Wheelchair and usage (Microsoft Word document)

Appendix E

Intra-class correlation coefficient of ATOM (Microsoft Word document)

Appendix F

Correlation between ATOM and QUEST

0.303*
Item number QUEST TotalOverall ATOM mean
ATOM item 1 0.072n/a
ATOM item 2 -0.004n/a
TOM item 3 0.039n/a
ATOM item 4 -0.058n/a
ATOM item 5 0.139n/a
ATOM item 6 0.318* n/a
ATOM item 7 0.524** n/a
ATOM item 8 0.325* n/a
ATOM item 9 0.329* n/a
ATOM item 10 -0.143 n/a
ATOM item 11 0.093 n/a
ATOM item 12 0.147 n/a
ATOM item 13 0.004n/a
ATOM item 14 0.274 n/a
ATOM item 15 0.167 n/a
ATOM item 16 0.206 n/a
ATOM item 17 0.335* n/a
ATOM item 18 0.416** n/a
QUEST item 1n/a 0.30
QUEST item 2n/a 0.106
QUEST item 3n/a 0.4**
QUEST item 4n/a 0.233
QUEST item 5n/a -0.026
QUEST item 6n/a 0.175
QUEST item 7n/a 0.29*
QUEST item 8n/a 0.384**
QUEST Total n/a
* Correlation significant at 0.05 (2 tailed) ** Correlation significant at 0.01 (2 tailed)