Choosing Assessment and Evaluation Tools
for Direct Practice
Jane F. Gilgun, Ph.D., LICSW
Smashwords Edition
Copyright 2010 Jane Gilgun
See other articles, books, and children’s stories by Jane Gilgun on Smashwords.com.
About the Author
Jane Gilgun is a professor, School of Social Work, University of Minnesota, Twin Cities, USA.
Summary
Assessment and evaluation tools can contribute to practice effectiveness if social service professionals choose them well. This paper provides guidelines for choosing tools for practice. The first section discusses standardized instruments; that is, instruments that have known psychometric properties of reliability and validity. The second section is on instruments that practitioners construct themselves or that they help clients construct. The third are issues that help or hinder social workers to use assessment and evaluation tools.
Choosing Assessment and Evaluation Tools
for Direct Practice
Assessment and evaluation tools can contribute to practice effectiveness if social service professionals choose them well. In this essay, I provide guidelines for choosing tools for practice. The first section discusses standardized instruments; that is, instruments that have known psychometric properties of reliability and validity. The second section is on instruments that practitioners construct themselves or that they help clients construct.
The third section is brief but points out some of the issues involved in practitioner use of instruments. In the discussion, I state the importance of practitioner involvement in the development, use, and modification of any tools that agencies may require and also point out that funders prefer to sponsor programs that demonstrate effectiveness.
Are they Useful?
Usefulness is the most important question to ask about practice tools. If you use these tools, will they help you do your job better? Tools that are useful have the following characteristics. I’ve arranged them in rough order of importance for social work practice.
They have good face validity
Face validity is the most important validity in assessment and evaluation tools. Face validity means that when knowledgeable professionals read the tools, they find that the tools cover important areas of practice.
They have good content validity
Content validity is an estimate of whether instruments cover relevant areas. It is similar to face validity in that experts decide whether tools have adequate coverage. There is no index for content validity. Drawing upon multiple sources of data helps to ensure content validity. In social work and other applied disciplines, content validity is more likely when the sources of items are research, theory, and practice wisdom that draws upon direct experience with clients and their issues.
Sometimes representatives of client groups contribute to the ideas and items of a tool. Item total analysis often helps streamline instruments because it helps to eliminate items that quantitative analysis shows are unrelated to other items in the tool. In item total analysis, the score on each item is correlated with the total score. Items with very low correlations are eliminated. If many items have high correlations—above .9—tool developers then inspect these items for redundancy and eliminate those that duplicate others.
They are culturally sensitive
Instruments that are useful draw upon information that is culturally sensitive. Practitioners can check for cultural sensitivity by finding information about the samples on which instrument developers draw for the ideas and items that compose the instrument. These samples ideally match the culture, social class, and other important social identities of the individuals who compose practitioners’ caseloads. If the sample differs, the instruments may still be useful if practitioners modify them in consultation with knowledgeable persons. Cultural sensitivity is part of content validity.
They are practice guidelines
Good tools provide practice guidelines in the sense that they help you keep important things about clients in mind. It is only human to have our own favorite ideas about what is important. Useful tools alert you to things that you might not otherwise have thought about.
They help practitioners formulate treatment goals
Tools that provide practice guidelines also help practitioners to formulate treatment goals. Treatment goals, in turn, can help you gauge whether you work is helping clients. If you use the tools periodically, they will also keep you focused on important practice principles. Of course, as you work with clients, you may formulate new treatment goals and find some goals that tools helped you develop are not appropriate for particular clients.
They are short, easy to use, and modifiable
Most useful tools are short, easy to use, and modifiable. If they are long and cumbersome, practitioners may not want to use them because they take time away from direct client contact. Useful tools are modifiable in the sense that when some items do not work, practitioners can modify them to fit their practice. It is better to modify them in consultation with other knowledgeable professionals in case you are missing something important that the tools provide.
They have good indices of internal consistency
Helpful tools have good indices of internal consistency, which is sometimes called reliability. The index should reach or come close to .90 in clinical assessment and evaluation tools. Some tools can have lower indices of reliability or none at all and still be useful if they have face validity and are useful in other ways. Cronbach’s alpha is the most common index of internal consistency, which gauged from a scale of 0-1. A good alpha and good face validity suggest a potentially useful tool.
They have good inter-rater reliabilities
When an instrument has an inter-rater reliability score, this means that two or more practitioners have completed the instrument on the same client or clients. If there are two raters, then the number of clients should be at least 15-20. If there are 15-20 raters or more, then the rating can be done on fewer clients. The higher the index, on a scale from 0-1, the more reliable the scale is. The closer to 1 the rating is, the more the raters have agreed. A scale with a high rating or one with a low rating may have poor face validity, and practitioners decide not to use it. High face validity and high inter-rater reliability are good indicators of potential usefulness.
An issue with inter-rater reliabilities, however, is that practitioners who fill out the instrument may have different perspectives, ideas, and training on the concepts that underlie the instruments. Raters, therefore, should understand the theory, research, and practice wisdom on which the tools are based. An excellent tool could receive a low inter-rater reliability score because the raters did not understand the concepts on which the tool is based.
They have adequate test-retest reliability (TRR)
Test-retest reliability arises when a group of practitioners fills out an instrument on a group of clients and days or weeks later fills out the same instrument on the same group of clients. The scores on the two different occasions are correlated. The indices that are closest to 1 are those that indicate the best TRRs. Test-retest reliabilities cannot be done if the clients are receiving services because any intervention could affect the second set of scores. Face validity in combination with the reliabilities already discussed suggests a potentially useful instrument.