Choosing 360: A Guide to Evaluating Multi-Rater Feedback Instruments for Management Developmen

Van Velsor, Leslie, Fleenor

• Verlag: Center for Creative Leadership Press
• Jahr: 1997
• ISBN:
  9781604916720
• Thema: Business
• Thema: Leadership

Feedback is a rare commodity in organizational life, but it is key to managerial effectiveness. One increasingly popular vehicle for getting feedback from one's boss, peers, and subordinates is the multiple-perspective, or 360-degree, feedback instrument. Use of such an instrument can enhance self-confidence by highlighting individual strengths and can facilitate greater self-awareness by pointing out areas in need of further development. Because of the availability of so many feedback instruments, finding the best instruments for an organization's needs is difficult. This book presents a step-by-step process that shows how to evaluate multiple-feedback instruments intended for management development. The steps take you through such issues as instrument development, validity and reliability, feedback display, scoring strategies, and cost.

STEP 1:
FIND OUT WHAT IS AVAILABLE

The availability of 360-degree-feedback instruments is increasing at a tremendous pace. You can expect that there are as many promising instruments under development as there are good instruments for sale. So your first task should be to gain some knowledge of what is out there in order to choose the best possible sample of instruments to review.

In the short run, a good way to familiarize yourself with what is available is to search one of several guides that categorize and review instruments. Feedback to Managers, Volume II (Van Velsor & Leslie, 1991) is one such guide. It provides basic descriptive and technical data on 360-degree-feedback instruments available for use for management development. Other guides include Mental Measurements Yearbook (Conoley & Impara, 1995); Business and Industry Testing: Current Practices and Test Reviews (Hogan & Hogan, 1990); Psychware Sourcebook (Krug, 1993); and Tests: A Comprehensive Reference for Assessment in Psychology, Education and Business (Sweetland & Keyser, 1990). These can usually be found in the reference section of libraries. Over time, it may be useful as well to keep a file of the instrument brochures you obtain, because many of the directories are not published often enough to keep you updated on the very newest products.

STEP 2:
COLLECT A COMPLETE SET OF MATERIALS

When you have identified several instruments you wish to evaluate, you need to obtain five pieces of information about each of them. You cannot make an informed decision using only a copy of the instrument or a promotional brochure.

Specifically, for each instrument you wish to consider, you should obtain the following:

•  A copy of the instrument itself. If the instrument has one form for the individual to rate himself or herself and a separate form for the others who will rate him or her, get both.

•  A sample feedback report (a representation of what the manager will receive after the instrument is scored). You can’t tell what type of feedback your managers will actually receive by looking at the instrument they will fill out. The sample could be a complete report, or it could be part of a report such as an example of the feedback display in the technical or trainer’s manual. Either type will do.

•  A technical manual or other publication that outlines in detail the developmental and psychometric research done on the instrument.

•  Information about any supporting materials that accompany the scored feedback, such as interpretive materials, development guides, goal-planning materials, and the like.

•  Information about price, scoring, and whatever certification or training may be required to purchase or use the instrument.

It is not at all unreasonable to request this quantity of information. American Psychological Association guidelines (APA, 1985) require that this information be available upon request when an instrument is offered for sale.

In addition to seeking the recommended information, you should, through all the steps that follow, look for evidence of a commitment to continuous improvement on the part of each instrument’s developer. This is especially true if an instrument has been around for awhile. As we will discuss in the section on validity, research should always be in progress, because no instrument can ever be considered valid once and for all. Expect revisions in the scales over time; these are often made when additional validation studies have been completed. Expect revisions in the presentation of feedback as well; these are often made as the developer learns from the experience of those who have used an instrument. It is not uncommon for good instruments to have more than one copyright date, because even small revisions to content can cause changes in other areas, such as scale weightings or instrument norms.

STEP 3:
COMPARE YOUR INTENDED USE TO INSTRUMENT CHARACTERISTICS

It is improbable that one instrument will meet the needs of all managers in an organization. Job demands differ somewhat by organizational level, and even at the same management level, skills that are needed for effectiveness may change over time. In addition, the dimensions on which managers are assessed should be in line with organizational visions for leadership. To the extent that these visions vary across organizations, it is also highly unlikely that one instrument will meet the needs of all kinds of organizations. Thus, in searching for an instrument to provide feedback to managers, a person is typically looking for one that will satisfy the needs of a particular group of managers in an organization with specific leadership or management needs.

Although nearly every 360-degree-feedback instrument has a statement of purpose describing the level of management it targets, there seems to be little relationship between management level and the domains of activity or behavior assessed. An instrument targeted toward all levels of management might not be right for middle managers in your organization because the capacities assessed are not in line with company-wide management-development goals. An instrument targeted toward higher levels might be right for your middle managers if the competencies assessed agree with your management-development goals.

More important than considering the advertised audience is discovering the norm group, if any, to which managers will be compared. By norm group, we mean the group of managers whose scores are stored in the vendor’s database and are output as the comparison group on every individual feedback report. If the norm group is comprised of senior-level managers, whose skills are likely to be more highly developed, the scores of middle managers will probably appear worse than they would if they were compared to managers similar to themselves. Therefore, look for instruments that have been normed on a sample similar to your target managers; consider level, organization type, and demographics (for example, ethnicity and gender).

But be forewarned: The feedback instruments we are concerned with here have been developed for use in management-development efforts, either in the classroom or in individual feedback settings. These are instruments that have not been developed or tested for other purposes—such as making selection or promotion decisions.

STEP 4:
EXAMINE THE FEEDBACK SCALES

In evaluating individual instruments, you should begin by examining the scales on which feedback will be received. Are you comfortable with what it measures?

There is a detailed discussion of scales in step 6, but what you need to know at this point is that the scales are made up of several items on the instrument and represent the content or competencies on which managers will be evaluated. Each individual scale represents a slice of managerial work (for example, planning) or a single kind of competency (for example, decisiveness); as a whole the scales provide a portrait of leadership or managerial effectiveness. Using the sample feedback you have obtained, you should consider the following when looking at the scales:

•  Is your organization wedded to a particular way of representing what it takes to be effective in your business or do you have a particular model underlying management-development efforts?

•  Does the range of scales fit with what you see as relevant competencies for managers in your target group?

•  Does the number of scales seem reasonable?

•  If, in your judgment, an instrument does not have enough scales that seem relevant to your target group, or if it has too many that seem irrelevant, drop it from further consideration.

STEP 5:
FAMILIARIZE YOURSELF WITH THE INSTRUMENT-DEVELOPMENT PROCESS

In order to know how to identify quality instruments, you must understand the basics of sound instrument development.

The development process can be seen as occurring in four stages:

•  developing instrument items and feedback scales,

•  assessing reliability and validity,

•  designing the feedback display, and

•  creating supporting materials.

At each stage different issues are being addressed.

When items and scales (fully defined in step 6 below) are being developed, the author must identify, as much as possible, the full range of behaviors or skills that he or she believes represents management or leadership competency. Another question at this stage is whether items of behavior or competency cluster in groups that are internally consistent, distinct from each other, and useful for feedback purposes.

To assess reliability, the author of an instrument must consider whether the measurement of these skills or competencies is stable in a variety of ways. To assess validity, the author must determine whether the scales really measure the dimensions they were intended to measure and whether they are related to effectiveness as a manager or leader. Because 360-degree-feedback instruments are primarily intended for individual development, the question of whether the areas assessed can be developed also must be considered.

When designing feedback, the author should try to maximize the manager’s understanding of the data to enhance its impact. In creating supporting materials, the aim of the author is to help the feedback recipient gain deeper understanding of the theory or research behind the instrument and thereby enhance the ability to interpret and work with the data. Your task as an evaluator is to assess the work completed in each of these four stages and balance what you find against the needs of your target group.

STEP 6:
LEARN HOW ITEMS AND FEEDBACK SCALES WERE DEVELOPED

Instruments that assess managerial competence or leadership effectiveness are dealing with complicated phenomena. These phenomena cannot be adequately represented by a single behavior or characteristic because they are comprised of many closely related behaviors and skills. To adequately measure these ...