Feedback to Managers: A Guide to Reviewing and Selecting Multirater Instruments for Leadership Development

Jean Brittain Leslie

• Publisher: Center for Creative Leadership Press
• Year: 2013
• Print ISBN: 9781604911664
• eBook ISBN: 9781604918359
• Topic: Business
• Subtopic: Leadership
• Index: Business

Feedback is a rare commodity in day-to-day organizational life, but it is a key to ongoing effectiveness.One popular vehicle for getting feedback from one's boss, peers, subordinates, and customers is the multiple-perspective or 360-degree-feedback instrument. Whether part of a management-development course or used alone, this kind of instrument can enhance self-awareness by highlighting a leader's strengths and areas in need of further development.Selecting the right instrument from among the dozens that are available can be difficult, however.This new edition of Feedback to Managers, the fourth, updates and expands the popular 1998 edition.It guides the selection process with an in-depth analysis of 32 publicly available instruments that relate self-view to the views of others on multiple management or leadership domains. Each of the instrument reports includes descriptive information, a look at the research behind the instrument, and descriptions of support materials.

Section 1

Steps in the Evaluation Process and Findings from the Field

The steps in evaluating a 360-degree feedback instrument are laid out here sequentially. All steps are not equal in complexity or importance. I include a checklist of the steps, for your convenience, at the end of this book. There you will also find a glossary of many of the technical words used in this publication.

In the steps that follow, you will find a discussion of key points, and where applicable, a comparison of the instruments reviewed in Section 2. This section ends with conclusions about trends over time in the development and use of 360-degree instruments, the state of the art today, and directions for future research and development.

Step 1

Find Out What Is Available

The quantity of 360-degree instruments is increasing at a tremendous pace. You can expect that there are as many promising instruments under development as there are good instruments for sale. So your first task should be to learn what is out there in order to choose the best possible sample of instruments to review.

In the short run, a good way to familiarize yourself with what is available is to search social and behavioral science and business databases through your local library or online, including Mental Measurements Yearbook with Tests in Print (Buros Institute of Mental Measurements, 1938–2010). Use keywords such as 360-degree, multirater, multisource, management, leadership, feedback, survey, questionnaire, instrument, assessment, and evaluation. Over time, it may be useful as well to keep a file of the instrument brochures you obtain, because many of the directories are not published often enough to keep you updated on the very newest products.

Step 2

Collect a Complete Set of Materials

When you have identified several instruments you wish to evaluate, you need to obtain six pieces of information about each of them. You cannot make an informed decision using only a copy of the instrument or a promotional brochure. For each instrument you wish to consider, you should obtain the following:

1. A copy of the instrument itself. If the instrument has one survey for the individual to rate himself or herself and a separate survey for the others who will rate him or her, get both.
2. A sample feedback report (a representation of what the manager will receive after the instrument is scored). You can't tell what type of feedback your managers will actually receive by looking at the instrument they will fill out. The sample could be a complete report, or it could be part of a report such as an example of the feedback display in the technical or trainer's manual. Either type will do.
3. A technical manual or other publication that outlines in detail the developmental and psychometric research done on the instrument.
4. Information about any supporting materials that accompany the scored feedback, such as interpretive materials, development guides, goal-planning materials, and the like.
5. Information about price, scoring, and whatever certification or training may be required to purchase or use the instrument.
6. Information about the ways the instrument and subsequent material can be customized.

It is not at all unreasonable to request this quantity of information. Standards set by AERA, APA, and NCME as reflected in Standards for Educational and Psychological Testing (1999) require that this information be available upon request when an instrument is offered for sale.

In addition to seeking the recommended information, you should, through all the steps that follow, look for evidence of a commitment to continuous improvement on the part of each instrument's developer. This is especially true if an instrument has been around for a while. Research should always be in progress, because no instrument can ever be considered valid once and for all. Expect revisions in the scales over time; these are often made when additional validation studies have been completed. Expect revisions in the presentation of feedback as well; these are often made as the developer learns from the experience of those who have used an instrument. It is not uncommon for good instruments to have more than one copyright date, because even small revisions to content can cause changes in other areas, such as scale weightings or instrument norms.

Step 3

Look for a Good Fit with Your Intended Audience

In evaluating an instrument, you should begin by examining the target audience and scales and items on which feedback will be received. Are you comfortable with what it measures for your intended audience?

It is improbable that one instrument will meet the needs of all managers in an organization. Job demands differ somewhat by organizational level, and even at the same management level, skills that are needed for effectiveness may change over time. In addition, the scales on which managers are assessed should be in line with organizational visions for leadership. To the extent that these visions vary across organizations, it is also highly unlikely that one instrument will meet the needs of all kinds of organizations. Thus, in searching for an instrument to provide feedback to managers, a person is typically looking for one that will satisfy the needs of a particular group of managers in an organization with specific leadership or management needs.

Although nearly every 360-degree instrument has a statement of purpose describing the level of management it targets, there seems to be little relationship between management level and the domains of activity or behavior assessed. An instrument targeted toward all levels of management might not be right for middle managers in your organization because the capacities assessed are not in line with company-wide management-development goals. An instrument targeted toward higher levels might be right for your middle managers if the competencies assessed agree with your management-development goals.

There is a detailed discussion of scales in Step 6, but for now, you need to know that the scales are made up of several items on the instrument and represent the content or competencies on which managers will be evaluated. Each individual scale represents a slice of managerial work (for example, planning) or a single kind of competency (for example, decisiveness); as a whole the scales provide a portrait of leadership or managerial effectiveness. You should consider the following when looking at the scales:

• Is your organization wedded to a particular way of representing what it takes to be effective in your business, or do you have a particular model underlying management-development efforts?
• Does the range of scales fit with what you see as relevant competencies for managers in your target group?
• Does the number of scales seem reasonable?

If, in your judgment, an instrument does not have enough scales that seem relevant to your target group, or if it has too many that seem irrelevant, drop it from further consideration.

Step 3 Findings

A little over half of the reviewed instruments (54 percent) can be used by managers at all organizational levels (360 By Design, LWS, CLI, ESCI, ECI, EIV360, Everything DISC 363 for Leaders, LEA 360, LSI, Linking Leader Profile, MEPS, Management/Impact, Manager View 360, MLQ, Prospector, SLP, SMP, SYMLOG, and Types of Work Profile). In practice, however, some of these instruments are often used by a narrower audience. Benchmarks, GELI, LBAII, LAQ, LVI, L/I, and VOICES are directed to middle- to senior-level managers. Only nine are more specific in defining their target audience: EISA, Leadership Competencies for Managers, Leadership Navigator for Corporate Leaders, LPI, and Social Style and Enhanced Versatility Index are advertised primarily for use by middle-level managers, whereas Executive Dimensions, EXEC, The Leadership Circle Profile, and Tilt 360 Leadership Predictor are meant for executives.

While many of the instruments are targeted toward a broad or multiple-level audience, those intended for a more focused group differ little in the behaviors they assess from those claiming a broader audience. For example, two instruments assessing basic management skills and behaviors (Emotional Intelligence View 360 and EISA) are very similar in terms of domains covered (achievement, influencing, self-control) but differ in terms of their target audience.

It may be that the choice of target audience is sometimes more of a marketing decision than a decision made based on research or development criteria. Or a final target audience may evolve during the process of instrument development and validation. For example, MLQ, which was developed from a theoretical model of transformational and transactional leadership, and which initially was thought to be applicable only to senior levels, was found to apply equally well at lower levels of management.

The lack of a clear relationship between the target audience and the domains assessed also may result from a difference in philosophy behind the different instruments. Some leadership instruments tend to treat leadership more as a demand of upper-level jobs, whereas others treat it as an activity or characteristic that can occur at any level. The latter type target a broader group, encompassing all managerial levels (for example, LWS and MLQ) or even adults in any occupation (CLI, LSI, and SYMLOG).

Finally, some instruments directed at all or multiple levels are focused on skills and behaviors needed in day-to-day operations, whereas others focus on skills needed to create or weather change in the organization. Two of Clark Wilson's instruments (SMP and SLP) exemplify this distinction. Although both are targeted toward multiple levels, SLP focuses more on leadership through change, whereas SMP assesses dimensions of behavior relevant to effective management of day-to-day operations. MLQ incorporates elements of both day-to-day and change skills, calling both leadership—one transformational and the other transactional.

A key to choosing among instruments is to match the instruments' domains with the audience with whom the instrument is intended to be used. As I have pointed out, there seems to be little relationship between management level and the domains of activity or behavior assessed. Also important is matching the norm group, if any, to which managers will be compared. If, for example, the norm group is composed of senior-level managers, whose skills are likely to be more highly developed, the scores of middle managers will probably appear worse than they would if they were compared to managers similar to themselves. All instruments reviewed that use norms in feedback have a norm group that matches their target audience.

Target Audience

Step 4

Determine Whether Customization Is Needed

The bulk of the instruments reviewed are considered standard, or off-the-shelf. These instruments lend themselves to easier reliability and validity studies because massive numbers of people have taken them. Items and scales are always the same for all people who use them, so data can be accumulated on which to do the studies. There are times, however, when competencies you want to assess are not included. In this case, you can consider customization options. Customization options can range from adding a few questions (360 By Design, CLI, Leadership Competencies for Managers, LPI, LVI, SMP, Tilt 360 Leadership Predictor) to allowing customers to select the scales to build their own unique instruments (360 By Design, EVI360, VOICES). More and more vendors are offering customization (for example, custom reports, rater labels, and norms) because it increases impact, organizational participation, and buy-in. The degree to which customization reduces reliability and validity of an i...

Table of contents

• Cover Page
• Title Page
• Copyright
• Contents
• Preface
• Acknowledgments
• Introduction
• Purpose of This Book
• Methods
• Section 1: Steps in the Evaluation Process and Findings from the Field
• Section 2: Instrument Summaries
• In Conclusion
• Instrument Evaluation Checklist
• Glossary of Terms
• Permissions