Collaborative Evaluations

Step-by-Step, Second Edition

Liliana Rodríguez-Campos, Rigoberto Rincones-Gómez

• Publisher: Stanford Business Books
• Year: 2012
• ISBN:
  9780804784856
• Edition: 2
• Topic: Business
• Subtopic: Processo decisionale
• Index: Business

Collaborative Evaluations: Step-by-Step, Second Edition is a comprehensive guide for evaluators who aim to master collaborative practice. Liliana Rodríguez-Campos and Rigoberto Rincones-Gómez present their Model for Collaborative Evaluations (MCE) with its six major components: identify the situation, clarify the expectations, establish a collective commitment, ensure open communication, encourage effective practices, and follow specific guidelines.

Fully updated to reflect the state-of-the-art in the field, each core chapter addresses one component of the model, providing step-by-step guidance, as well as helpful tips for successful application. To further demonstrate the utility of the MCE, this new edition includes recurring vignettes about several evaluators and clients, illustrating frequent questions and specific challenges that arise when evaluators take a collaborative approach. Drawing on a wide range of collaborative evaluations conducted in the business, nonprofit, and education sectors, this precise and easy-to-understand guide is ideal for students and practitioners who want to use its tools immediately.

Chapter 1

Identify the Situation

The situation is a combination of formal and informal circumstances determined by the relationships that surround and sustain the collaborative evaluation. This component is very important because the success of the evaluation will depend, to a large degree, on how well the situation is understood. Although you and the collaboration members (CMs) may be able to influence the environment in some way, keep in mind that you do not have total control over it. Understanding the situation is the foundation for all the evaluation decisions and prepares you to anticipate opportunities and threats throughout the collaborative process.

The way you perceive the situation will determine the approach you will take to achieve the desired results. Those evaluation results will be based on specific situational characteristics, such as resources (for instance, people or time) and a particular setting (for example, physical, social, political, economic, geographical, or demographic). Rather than simply accepting the perception that the client has of the situation, you should gain a broad sense of the situation on your own. In addition, instead of just stopping with what the client wants, pursue what the client needs, because the difference between “want” and “need” is your value added (Weiss, 2009).

An early indication of the situation with regard to the potential constraints and benefits needed to support the collaborative evaluation (such as funds, staff, materials, scope, and context) helps you better manage it and be prepared to overcome barriers. It also helps to determine strategies to deal with any concerns raised and to develop a beneficial evaluation plan. To gain a better understanding about the situation surrounding the evaluation you may conduct observations of the evaluand and stakeholders in their day-to-day activities. Always have multiple inside and outside sources for gathering helpful information, in order to understand the evaluand culture (for example, interview key stakeholders or review documents and reports).

The situation sets the foundation for everything that follows in the collaborative evaluation. Among other resources, the time needed to identify the situation varies depending on the complexity of each evaluation, which also leads to the amount of effort invested in it (for example, the number of meetings needed). Because people are going to collaborate within the evaluation, it is also necessary to determine what level of collaboration will be required depending on the situation. The better you understand the situation and the more fully you analyze it, the easier your collaborative evaluation will be (and the application of the remaining components of the Model for Collaborative Evaluations [MCE]).

This chapter provides you with step-by-step suggestions on how to identify information in order to understand the current nature of the evaluand, its surroundings, and the scope for the collaborative evaluation. In other words, we show you how to carefully identify the situation. In addition, we present information on how to prioritize the activities toward which you need to focus your resources in order to accomplish clear evaluation results. Thus the MCE component presented in this chapter is divided into the following subcomponents: (a) identify stakeholders, (b) identify logic model elements, (c) identify potential SWOTs, (d) identify the evaluation scope, and (e) identify critical evaluation activities (see Figure I.2).

IDENTIFY STAKEHOLDERS

A stakeholder is a person who is interested in the collaborative evaluation because he or she may be directly or indirectly affected by its results (see Figure 1.1). Identify early the individuals who have at least some interest or stake in the evaluation and how their roles can help to accomplish the overall goal. The audience is a person, or group of persons, who receives the evaluation results (for example, a written or an oral evaluation report).

There are five types of stakeholders who are associated with almost any evaluand: (a) policymakers, such as governing board members; (b) administrators or managers, those who direct and administer the evaluand; (c) practitioners, those who operate the evaluand; (d) primary consumers, those who use the evaluation results, such as evaluation clients; and (e) secondary consumers, those who are affected by what happens to primary consumers, such as community groups (Fitzpatrick, Sanders, & Worthen, 2011). After you have a comprehensive knowledge of the stakeholders, then you are ready to identify the CMs.

Figure 1.1 From Stakeholders to CMs

Identifying the stakeholders is critical for the collaborative evaluation because their specific needs must be understood in order for them to be addressed in a useful way. Therefore, make a favorable first impression to develop strong relationships from the very beginning. From your first meeting, make direct eye contact, display excellent manners, and use appropriate body language in order to develop effective rapport.

Your goal is to understand who the stakeholders are, what they want and need, and how the evaluand is being affected by them. As a result, you can identify and select specific stakeholders to become CMs who will be involved in depth in the collaborative effort (see Figure 1.1). Of course, always consider the views of the broader range of stakeholders who also collaborate in a less active way.

Identify General Stakeholders

As we mentioned in the Introduction, the MCE is a systematic approach with feedback mechanisms that allow you to come back to this subcomponent and increase (or decrease) the number of identified stakeholders as needed. You may have to seek new CMs, in response to changing needs (for example, a negative climate due to an inappropriate cross-section of members). We suggest the following steps to identify general stakeholders:

1. Be clear that your responsibility is to meet, or even better, exceed your client’s expectations while ensuring that particular interests of all relevant stakeholders are considered throughout a balanced collaborative evaluation.

2. Learn as much as you can about your evaluation client, and later about other potential stakeholders, in order to establish a beneficial relationship. For example, read from the evaluand’s or client’s website and other sources of information.

3. Think carefully and write down all the questions you need the client, and later other potential stakeholders, to answer (for instance, who needs to be informed, who is to receive the evaluation results, and whose collaboration is crucial).

4. Seek clarification (with a courteous attitude) to your questions when meeting with the client, and later with other potential stakeholders, as feasible. For example, if doubts arise after a meeting, then seek to clarify all those doubts.

5. Listen rather than just making statements or educating about the evaluation process (avoid interrupting). For example, wait silently and patiently during pauses, because generally the most important information follows them.

6. Identify diverse points of view (such as the strongest supporters and opponents) in order to diffuse any resistance toward the collaborative evaluation and ensure that everyone’s contributions will be adequately considered.

7. Make a preliminary list with the client’s assistance (and others as applicable) of the key stakeholders affected by the evaluation. For example, distinguish who would gain or lose the most from the evaluation under different scenarios.

8. Invite your client to review and add or delete names from the preliminary stakeholder list. By doing this, you can have a better idea of who the key stakeholders really are and their possible contributions to the evaluation as future CMs.

9. Gather feedback on a regular basis using a previously agreed-upon system (including meetings, emails, and surveys as feasible). This will help in monitoring the stakeholders’ identification process and making updates as needed.

10. Provide the client and other stakeholders as appropriate with a summary report of this experience (for example, an updated list of the stakeholders) as a baseline for identifying the situation of the collaborative evaluation.

Identify Collaboration Members

Now that you have a list of the stakeholders, identify the CMs who will work jointly with you to help with particular evaluation tasks in order to achieve the evaluation vision (see Chapter 3 on how to establish a shared evaluation vision). The CMs should want to be members, be able to fulfill specific evaluation requirements, and work well together. Also, the client must agree with the selected CMs and allow, as applicable, time from their regular duties for them to participate.

This step is critical for the collaborative evaluation process because the CMs can help ensure their own (and other people’s) involvement and consequent use of the evaluation results. In other words, the use of the evaluation results will be improved by involving stakeholders in the evaluation process. Specifically, their buy-in proves to be very helpful when evaluation recommendations are being implemented. We suggest the following steps to identify the CMs:

1. Make a preliminary list of essential characteristics that are desirable in the CMs. With this goal in mind, you will be able to later match specific stakeholders to this list while being prepared to be flexible in the selection process.

2. Agree with your client, and other key stakeholders as feasible, which stakeholders from the list can have the most active involvement in the collaborative evaluation, becoming potential CMs (including the client or not, as appropriate).

3. Create, with the potential CMs’ help, a calendar that shows their availability during the different evaluation stages and how they see themselves fitting into the collaboration. For example, determine who will be available and when.

4. Learn about each potential CM in terms of their individual characteristics (such as strengths and weaknesses) before making a decision on which individuals you are going to officially invite to become CMs of the evaluation.

5. Identify which of the potential CMs may require special training to enhance their evaluation skills. Sometimes you may not find enough qualified CMs, so they will need appropriate training to build specific evaluation skills.

6. Agree on the final CMs with the client, once you have met with all the key stakeholders, as feasible. Be sure that major stakeholders are represented (that is, specific characteristics) across the main areas affected by the evaluation.

7. Ensure that you and the client are making a fair decision on the CMs’ selection (for example, that you have all the relevant facts to make a decision) and on how to match their skills and abilities to the expectations of the collaborative evaluation.

8. Determine if the selected CMs agree to formally collaborate as such in the evaluation, and get started with those who are willing to do it. In other words, find who is genuinely interested to collaborate in the evaluation efforts.

9. Consider the morale of the non-selected stakeholders, who may feel disappointed with the selection. For example, ensure open communication with them, because they can provide important feedback for the collaborative evaluation.

10. Gather feedback on a regular basis using a previously agreed-upon system, such as meetings, and summarize it in a written format (including an updated list of the CMs) so it is available to each CM and other stakeholders as appropriate.

Carefully select CMs with varied characteristics to facilitate involvement within the collaborative evaluation and achievement of its vision. Through the years, we have noticed that differences in education, authority, or other personal characteristics can result in lack of involvement by those who are at the end of the scale (typically the lower end) in regard to those characteristics. Hence, keep in mind that the same evaluation may have a different set of CMs for each of its main needs.

For large, more complex evaluations you may need to create several groups of CMs, so that greater diversity of expertise can be represented. Avoid creating groups larger than six members, because then people tend to feel less individual responsibility for the evaluation results. Be aware that sometimes there may be some poor fits among the CMs selected even though you have been very careful with the identification process. Learn from this experience and continuously monitor the CMs’ commitment, performance, and satisfaction by obtaining accurate information (see Chapter 4 on how to ensure immediate feedback).

Identify Stakeholders

Eliot: I am currently working on an external evaluation in a private company and have already met with several stakeholders. However, I just talked with Mr. Smith, who has worked for over twenty years for the program I am evaluating. He says he doesn’t consider himself a “stakeholder” and therefore he is not planning to go to any of the meetings. Given his time with the program...