Insights on Program Evaluation

Having just completed a Program Evaluation course at Walden University, I reflect here on the experience.

I didn’t realize that the process of creating and presenting my program evaluation plan was central to my learning until it was done. It was in pulling the disparate pieces of the plan together into a cohesive whole that I found meaning. Only then did I realize that had I not thoroughly explored the program context; not stretched myself to identify primary and secondary stakeholders and their interests, needs, and biases; not named my values and committed myself to them; not reflected on my own biases; not considered the impact of my report, my evaluation plan, which may have held together well enough to bring me to the last week of the course, would ultimately have failed the final analysis.

Program evaluation can inform what needs to change and form the basis of a change management plan, but first, it must fit the program context. It is incumbent on the evaluator to consider how contextual factors may inform the selection of an evaluation model. Program evaluation is inherently a political process, and an evaluator who ignores, avoids, or mismanages the political realities of evaluation limits the effectiveness and usefulness of the process (Fitzpatrick, Sanders, & Worthen, 2010.

This experience has shown me that working with stakeholders is one of the most challenging aspects of program evaluation. The important work of planning a program evaluation can be upset by stakeholder conflict, politics, bias, and unexpected manifestations of organizational culture. And yet, as an evaluator, I have a professional obligation to find my way to promote meaningful evaluation and the application of evaluation results by stakeholders (Fitzpatrick et al., 2010).

Technology can facilitate communication with stakeholders and simplify the processes of data collection, data management, and research (Laureate Education, Inc., n.d.), but it is only a tool; the evaluator must provide the raw material and craft the work. In every phase of evaluation it is incumbent on the evaluator to uphold the priority of justice (Schweigert, 2007); to mine the program context for cultural cues, gaps in understanding, potential bias; and feasibility; and to demonstrate and promote respect for stakeholders and the evaluation process. Bias is the weed that pervades the evaluation process, from the evaluator’s preference for a particular approach or data collection design to overt or covert liking of some stakeholders more than others, finding some steps of the evaluation process more interesting, more compelling, or more exhausting than others. The presence of bias is a given. Evaluators must be frankly self-reflective about their role in the evaluation process and circumspect about client requests, so as to minimize the potential for bias and ethical compromise (Fitzpatrick et al., 2010).

At some point, situational circumstance requires evaluators to make interpretations and best guesses (Schweigert, 2007), which are subject to bias and ethical compromise. I carry with me from this course Sieber’s (1980) conclusion that “being ethical in program evaluation is a process of growth in understanding, perception, and creative problem-solving ability that respects the interests of individuals and of society” (p. 53).

Sally Bacchetta
 
References

American Evaluation Association, 2004. Guiding principles. Retrieved from www.eval.org/Publications/Guiding Principles.asp.

Fetterman, D. (2001). The transformation of evaluation into a collaboration: A vision of evaluation in the 21st century. American Journal of Evaluation, 22(3), 381–384. Retrieved from the Education Research Complete database

Fitzpatrick, J., Sanders, J., & Worthen, B. (2010). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Boston, MA: Pearson

Laureate Education, Inc. (Producer). (2009). Formative and Summative Evaluation. [DVD] United States.

Schweigert, F. J. (2007). The priority of justice: A framework approach to ethics in program evaluation. Evaluation and Program Planning, 30(4), 394–399.

Sieber, J. E. (1980). Being ethical: Professional and personal decisions in program evaluation. In R.E. Perloff & E. Perloff (Eds.), Values, ethics, and standards in evaluation. New Directions for Program Evaluation, No. 7, 51-61. San Francisco: Jossey-Bass.

Worthen, B. (2001). Whither evaluation? That all depends. American Journal of Evaluation, 22(3), 409–416. Retrieved from the Education Research Complete database

The Political Nature of Program Evaluation

Program evaluation is a political process. An evaluator who ignores, avoids, or mismanages the political realities of evaluation limits the effectiveness and usefulness of the process (Fitzpatrick, Sanders, & Worthen, 2010). Ethical complexities wind in and among the more overt political features of evaluation such as financial support, stakeholder allegiance, and social impact. Morris and Cohn (1993) detail several ways in which stakeholders may seek to influence evaluation outcomes, and Fitzpatrick et al. (2010) caution that evaluators also need to be aware of their own potential to taint the evaluative process.

If we accept that evaluation is political (and, therefore, ripe for ethical complication), then we must ask how best to balance the objectivity required in a program evaluation with the political interests of stakeholders. We must ask, “What ethical standards and values need to be emphasized in program evaluation?”

The American Evaluation Association’s (AEA) Program Evaluation Standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011) and Guiding Principles for Evaluators (American Evaluation Association, 2004) provide a broad, somewhat obvious, framework for ethical conduct.

Fitzpatrick et al. (2010) are more specific, encouraging evaluators to be both self-reflective about their role in the evaluation process and circumspect about client requests, so as to minimize the potential for bias and ethical compromise: “…the client may be asking for what the client perceives as editing changes, but the evaluator sees as watering down the clarity or strength of the judgments made” (p. 81). And Schweigert (2007) roots evaluator responsibility in the notion of justice – public, procedural, and distributive.

From this we can extract answers to the question “What ethical standards and values need to be emphasized in program evaluation?” 

Ethical standards:

• Those detailed in the AEA’s and other professionally recognized codes of conduct.

Values:

• Commitment to truth – what Schweigert (2007) calls the priority of justice
• Cultural sensitivity
• Respect (for stakeholders, ourselves, and the evaluation process)

It seems that no professional code nor personal charter can do the whole job. No matter how pointed the professional standards, situational circumstance requires evaluators to make interpretations and best guesses (Schweigert, 2007), which are subject to bias and ethical compromise, as Weiss (2006) lays bare any illusions we may have that we are above or beyond the snare of bias and ethical confusion: “You never start from scratch. We pick up the ideas that are congenial to our own perspective. Therefore, people pick up this thought or that interpretation of a research report that fits with what they know or what they want to do” (p. 480).

I have thought about this a lot over the past few days, returning again and again to Sieber’s (1980) conclusion that “being ethical in program evaluation is a process of growth in understanding, perception, and creative problem-solving ability that respects the interests of individuals and of society” (p. 53). 

Sally Bacchetta

References
 

American Evaluation Association, 2004. Guiding principles. Retrieved from www.eval.org/Publications/Guiding Principles.asp.

Fitzpatrick, J., Sanders, J., & Worthen, B. (2010). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Boston, MA: Pearson

Morris, M., & Cohn, R. (1993). Program evaluators and ethical challenges: A national survey. Evaluation Review, 17, 621-642.

Schweigert, F. J. (2007). The priority of justice: A framework approach to ethics in program evaluation. Evaluation and Program Planning, 30(4), 394–399.

Sieber, J. E. (1980). Being ethical: Professional and personal decisions in program evaluation. In R.E. Perloff & E. Perloff (Eds.), Values, ethics, and standards in evaluation. New Directions for Program Evaluation, No. 7, 51-61. San Francisco: Jossey-Bass. 

Weiss, C. H., & Mark, M. M. (2006). The oral history of evaluation Part IV: The professional evolution of Carol Weiss. American Journal of Evaluation, 27(4), p. 474-483.