|
May 31: Last day to save $75 on registration for 2012 Annual Conference in Boston More Info June 30: Board of Trustees nomination deadline More Info July 22-25: 2012 Summer Conference in Denver More Info |
 Taking Measure: Establish criteria that will raise the bar on big program evaluationsBy Robby ChampionJSD, Fall 2005 (Vol. 26, No. 4) Copyright, National Staff Development Council, 2005. All rights reserved. Many big organizations, including school districts, regional centers, state education offices, and educational foundations, are attempting to raise the bar on evaluations of professional learning programs that they sponsor. These program evaluation studies usually are contracted to outside evaluation companies to better ensure that quality evaluation practices are used, which can boost the final report credibility. Contracting out relieves those who work in the organization of the pressure of undertaking a time-consuming, technical task potentially fraught with political issues. The conundrum is that contracting out doesn't always work as the sponsoring organization intends, and disappointment is not uncommon. The all-important final report sometimes ends up as a bulky narrative that describes what happened in the program. The report may not be program evaluation. What can go wrong?When things go wrong in contracting an evaluation, the problem is not usually a solitary blunder but a series of missteps by those in charge. These issues are generally avoidable with some big-picture thinking to judge the evaluation design before signing the final contract and launching the effort. The "bad evaluation" scenario often starts to go wrong when the sponsoring organization waits to call evaluation experts until the professional development initiative is well into implementation. This misstep means many golden opportunities to gather pivotal data and document early decisions have passed. The evaluation's purpose may never be firmly agreed upon, or the organization hopes one evaluation will meet too many purposes. The initiative to be evaluated frequently is a collection of staff learning activities, but the activities do not constitute a real program. The underlying logic about how change will occur in the initiative is unstated or fuzzy. Evaluators find themselves chasing their tails to find the real identity of the program as they check out one activity after another. The evaluation timeline may be unworkable given the reality of schools today. Crowded with important testing days and other schoolwide events, evaluators may find that they are all but locked out of schools for extended periods of time. Limited resources for the whole evaluation effort often force reviewers to gather only superficial self-report data - surveys, anecdotal stories, focus group conversations, interviews with key players. Data such as observations of teachers using what they have learned, artifacts such as meeting records and action plans, and samples of student learning products may be beyond the budget of the study. Site visits may be so brief and infrequent that contextual differences that greatly influence how a program gets implemented in School A vs. School B vs. School C can get blurred. The list of potential blunders and missteps goes on. In short, just because a major evaluation has a budget does not guarantee that it will provide useful, credible data and quality program evaluation.How can you ensure success?A successful evaluation requires leaders to think of the big picture, exercise sound project management, and plan collaboratively. Organizations undertaking serious program evaluations must be prepared to cut the scope of the study to fit their budget and timeline. A modest study of the highest possible quality is preferable over one that studies every professional development program but uses shallow data or questionable methods. Leaders should:Appoint a designated project leader. This individual will steer the project and serve as the ombudsman. The designated leader's first task is to ensure that the process for selecting the study proposal is thorough, systematic, and fair. Organize a forum. Inside experts and stakeholders must have a chance to get in their two cents' worth about their expectations for the study. Discuss and agree on the effort's major purpose, the design approach for the study, the schedule for data collection, whether to use a sample rather than all program participants, specifications for the study milestones, and deliverables, including the final written reports and presentations. Obtain and examine actual final reports from projects done by all evaluators who vie for the job. Their past designs and written reports should offer useful insights into the kind of work at which each company excels. Use a set of criteria, a rubric, or a tool to delineate the ideal evaluation. (See table below.) The evaluation companies competing for the project should be informed of your criteria before they submit their proposals. To use criteria in the selection process, invite various staff or other stakeholders to examine the study proposals using the list of criteria above to rate each one. Ideally, the study proposal will meet all criteria. Perfection is not necessarily attainable in program evaluation, or even necessary, but a solid evaluation is definitely within reach. Aim for a program evaluation that raises the bar. Forge a partnership with the outside evaluators selected for the job and work side by side with them. Organizational leaders sometimes think they should steer clear of the evaluators and let them do their job unimpeded. That is true only to some extent. The project leader must keep close track of the project's progress - or lack of it. Attention to the process helps ensure that the evaluation gets done, that it meets the milestones, that the quality of the work meets or exceeds expectations, and that it comes in on time. An evaluation that comes in months after the due date, regardless of how well done it is, often ends up on the shelf gathering dust because important program decisions have already been made. Monitor the study expenditures carefully. Actual study costs often exceed early budget projections. The evaluators may find they need to cut corners to avoid an overrun. Sometimes these cuts work out fine, but in other instances, cutting corners means collecting less data or impacting quality in other ways. Debrief. After the evaluation study has been completed, take time to debrief thoroughly with key stakeholders. Reflecting with colleagues about the lessons learned can help ensure that the next program evaluation is even better. REFERENCEChampion, R. (2002, Summer). Choose the right data for the job. Journal of Staff Development, 23(3), 78-80.Checklist for an evaluation proposalThis tool can help organizations in the selection process and in judging the value of an evaluator's proposal to study a program initiative.
About the AuthorRobby Champion works as an independent consultant. You can contact her at Champion Consulting, 322 Mt. Tabor Road, Staunton, VA 24401, (540) 886-0655, e-mail: Robbychampion@aol.com. |
|||||||||||||||||||||||||||||||||||||||||||||||||