This question has nagged me for a while. It emerged to the forefront as I recently considered the most efficient way to carry out my responsibilities, spurred on by the 20-80 rule: 80% of our outcomes come from 20% of our efforts. So how do you evaluate on a budget? Should you do-it-yourself (DIY) or outsource? We’ll begin by considering some of the advantages of either option.
Advantages of the Do-It-Yourself Approach
Firsthand Knowledge of Issues
We are sometimes our own worst critics. But we may have more to offer than we’d imagine. Recently I had the privilege of working with stakeholders who have personally experienced the challenges their program addresses. I emerged from that meeting with a deep appreciation for the collective wisdom of program participants and those who work closest with the people who receive a non-profit’s services. It is also vital to fully coordinate evaluation efforts with existing program operations, especially to plan program evaluation during program planning stages.
Save Money
It goes without saying that you can streamline evaluation-related costs and increase efficiency by using in-house staff who are most familiar with the way a program operates.
Save Time
In-house staff already have relationships with program participants and thus need not go out of their way to build trust. Building trust takes time! Good evaluations are founded on relationships of trust with program participants. If they do not like or trust the person who is administering the evaluation, can we truly expect genuine and forthcoming answers? Also, staff can gather data (via surveys, interviews, etc) in the course of carrying out their responsibilities. Since they know the ins and outs of a program, program staff’s expertise is vital to the success of any program evaluation.
While in-house staff bring indispensable strengths to an evaluation, please also consider the advantages of using an external evaluator.
Advantages of Using an External Evaluator
Expertise
We’ve all probably had these thoughts at some point or the other:
“Evaluation is not rocket science, anyone can do it!”
“What is so hard about designing a survey, distributing it and analyzing the data? Even a high-school student can do it.”
This topic was recently broached within an online American Evaluation Association Thought Leader Discussion Group. The danger is that evaluation can seem a lot easier than it actually is. But if you look at Program Evaluator Job Announcements, you’ll notice specific qualifications. The following is a sample of what you may see for a junior level position:
- at least a Masters in Social Sciences or Public Health, Ph.D. preferred
- at least 1-2 years of experience conducting program evaluations
- strong communication and interpersonal skills
- experience with quantitative data analysis and with data analysis software, such as SPSS
The general consensus seems to include the above qualifications as well as engagement in professional continuing education in the field of program evaluation. (Before you stop reading this post and throw up your hands in despair because you do not have the budget to hire another person, please keep reading. we’ll get to possible solutions to this dilemma soon!)
There have been reams of papers and chapters written on program evaluation, specific models, quantitative and qualitative methods and survey design. Enough to make even the most experienced evaluator feel that there is always so much more to learn! Other evaluation colleagues often call attention to mistakes that well-intentioned novices have made, for example, in sharing evaluation results and data.
Sure, it is important to maintain a level of healthy skepticism: of course, any evaluator will want to make a case for why you should hire a professional to do the job right. But I’d encourage you to also consider how much risk you are willing to take if an evaluation is not done properly. What do you stand to lose? Think about all the ways that misrepresented data could hurt your program. Think about all the opportunities to grow and improve your program that may be lost through a botched evaluation.
Objectivity
Again from first hand experience, I’ve heard stakeholders closest to the program express appreciation for the objectivity and fresh perspective that an external evaluator (also known as an independent evaluator) brings to the table. Program participants may feel awkward being forthcoming with those actually delivering the services. Due to their existing relationships with program staff, participants may feel subconsciously pressured to give answers that they think staff want to hear.
Ability to Coordinate
Having a skilled evaluator coordinate an evaluation effort will spare you headaches and worry. This may be a better option over merely dividing up responsibilities among maxed-out staff or having to worry about coordinating an extra project on your own.
Possible Solutions
Alright, now we come to the simple solutions. Coordination implies that you do not have to hire the evaluator to do everything! This translates into cost savings. Carter McNamara, Ph.D. astutely applied the 20-80 rule to program evaluation in his article Basic Guide to Program Evaluation. Since 20% of the effort can produce 80% of the outcomes, a good option may be to contract with an external evaluator just for the 20% of effort that produces the 80% of outcomes. I agree with Dr. McNamara that the best responsibilities to outsource would be the design of the evaluation and surveys. I’d also add data analysis and reporting.
Data collection can be most time-consuming and expensive. Using program staff for these functions may be a good compromise, as long as measures are taken to encourage objectivity, for example, a volunteer placing surveys in sealed envelopes.
If you still find the notion of contracting with an independent evaluator daunting, consider the following solutions that I have observed other non-profits use. Once you have clearly defined the scope of a very feasible evaluation project, carefully recruit and choose a well-qualified:
- graduate student whose rates and schedule may be more flexible
- pro bono evaluator who may be trying to gain expertise in a new area or may just be interested in giving back
- a stay-at-home parent who may be willing to trade their program evaluation expertise for more flexible terms.
You may considering beginning your recruiting efforts at the American Evaluation’s Association’s Career Center or on AEA’s LinkedIn Group. Here is some further reading on using an external evaluator from the Centers for Disease Control and Prevention.
Evaluating on a budget doesn’t have to be an unattainable dream and neither does it have to be a do-it-yourself disaster. With creative and strategic solutions and careful planning, it can be a practical reality.
——————
For more resources, see our Library topic Nonprofit Capacity Building.
_______________________________________________________________________________________________
Priya Small has extensive experience in collaborative evaluation planning, instrument design, data collection, grant writing and facilitation. Contact her at priyasusansmall@gmail.com. Visit her website at http://www.priyasmall.wordpress.com. See her profile at http://www.linkedin.com/in/priyasmall/