Evaluation experts often define evaluation as a systematic endeavor. Recently I have been considering what this really means. How do we carry out a more systematic evaluation? How do we translate this into practice?
Aim for Consistency
Aim for consistency in data collection efforts. Data should be collected the same way every time. How can this be practically achieved? Write out instructions for completing survey questionnaires and share them out loud. Do this even though it might seem unnecessary. There are various ways individuals can interpret how to answer a question. For example, including the directions “please check off only one option” avoids potential problems such as some individuals only checking off the best response while others checking off multiple responses.
Also, use the same questions for the survey or semi-structured interview or focus group, each time. This helps with consistency in data analysis across various groups.
Aim for Replicability
Include detailed instructions for those who will be administering your survey with the goal of someone else being able to replicate your evaluation study. Though evaluation is not the same as research, aiming for replicability will make your evaluation efforts more consistent and systematic. I like to think of it as akin to leaving detailed instructions for a friend who will be caring for a temperamental pet or plant. Such a pet or a plant will thrive better on consistent care. So it is wise to attempt to replicate the same level of care you’d give the pet or plant by writing out a plan of care.
Sometimes stakeholders can feel that such instructions are insulting to their intelligence. So it helps to emphasize the need for consistency and to explain why you are doing things the way you have chosen to do them. Data that is collected inconsistently can result in difficulties in analyzing the data and having to exclude responses, which can further complicate analysis. This also decreases the validity of the data collection method—that is, we are not really measuring what we think we are measuring.
Involve Stakeholders in Every Stage of the Evaluation, especially Planning
How do we maintain consistency especially when others are involved in data collection? Involving key program stakeholders in planning the evaluation can increase consistency in data collection efforts. Brainstorm with them ways to collect data consistently. Provide an interactive training in data collection.
Draft a Written Plan for Data Collection
A written plan for data collection can help identify pitfalls ahead of time. It also provides a game plan to stick to each time. Once data collection has started, have regular meetings with program stakeholders or staff to discuss the data collection plan and how adherence to the plan can be maintained.
Pilot test Your Data Collection Method
Pilot testing your data collection method can help bring awareness of potential problems with your data collection tool. It also provides a good opportunity for program stakeholders such as clients to provide input about your data collection method and tool. A pilot testing survey includes questions such as:
- Were all the questions easy to understand?
- Were all the survey directions clear?
- Is there any other feedback you have about the survey process?
- How can we further improve this survey?