Last week, I attended the American Evaluation Association's conference, and I was reminded of a common challenge and some tools to help. Often an organization or a funder wants a randomized control trial or other rigorous study of a program. Of course, studies that provide a comparison or control group, or some kind of counterfactual, are important. What is also important?--that these studies actually study "the program" compared to "not the program."
What do I mean by this? Well, imagine you are still designing a program when you start it. Imagine you have just started to implement it in a new place--maybe a place you don't know so well yet. So then, maybe in the first days or months there, you aren't able to actually deliver much of the program as intended. You are still working out kinks in your partnerships or training staff who are new to your model. If we compare your performance in that scenario to a control situation, we may not yet be measuring your program. If we see poor results, what have we learned? We haven't really learned that your program doesn't work, or which elements are most effective for whom, because there is so much noise in the way, and your program wasn't yet really your program.
Is the first time you ever rehearse a play with a new cast in a new location, a true reflection of the production's quality? If we watched you flub the lines, your cast miss cues, without costumes and lights, would we yet really be seeing the play as written? Wouldn't it be better to give you a little time and structure to rehearse so that we can compare your performance to another play we've seen?
So, I'm not saying a newer program can't be evaluated. But wouldn't it helpful to know, assess, and then establish some pre-conditions before spending time and resources on a study? Recently, a colleague from an evaluation firm I've worked with shared that she often gets approached to do a study, only to find the program isn't ready. The program may not yet have a logic model or theory of change, a defined intended duration of the program, data on participation, or a sense of the staff time it will take to work with the evaluators.
There are a few tools out there that can help. This evaluability assessment from the Corporation for National and Community Service (CNCS)/Social Innovation Fund (SIF), developed by Lily Zandniapour and Nicole Vicinanza is one example that I like and that you could modify. It helps you think about organizational readiness, program readiness, and evaluation readiness.
Have you seen or used similar tools? How are you getting ready? Can I help? Let me know.
Best,
Gretchen