My Two Cents on the RCT Debate

I’ve been doing a lot of thinking recently about my contribution to the debate on RCTs. Several weeks after the Evaluation Conclave in Kathmandu, I’m ready to give my two cents. First things first: A little context. RCT = Randomized Control Trial, an impact evaluation method that establishes “rigor” by using control and treatment group(s) to determine whether particular outcomes can be attributed to a particular program or intervention. A quick review of literature or participation in enough conferences, and one can see that RCTs are often presented as the “gold standard” in evaluation for their ability to show statistically significant differences in outcomes while controlling for various influencing factors. Sounds good, right? Certainly many students, practitioners and policymakers are seduced by its empirical and scientific nature. As with any subject in international development, however, it’s not so simple. Michael Quinn Patton’s keynote at the Evaluation Conclave presents a strong argument against the uncritical acceptance of RCTs as the method for showing impact. I urge you to watch it (and read his book on Developmental Evaluation while you’re at it!)

Now some in the evaluation field may think the RCT debate is “stale,” yet the sheer proliferation of donors and implementing organizations commissioning such evaluations proves that this is not the case. In fact, graduate schools across the country are churning out impact evaluators by the dozen. On the one hand, top schools can’t be blamed for teaching skills that are high in demand; their students will surely get jobs, and with high profile organizations at that. But they are producing far too many “development as usual” professionals who are hesitant to engage in critiques about the way development is done and about the way development projects are evaluated. Is evaluation just another manifestation of development being “done to” countries rather than “done with” countries? The trend towards RCTs surely seems to lead to this conclusion. Who are evaluations being produced for? And why? Local governments aren’t the ones begging for RCTs; the donors are asking and implementing organizations are producing! De facto policy can be made pretty quickly with enough money to incentivize it. Is there a time and a place for RCTs? Of course (Read here for a great post about various options for impact evaluations). RCTs in and of themselves are not “evil” as some opponents would suggest—they have strong merits in many cases (though sometimes questionable ethics when it comes to assigning beneficiaries to life changing programs!)

It comes down to balancing accountability with learning and research with evaluation. Donors must make smart investments, and organizations must be accountable for funds they’ve been awarded. But there is too much pressure to take learning out of the equation. Who are the end users of evaluations? What purpose(s) do they serve? We cannot remove context when years of research and experience show that context can make or break a project. If a randomized control trial experiment finds that increased school attendance in Honduras can be attributed to a specific education program, what are the implications? Will we attempt to “scale up” the project based on information that tells us little about why the program worked in a particular place and time? Can we use that evidence to justify a similar project in Cambodia? Tajikistan? Mozambique? I think we can do better in terms of evaluating program impact in context-specific ways that provide useful information for those on the ground. This is particularly important for those who may not find highly technical RCT results to be readily accessible, but who need to understand why programs succeed or fail. After all, if partnering with local governments and local NGOs can lead to more successful program implementation, it can also lead to more successful (and useful) program evaluation. But the evaluations should be designed according to terms agreed to by everyone involved. Given all available options, I’d be interested to see how many times an RCT would be universally selected.

As the blog post title suggests, this is a debate folks, so let me know where you fall in the “continuum” of opinions!

Advertisements

The Big Easy leads to Big Research Questions

New Orleans: A vibrant hub of jazz music, Cajun and Creole cuisine, and hundreds of international education scholars and practitioners. A city with so much to offer was a fitting choice for this year’s installment of the Comparative and International Education Society’s Annual Conference. Somehow I’ve managed to jog through the French Quarter, enjoy the views from a streetcar on St. Charles Avenue, and delight in a traditional jazz brunch at Arnaud’s all while keeping up to date on the latest trends in international educational research and practice (not an easy feat!)

This year, as last year, reading is a hot topic. This trend is shaped by USAID’s All Children Reading initiative. With the lofty goal of helping 100 million children learn to read (and read to learn) by 2015, implementing agencies and local governments are feeling real pressure, not only to meet beneficiary targets but to do so in a quality way. After all, putting books in children’s hands does not guarantee they will learn to read. Interventions must be complex while operating within intense time, resource, and political constraints. How will the international community meet these goals? Most importantly, how will we meet these goals while ensuring that children are reading and learning

The major takeaways: (1) We know a lot about what works, (2) We know some of what we have been trying does not work, and (3) We know there is a lot to learn.

Where there’s a need for learning, there’s a need for research. And there is no shortage of research questions!  Here are some of the thought-provoking questions researchers and evaluators are asking themselves in their quest to reach 100 million:

  • What is the role of early childhood education in supporting school-based literacy efforts? How can we ensure all children are ready to learn by the time they enter primary school?
  • What is the effect of community and parental involvement on the ability of children to learn to read? How can projects more effectively address what happens outside of school in order to achieve literacy results?
  • How can projects generate real buy-in from local governments and Ministries of Education so that the projects are actually owned—and sustained—after funding ends? How much failure to achieve results can be explained by the inability to generate true collaboration?
  • How can bilingual education be implemented more effectively? What is the effect of policies and projects that mandate instruction in a language that is not mature enough to be taught academically in its written form (and where teachers are not qualified to be teaching it)?
  • How can we provide more long-term interventions that can create the necessary change in teacher instructional practices? How can you address issues of teacher belief systems and motivation in order to achieve student learning?

This last set of questions strikes me as the most important. 100 million children can be hard to conceptualize, so let’s think on a smaller scale. If you were charged with helping 10 students learn to read, where would you start? I suspect you might say you’d find a good reading teacher. Or do research on what makes a good reading teacher so you could do the job yourself. Regardless, it’s quite hard to turn a non-reader into a reader without a good teacher.

I’m a firm believer in the importance of quality teachers. The international development field has done a nice job of investing in tools to assess student literacy results, but there is less information about how to measure change in instructional practice. And I’m not talking about using student tests to measure teacher quality. If we want to create better teachers, let’s start thinking more critically about the types of systems that support true change in teaching practice. New materials coupled with short-term trainings are simply not enough. Intensive teacher professional development urgently needs to be funded. And while we’re at it, let’s not forget that teachers come in all shapes and sizes. If less than an hour of the school day is spent on reading—and much less in many cases—we need to address the ability of other teachers—parents and community members—in helping our children learn to read.

This research agenda, of course, is a collaborative work in progress. What else would you add?

Learning Culture Roadmap

What is a journey without a roadmap?

Sure, not all those who wander are lost. And wandering can take you to some pretty spectacular destinations.  

 

Definitely not lost.

Definitely not lost.

But most evaluation folk think and speak in roadmaps. At the very least, we attempt to order—or describe the complexity of ordering—some otherwise quite disorderly phenomena. Like most blogs, this one is a written, verbal, and visual representation of the author’s thought processes. And mapping the mind is quite a task! After all, our thoughts are evolutionary and dynamic. This blog will show the evolution of my thoughts as they relate to evaluation, specifically in the field of international development and education. As such, I expect that my views and perspectives will change as I change. It is my hope that you, the reader, will help contribute to this change—to the evolution of the evaluation field and its professionals—by engaging, challenging, and contributing to content posted here. The journey, of course, will not be neat or linear, but it will be full of interesting questions.

Here’s a tentative roadmap of what to expect from Learning Culture.

  1. Knowledge: A collection of information on evaluation theory, approaches, and methods complemented by parallel information on education and development.
  2. Skills: A virtual toolbox of skills that have proved useful in my work and opportunities to learn technical skills from trusted sources.
  3. Practice: An exhibition of the trials and tribulations of putting evaluation into practice in various development contexts, plus lessons learned during that practice.
  4. Collaboration: A collaborative learning community connecting M&E and development professionals, with featured guest posts supporting the blog’s key focus areas.
  5. Reflection: An honest reflection of evaluation theory and practice—the good, the bad, and the ugly—from my own perspective and from that of others in the field.

I hope you’ll join me on this journey. Whether it be as a reader or as a contributor, we’re in this together!