November 29, 2017
From RCT to Policy
Scaling up RCTs involves countless challenges. One NGO overcame all of them.
A new approach to education is helping millions of students in India.
Fotoember/Bigstock
Schoolchildren in India were falling behind. Even worse, they did not have a good chance of catching up; teachers were expected to teach the grade-level curriculum — even if many of their students needed more basic instruction.
Pratham, an Indian NGO, had a solution: for part of the school day or year, students should be grouped by what they know rather than by their ages. This “teaching at the right level” approach seemed promising, and so Pratham partnered with researchers to evaluate the program. The researchers conducted a randomized controlled trial (RCT), a highly credible tool used across the social sciences, and found that the Pratham methodology was working.
The organization realized that the program could help millions of schoolchildren if it spread across the country. Unfortunately, moving from one local success to a national initiative was not that easy. Scaling up took ten years, several rounds of RCTs, and countless adjustments to the program. Of course, this timeline and process are not unique to Pratham. When moving from any proof of concept to policy, commented Esther Duflo (one of the researchers), “You have to be patient, and you also have to be persistent.”
In a new paper in the fall issue of the Journal of Economic Perspectives, authors Abhijit Banerjee, Rukmini Banerji, James Berry, Esther Duflo, Harini Kannan, Shobhini Mukerji, Marc Shotland, and Michael Walton discuss the challenges of scaling up RCTs and how to overcome them. The challenges include corruption, changing costs of implementation, and context dependence. These, among other issues, are further described in the infographic below.
(Research highlight continues below infographic)
Adapted from Banerjee et al. (2017)
For Pratham, the first obstacle was financial viability. When moving their program to rural North India, they switched from using paid teachers to using trained volunteers who followed a more structured curriculum. The methodology was still effective, but the new context introduced other challenges. The children who most needed the program did not show up to the sessions, and volunteers had a substantial attrition rate.
Nevertheless, state governments began to integrate the Pratham approach into public schools. The program existed in several versions, including a one-month summer camp, a version that distributed Pratham materials, a version that trained teachers in the “teaching at the right level” method and distributed materials, and a version that included training, materials, and volunteer support in the classroom. The next series of RCTs tested how well each of these worked.
The training and materials did not have an impact, and the volunteers only made a difference when they held sessions outside of the school day. When they provided support in a traditional classroom, they ended up teaching the grade-level curriculum alongside the teachers rather than incorporating the Pratham methodology. And, even though the government teachers did not change their pedagogy during the year, they did implement the program during the summer camps.
The researchers realized that parents and school administrations viewed finishing the grade-level curriculum as a priority. Teachers faced pressure to get through the curriculum, leaving them without the time to implement the Pratham program. But, when instructors — whether volunteers or teachers — were given a time dedicated to the Pratham program, they could successfully use the new methods.
Building on these insights, the next RCTs focused on creating time for the Pratham program and signaling the schools’ support for the new methods. The first RCT added a Pratham-specific hour to the school day, and the second held “Learning Camps” in which volunteers held Pratham-based sessions during school. Finally, these versions were robust enough to be implemented at scale. Today, the additional-hour model exists in 107,921 schools across 13 states of India, reaching nearly five million schoolchildren. The “Learning Camps” model is in 4.210 schools, reaching over 200,000 schoolchildren.
After a decade of work, there are now two versions of the Pratham program that reach millions of children each year, utilizing proven strategies to help children learn. Along the way, the RCTs played a critical role in developing this program and making it as effective as possible. They also provided economists — as both academics and practitioners — with opportunities to understand the nuances of India’s educational system, from individual motivations to national policy. Moreover, the lessons learned can now guide other researchers working on RCTs.
Scaling up RCTs can be frustrating, involve compromise, and take many more years than expected. However, this should not deter researchers. Success means having a broad, tangible impact that lasts. “The reality is that most things take time,” says Duflo. “That’s what happens when you want to deal with real life.”
♦
"From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application" appears in the fall issue of the Journal of Economic Perspectives.