June 3, 2016
Why are some schools surprisingly good at producing star math students?
Evidence from the American Mathematics Competitions
Winners of the 2015 United States of America Mathematical Olympiad (USAMO) visit the U.S. State Department in Washington, D.C. for an awards ceremony, June 2, 2015.
Mathematical Association of America/Flickr
When it comes to standout math achievement for high school students, the U.S. is falling behind. Results from global standardized tests like the Program for International Student Assessment show that relatively few American students are able to achieve lofty scores in mathematics. On the 2012 PISA test administered to 15-year-olds in dozens of countries, only 9% of U.S. students reached Level 5 or better, lagging far behind South Korea (31%) and Singapore (40%) and leaving the U.S. in the company of middle-income countries like Hungary and Lithuania.
This matters because students with very high math achievement in school tend to go on to solve engineering problems or invent new technologies that can boost economic growth and even create entirely novel industries. In the past half-century the U.S. has been a global leader in many knowledge-intensive industries from aerospace to finance to software, sectors that have been buoyed by a relative abundance of educated and highly knowledgeable workers graduating from American schools. The U.S. stands to lose its status as a human-capital economic powerhouse if it can no longer churn out these students in appreciable numbers.
Do these disappointing test results reflect some sort of inevitable deficiency in U.S. mathematical ability, or can schools play a role in nurturing mathematical talent and potentially help the U.S. climb the rankings? An article appearing in this month’s issue of the American Economic Review studies test results from the American Mathematics Competition and finds that some schools seem to have figured out a way to produce more high-achieving students.
In Do Schools Matter for High Math Achievement? Evidence from the American Mathematics Competitions (PDF), authors Glenn Ellison and Ashley Swanson gather data on high math achievement at thousands of U.S. public schools that participate in the AMC 12, a multiple-choice test on precalculus topics administered each year by the Mathematical Association of America to approximately 100,000 U.S. high schoolers. The test is designed to test problem-solving skills and creativity rather than memorization and includes increasingly difficult questions toward the end of the exam to allow extremely talented students to stand out (readers can see some sample questions here).
The authors use counts of students who scored 100 or better on the AMC 12 in 2007 as a measure of high math achievement at each school. Their biggest obstacle is that the AMC is not given to every student: only about 10% of U.S. high schools offer the test, and students at these schools must opt in to take it, so some high achievers do not even take the exam at all. However, schools with many talented math students will tend to offer the test and recruit top students to take it (the exam is free for students), so the results still generally reflect a fairly broad sample of top students.
The number of high scorers at a school is fairly predictable based on school size, the racial composition of the student body, and the demographics of the school’s ZIP code (the authors exclude charter schools, magnet schools, and private schools that might attract unusual concentrations of high achievers from outside the immediate neighborhood). The authors find that there tend to be more high scorers at schools in neighborhoods where more adults have graduate degrees and where the student body has more Asian-Americans. High scorers are relatively more rare at schools with a larger proportion of students on free or reduced-price lunch or larger populations of Hispanic or black students.
Some schools do surprisingly well on the AMC 12
Actual and predicted counts of high-scoring students (scoring 100 or more on the 2007 AMC 12) for 1,984 U.S. public high schools. Schools are displayed in ascending order of their expected high-score count, which is based on school size, student body makeup, and local neighborhood characteristics like education and income. A large number of schools dramatically outperform expectations, suggesting that demographics aren't necessarily destiny for high math achievement.
Source: data provided by Ellison & Swanson (2016)
But despite these broad trends, there are some exceptional schools that do much better than expected based on demographics. The graphic above shows the predicted and actual counts of high-scoring students at 1,984 high schools, arranged in order of predicted performance. Schools with disadvantageous demographics– perhaps they have many students on free lunch or are situated in a poorly-educated ZIP code – are on the left, and most don’t have any high-scoring students at all. Schools with the kinds of student bodies that are more likely to score highly on the AMC 12 are on the right, and naturally they tend to be home to at least a few high-scoring students.
There are definitely some underperforming schools, but more notable is the thick right tail of schools that do way better than expected. A substantial number of schools produce high scorers at more than twice the expected rate (pink), and a select few produce them at more than five times the expected rate (dark red). These schools are not only scoring highly – they are squeezing more high math achievement out of the local population than the local demographics would suggest was there.
Does this mean some schools have cracked the code and can mint high-achieving students at much greater rates than the typical school, or is this just random variation? There are bound to be schools that beat the odds, but the authors note that a similar analysis using counts of perfect SAT/ACT math scores and the same demographic data finds that only about one-fifth as many schools are performing dramatically above expectations.
Something about the high math achievement measured by the AMC 12 seems to be especially sensitive to school environment, which might be a good sign that schools are not powerless to foster high math achievement in bright students.
Some of this could be attributable to peer effects: if a few enthusiastic students at a school are able to motivate each other to work harder or start a math club, high scores would be expected to cluster in a few schools (and wouldn’t necessarily be expected to continue after a star student leaves). The authors analyze data from 2003 and find that the same schools tend to overperform in 2003 and 2007, suggesting that transitory effects from one student are less important than the institutional characteristics of a school that are stable over time.
Our findings about the thick upper tail are intriguing because they suggest that the U.S. could dramatically increase the number of high-achieving math students it produces. Of course, this would require that the effects be due to environmental differences and the environments are something that could be reproduced.
Ellison & Swanson (2016)
However, one of those institutional characteristics could be the aggressiveness of efforts to get top students to sign up to take the test. If this is an important factor, then the overachieving schools aren’t necessarily any better at coaching students to excel in math. They are just better at showcasing the talent they already have.
But the authors analyze data on students who score exceptionally well (120 or better on the AMC 12) and find similar patterns. Students capable of scoring this highly are few and far between and are likely to come to the attention of math teachers in the school, who would then assiduously recruit them to take the test. The authors argue there is basically no concern about self-selection with these students, suggesting that the outstanding results at some schools are mostly real and not just an artifact of high test participation.
The authors dig more deeply into some of the high schools they identify as especially high performing relative to demographics, and find a few patterns. They found that many overachieving schools had star teachers – teachers who had led math clubs for decades, been honored with teaching awards and covered in the local press, or taught special before-school math classes for highly motivated students.
This might be slightly discouraging news in the sense that such teachers are obviously quite rare and can’t just be dropped into every school in America. Indeed, when a long-serving star teacher in Nebraska retired in 2012, a local official worried that the elements of her class that helped students excel were “not possible to be duplicated.”
But the schools also had other commonalities that are more easily recreated in other contexts, including curricula with advanced material on multivariable calculus or special classes that prepared students for research competitions like the Intel Science Talent Search. It’s too early to say whether these moves actually increase student achievement directly or just reflect a broader school culture of valuing or encouraging high math achievement. Either way, the authors’ findings are a hopeful sign that, with some effort, the U.S. educational system can start producing more star math students and catch up to the rest of the world. ♦
“Do Schools Matter for High Math Achievement? Evidence from the American Mathematics Competitions” appears in the June 2016 issue of the American Economic Review.