November 30, 2020
Incentivizing innovation
Could the "interim-payment" model solve the key flaws with research contests?
Illustration by Chris Fleisher
Netflix was still mailing DVDs in red envelopes when it announced a contest to improve its recommendation system.
The algorithm for suggesting movies was among its most valuable assets. After all, picking a disappointing movie would force customers to mail it back and potentially wait weeks for a better one. So Netflix offered $1 million to whoever improved the algorithm’s accuracy by 10 percent.
Somebody won the competition, but Netflix lost. The company determined that “the additional accuracy gains … did not seem to justify the engineering effort needed” to implement it.
Netflix thought it just wanted a better algorithm, but didn’t realize there were other important criteria it required.
“That's the nature of research,” said Jean-Michel Benkert, an economist currently working in the financial services industry, in an interview. “You often don’t know exactly what you’re looking for at the outset.”
In the November issue of the American Economic Journal: Microeconomics, Benkert and co-author Igor Letina propose a new design for research contests that might have solved Netflix’s conundrum. It combines features of the two most common types of competitions—“fixed price” contests and “innovation races”—while eliminating their biggest flaws.
That's the nature of research. You often don’t know exactly what you’re looking for at the outset.
Jean-Michel Benkert
In innovation races, some kind of goal is set and whoever achieves it first wins. But it’s tough to know whether what you want will ultimately be useful. That’s what happened with the Netflix contest.
The “fixed price” format awards the prize to the best innovation, regardless of whether they “win the race.” The problem here is the deadline. Setting a time frame might not give contestants enough time to solve the problem or, if it’s too long, then it would delay the implementation of the technology.
Benkert and Letina propose a third way: the “interim-payment contest.” In this approach, payments are made to the competitors while the contest is ongoing. The organizer in each period can decide whether to stop the contest or not. If the contest is stopped, the final prize is paid out.
Interim-payment contests
The most common types of research contests have flaws. This model proposes a way that might realign the incentives to solve those problems.
The authors say it solves three problems. First, it eliminates wasteful duplication of research that happens in fixed-prize contests, since the contest stops as soon as the winning criteria have been met. It also eliminates the chances that a contest will end before the necessary innovation has been found. If the organizer determines that the innovation won’t be useful, even if the original criteria was met, it can just keep the contest going. Finally, the innovation is made available to the contest sponsor as soon as it’s available, so that it doesn’t have to wait until some arbitrary deadline to implement it.
It realigns everyone’s incentives in the best way.
“This final prize is a solution to provide incentives to the contestants. They're gunning for the final prize. These interim transfers are only there to provide the right incentives to the organizer, to actually say stop,” Letina said. “That tradeoff from an organizer's perspective is where to expect that optimal level of innovation to be.”
That’s not to say this will always be the right model for research contests. First, it’s more complicated than other common approaches. Also, there’s a nuance in how the price set for the interim payments and final prize could affect the participants’ incentives. Suppose the interim payments are too big. Contestants who want to keep collecting those checks may not immediately disclose that they solved the problem, waiting a period or two and delaying the implementation of that innovation.
“We would really like to see people experiment with it and test it out in the real world and see how it performs,” Letina said. “These payments during the contest seem to be a good lever and we would like to see a little bit of experimentation with that policy lever and see what can be achieved with it.”
♦
“Designing Dynamic Research Contests” appears in the November issue of the American Economic Journal: Microeconomics.