In the critically acclaimed film, The Spanish Prisoner, Steve Martin plays a confidence man trying to swindle an unsuspecting engineer out of “the process,” a super-secret industrial formula that generates boatloads of cash. Martin and his co-conspirators are sent away in handcuffs with the process safe and sound as the credits roll. What remains mysteriously unresolved is how the process works and what makes it so lucrative.

It’s hard to look at what’s going on within development today and avoid the conclusion that donors are searching for their own version of “the process”: magical solutions to cure the world’s ills at lower costs.

There are a lot of factors driving the focus on innovation in development. In some ways this is a money problem. Official aid spending among developed countries has peaked—now hovering around $135 billion a year following a decade of steep growth—even as expectations rise for what it will accomplish. Ending extreme poverty, for example, is a high bar by anyone’s standards. But the embrace of innovation goes beyond that. The concept is an alluring one that taps into our belief in the wonders of science, technology, and entrepreneurialism. Watching a website take down the printed book industry or a mobile phone application upend one taxi cabal after another has fueled our  faith that we are only a start-up or two away from solving the world’s most vexing development challenges.

Yet here we are waiting for a modern-day version of the Green Revolution, crops springing from the wilderness.

Our want is not for lack of ideas. Marking 10 years after the launch of his foundation’s Grand Challenges—not to be confused with a U.S. Agency for International Development (USAID) initiative with the same name—Bill Gates used the words “failure” and “naïveté” to explain why nearly $1 billion in funds for 1,500 research projects failed to produce any real game changers, Humanosphere reports. Gates says he has learned from his mistakes. To secure funding, future applicants will have to pair their “far-fetched ideas” with experts in product development from manufacturers, biomedical companies, and so on.

What Funders do Best

Bill Gates’s honest recognition of the foundation’s struggles reveals some important lessons about where donors can have a greater impact on the innovation agenda and where more help is needed.

Funding Outcomes

One area of increasing focus for how governments can better promote innovation is in the power of the purse. Smarter governments understand the power of linking spending to outcomes as opposed to covering costs. “There may be no single recipe for success,” says a report from PWC. “However, across countries, economic development systems, and private markets, one common characteristic is worth singling out: Sustained success in fostering innovation and enhancing private markets, by encouraging the creation of new participants, always hinges on linking funds to performance.”

The U.S. Government has recently started to explore how it can better link payments to outcomes as the world’s largest buyer of goods and services, delivering $405 billion in contracts and $588 billion in grants in the last fiscal year. In 2009, economist and entrepreneur Sonal Shah became the first ever director of the White House Office of Social Innovation and Civic Participation, which was set up to catalyze innovations to change the way government does business. She sees the need to shift away from the traditional ways government measures impact. “The metric we use on social services is always the number of people we served. How many people came through the door?” said Shah in a discussion at DAI. ”But what if we flipped that metric? How many people got better? How many people can read better?”

Shah recently kicked off the final competition for DAI’s Innovation Challenge. Innovation is not an invention or an app, she emphasized to DAI employees, but an idea that makes a difference in people’s lives. Shah now directs the Beeck Center for Social Impact & Innovation at Georgetown University, which recently released a new report that reviews the government’s attempts to pay for outcomes through initiatives such as innovation funds and pay-for-success pilots, many of which have shown promise but have yet to get beyond their pilot phase.

Outcomes-based funding gets even trickier when it comes to international development. The places we work are inherently challenging and unpredictable, yet no agency attracts as much oversight as USAID, which accounts for less than 1 percent of federal contract obligations. While USAID’s leadership has made innovation a key priority, its programs to foster innovation tend to be standalone, Washington-based activities that are still trickling down to its missions, where most of its money is spent, and the operational policies that guide it. As one example, the word innovation appears only three times in the 75-page guide on how to assess contractor performance. Compliance appears 40 times.

The U.K. Department for International Development (DFID), which spent an estimated $1.9 billion on contracts last year, has arguably made the most progress in linking payments to outcomes. It makes payments only when a service provider hits milestones written into a contract. Facing similar domestic ambivalence to foreign aid, DFID has also complemented its focus on outcomes with proposal evaluation criteria that rank bidders on their ability to deliver more for less, or value for money.

Enabling and De-Risking Innovation

Donors have long been focused on promoting systemic changes to improve business environments—removing barriers to trade and investment, promoting stronger and fairer tax systems, or reducing regulatory burdens for small businesses, among others. These approaches can also help in the development of more predictable business environments that allow for calculated risk-taking. Startups need an ecosystem of support to nurture an idea or bring a product or service to scale. DAI is working with USAID’s Center for Development Innovation to develop such an ecosystem, bringing together universities, nonprofits, corporations, small businesses, and associated alliances to provide an array of support services: acceleration, design and management of prize competitions, product development, mentorship and networking, human-centered design, legal counseling, access to capital, and impact measurement.

Financial tools are also important for reducing risk, and donors have developed a variety of programs to encourage private sector investment around shared values—a better trained workforce, more reliable electricity, healthier populations. These tools have become even more important as official development flows have been eclipsed in scale by private investment—now seven times larger than official aid flows and counting—and remittances.

It’s fair to say that donors are better at creating environments that allow for failure than allowing it from themselves. One of the biggest challenges donors face in promoting innovation is that innovation relies on a model that embraces failure—and failing fast—almost as much as success. Retired Ambassador Thomas Miller, who had a 29-year career in government, writes in The Hill that he went into the foreign service to “make the world a better place” but instead came up against an “internal monster of our own making…a risk-averse culture.” He believes the government’s oversight- and compliance-driven culture has led to billions of dollars in spending on activities that deliver mediocre results because they are designed not to fail.

“We need to reject a culture that would [rather] see three low-risk mediocre projects with questionable long-term value succeed than hit one out of the ballpark with stunning, transformative results,” writes Miller. “Governments need to learn from American venture capitalists in Silicon Valley: if three out of ten startups get off the ground, you’re doing great. And if only one of them makes it really big, that’s still considered success.”

Creating Public Goods

While aid has declined in relative terms compared to private investment, it will remain crucial to developing countries as long as poverty, conflict, disease, and natural disasters continue to afflict them—that’s because aid is one of the few remaining sources for public goods, say analysts from Dalberg in an article explaining why “ODA is not Dead.” “Private flows are not a substitute for aid in developing the world’s poorest countries, because private flows, as we will show, do not greatly increase the supply of public goods and services,” they write.

Gates sees this dynamic as one of the driving forces of his foundation. “Most innovation is capitalistic innovation. Unfortunately when it comes to helping the poorest in the world, there is no market,” he says. “So if you’re talking about innovation for the poorest, it’s the most under-funded thing. You won’t get there.”

We traditionally think of public goods as increasing access to and improving public services such as health, education, and sanitation. But a public good looks different in the innovation space. Perhaps most importantly, a public good gives entrepreneurs with ideas that could benefit the poor the time and resources needed to pursue those ideas without facing the penalties associated with breaking ground in unproven markets. “Pioneering enterprises face first-mover costs that might otherwise deter entrance to nascent markets,” writes Molly Kinder of the Global Innovation Fund (GIF). “Subsidized pioneers can not only become successful individually but can also pave the way for replication of the model more quickly, more easily and more cheaply.”

Launched this year with $200 million in initial funding from the United States, the United Kingdom, Sweden, and Australia, GIF focuses on an area where many similar programs have yet to see much success: scaling social innovations. “Too many promising innovations get lost in the valley of death between invention and scale—stymied by a dearth of funding, the challenges of achieving economies of scale and profitability, and the difficulties of recruiting talent and high volatility,” writes Kinder, in an article pre-dating GIF’s launch.

What’s Missing?

As donors develop more sophisticated approaches to innovation, what they need from partners is rapidly evolving.

“This is the wild west,” says Kristi Ragan, who leads a DAI project supporting USAID’s Center for Development Innovation and has worked on various Grand Challenges aimed at securing water for food, powering agriculture, saving lives at birth, and so on. She interprets the emphasis on innovation among donors as a recognition that we need to move faster and at greater scale.

DAI recently helped USAID complete a challenge that generated 1,700 ideas for improving ways to treat the Ebola outbreak in West Africa. Ragan marvels at USAID’s ability to build a mechanism that could bring in so many new ideas and potential partners, but points out that these ideas will only gather dust if there is no concerted effort to identify where they are most needed and adapt them to those unique environments.

Making these connections requires resources that are not typically found among donors, a lack of integration that many observers have identified as a critical gap facing the innovation agenda.

In a report examining the potential for market-based innovations to address the needs of the poor, writers from the Monitor Group, now Deloitte, and the Acumen Fund describe the need for specialist intermediaries to channel funding and ideas from a wide range of donors to meet the evolving challenges in the field. These intermediaries “will need to develop strong on-the-ground capabilities in less-developed countries—hiring staff, building networks, finding opportunities, delivering technical assistance, managing portfolios and measuring impact—and maintain a strong connection to funders that are predominantly based in more-developed countries,” they write.

Sonal Shah has a word to describe the role of those working between government and the private sector to illuminate what’s happening on the ground and connect the right players together. She calls them “translators.”

New worlds have new languages.