Want to Drive Innovation through Crowdsourcing? Learn the 5 Steps Get the eBook ×

[topcoder] Testimony – How Prizes Drive Innovation

By wwwtc In Uncategorized

Posted April 11th, 2014

@timhicks24 – Tim Hicks

On Wednesday morning, [topcoder] President Narinder Singh had the great honor of testifying to a congressional subcommittee, as an expert witness on how prize competitions drive innovation. Aside from being the first time [topcoder] was asked to be represented, it was also the first time Google Glass were worn in a hearing. Very fitting that the two went hand in hand.

Due to our extreme excitement over this, we wanted to open up Narinder’s transcript of his testimony for all to see and share our sense of pride in the great [topcoder] community. In addition to the the written testimony, we uploaded the following three videos of the day’s events:

Without further ado, check out the written testimony from Narinder below and keep an eye out for more mentions of [topcoder] in the news!

Written Testimony of Narinder Singh

President [topcoder], Co-founder and Chief Strategy Officer, Appirio

Before the Subcommittee on Research and Technology

Hearing – Prizes to Spur Innovation and Technology Breakthroughs

US House of Representatives

April 9, 2014

Chairman, Ranking Member, and Members of the Subcommittee, my name is Narinder Singh, and I am the co-founder and Chief Strategy Officer of Appirio, and the President of it’s [topcoder] division. I appreciate the invitation to speak before this esteemed body to address these exciting topics. We believe crowdsourcing–and specifically community-based development incented by prizes–can lead to dramatic innovation, substantially increase efficiency (reduce cost) and produce much higher quality levels of technology development for both the private and public sectors.

We started Appirio in 2006. Today, it employs over 900 people with headquarters in San Francisco and Indianapolis. We have routinely been named on “Best Places to Work” lists in the San Francisco area and in Indiana, have received numerous innovation awards, and in 2012 were named a World Economic Forum Technology Pioneer.

Our [topcoder] community consists of over 600,000 designers, developers, and data scientists. Through [topcoder] we have taken hundreds of projects for our clients and crowdsourced them through the community. This involves breaking down complex projects into smaller components and using prizes – challenge based competitions – to complete them. As a result we run five to ten thousand competitive challenges (prizes) each year through the [topcoder] community. Our average amount for each of these prizes ranges from a few hundred to few thousand dollars.

This approach has helped leading private sector and government agencies to achieve amazing results in technology that impact science and technology. In 2013 Harvard Medical School and the February edition of Nature Biotechnology (“Prize-based contests can provide solutions to computational biology problems”) described how the [topcoder] community helped Harvard Medical School improve NIH BLAST, an algorithm that aids in genetic research, by 1000x in just two weeks. The challenge received 122 different submissions and awarded a total of $6,000 in prizes.

More recently with [topcoder], Appirio helped the research organization of a large pharmaceutical firm improve the performance of software that runs Genome Wide Association Studies (GWAS) – an approach to rapidly scan markers across complete sets of DNA. We reduced the run-time from approximately ten hours to less than thirty seconds. This advancement is soon to be shared with the scientific community and will change the way research is conducted. Remarkably, the core of this advancement was driven by a series of less than a dozen contests with approximately $50,000 in prizes.

In government, NASA has partnered with Harvard Business School and [topcoder] to create the Harvard NASA Tournament Lab (NTL). This lab focuses on creating cutting edge insights on the optimal design of contests and ways in which the federal government can be more effective in the use of prize-based competitions. Together we have repeatedly used the concept of connecting a string of smaller prizes together to achieve large scale success.

For example:

  • NASA used a set of challenges to reduce the time for an algorithm to optimize medical supplies on space excursions from three hours to less than thirty seconds (completed).
  • The ISS FIT application, a mobile application to help astronauts track nutrition and health on the International Space Station, was developed through 18 challenges on [topcoder] for less than $60,000 in prizes (in final testing).
  • Currently, the NASA Asteroid Grand Challenge series is running on [topcoder] with a partnership between NASA and Planetary Resources Inc. (PRI) to improve the ability to detect asteroids. It consists of a series of ten interconnected challenges and we hope for a similar leap forward in capability (as we have seen in other challenges) for well under one hundred thousand dollars (recently initiated).

In addition, under NASA’s leadership with their Center of Excellence for Collaborative Innovation (CoECI), the NTL and [topcoder] have worked with several other government agencies including the Center of Medicare and Medicaid Services. A project to modernize CMS infrastructure so that health care providers can easily register (while at the same time states can be proactive in limiting access to bad actors) was developed for the state of Minnesota through a series of challenges. Over eleven months over 140 challenges were executed to complete the project. The total cost of labor and challenge funds was approximately $1.5 million. Forthcoming research from the NTL indicates that done through traditional models, the initiative would have cost the government nearly $7.5 million.

Predominantly, our business is with private sector companies. Organizations like Comcast and Ferguson have successfully used [topcoder] by breaking large software development projects into smaller competitive prizes. Breaking down large problems into smaller ones offers several key benefits for both projects and prizes.

  • It creates more competitive markets. By breaking down prizes you increase the number of companies or individuals who can compete in the market. For example there are only a few firms in the world that can compete for a $700M project, but there are tens of thousands of firms that can compete for a $7M one – and hundreds of millions of individuals who could potentially compete in a $7000 challenge.
  • It de-risks work through specialization. Crowdsourcing works in part because it allows organizations to tap into a much broader set of talent than otherwise possible. It also allows organizations to use markets to match the best supply (talent) to the demand of the individual task. With market volume, community members start to specialize in the areas they are best qualified to deliver outstanding results in. Quite simply, people can work at what they are best at. When every task in a larger initiative is done in this manner, dramatic results become the norm.
  • It increases participation and education. While somewhat counter-intuitive, a smaller number of connected prizes can actually increase participation. Participants in prize challenges certainly compete for the chance to be rewarded. But with micro competitions, learning is often also a significant objective. The community of participants shares knowledge extensively and broadly after each competition – creating amongst the best on the job training available.
  • It creates objective measures of skill. Each competition contributes to a skill rating for the participant. With a volume of competitions you can objectively see the ratings of an individual and the community improve. At [topcoder] we will often run a series of challenges to create an output (like the ones described above) but also to educate a community on a new technology or product. As a result, we know our community is better skilled to take on the next set of challenges in that domain.

We believe in large prizes and their capability to create entire markets as proven again and again by my esteemed colleagues from xPrize. But we also believe micro challenges allow the concept of prizes to be applied to a much larger category of work. By breaking innovative or even common problems down, exponential innovation can occur inside of existing markets.

The rationale for government involvement in prize competitions

We all recognize the importance of scientific and technological advancements in spurring growth. Traditionally, the government’s role in this has been to subsidize research and development of novel ideas and technologies, in the hope that these will produce fruitful advancements. This remains a critical function.

Note that prize-based competitions constitute a novel complement to this strategy. They differ from this traditional approach in two fundamental ways. First, prizes are paid not based on potential, but rather upon the delivery of results. Practical application and performance is therefore tied to disbursement of funds. Second, they don’t require taking a guess on who will produce the sought-after solution. They leverage competitive forces to reward the individual who delivers best solution. Thus, as demonstrated by the examples above, prizes use government funds judiciously–ensuring practical application and rewarding performance.

Commentary on the FIRST Act as related to prize competitions

We applaud the desire to include prizes and competitions in the FIRST act, but also have concerns around unintended consequences for micro challenges and connected initiatives.

The FIRST Act currently states that even for private sector judges, “All judges shall be required to disclose all personal financial interests.” It’s not immediately clear if this refers to all interests related to the prize they are administering, or simply all their financial interests. In the case of the latter, for challenges of a few hundred to a few thousand dollars, it’s an onerous burden and counterproductive to the Act’s intent.

For example, often at [topcoder] we use certified, non competing, community members to judge competitions. In these cases their scorecards and evaluations are open and subject to public peer review. In other cases, the formal judge may only enforce high level guidelines and settle disputes because the competition itself is judged and evaluated by a computer program that scores each entry. The intent in the FIRST Act in these regards is appropriate – to create transparency and fairness in oversight. But the current language could instead hinder adoption of prizes inside the government.

Broader view on making government more tech savvy

More broadly, applying this market-based approach of a large volume of micro-prizes in government requires a reduction in bureaucracy and friction of engagement. More directly, it would be unlikely [topcoder] or similar communities would choose to greatly expand presence in government sectors if requirements of audit and financial reporting remain hardwired for legacy approaches to government contracting. It’s just too hard to do business with the government. Even in submitting this written testimony it was required that forty five hard copies be hand delivered in advance – an approach to information sharing that exists only with the government.

All of this in combination can leave firms like mine with few practical options beyond either working with niche groups in the government who are willing to invest significant resources in helping navigate its complicated waters (like NASA); or through third party firms that specialize predominantly in government work. Ultimately, rather than government guidelines that attempt to enforce fairness in every scenario, a more competitive market is what will most help government operate more efficiently and be exposed to the tools and techniques that advance innovation.

Creating conditions for success in competitions

Research by Professor Karim Lakhani at Harvard and his colleagues* has shown that contests bring into play three distinct mechanisms:

  • They enable many independent “shots on goal” so that the likelihood of finding the best solution increases substantially;
  • Open entry in contests means that people outside of the traditional siloed knowledge domains of the problem can now have the chance to propose unconventional and radical solutions;
  • People are self-motivated – a variety of reasons drive participation in contests including winning the cash prize, demonstrating expertise to potential employers or their peers or just plain having fun.

All of these mechanisms have been shown to be important to successful contest execution. For government it is important to pair this with a clear sense of the objective of the prize. In some cases large cash prizes are used to draw attention to a problem area and spur private investment to advance or even create an industry. In other cases, many of which have been cited here, prizes provide an opportunity to exponentially advance an area, remove a constraint or just build more effectively.

In all cases it is important to create a clear sets of criteria for participants. What does it take to ‘win’, how will entries be scored, what timelines will be adhered to, etc. Idea generation will have very different dynamics than executing on very specific tasks. Large prizes will have more scrutiny than smaller ones, but the need for clarity and transparency transcend all categories. You will ultimately get what your rules incent.

Conclusion: the pace of change will not slow down

The rate of technology advancement will not slow down. Over the past decade we’ve seen our lives transformed into a world of constant connectivity. In addition to this new global platform of information exchange, we are seeing dramatic advancements in 3D printing, robotics, bio technologies, artificial intelligence, wearable computing and many other domains. More than ever, we will need the ability to quickly and efficiently tap into the right skills instantaneously. Even in traditional domains, like creating web based solutions for government, the scale of adoption and need to evolve quickly require a new pace and capability.

The nature of our economic system is built upon the free market and for good reason – in most cases it’s the most efficient system. Crowdsourcing taps into the power of markets but also democratizes participation beyond select firms to anyone who has the capabilities to contribute.

I thank the Committee for the opportunity to share my perspectives and would be happy to respond to any questions you may have on these topics.

* Boudreau, Kevin J., Nicola Lacetera, and Karim R. Lakhani. “Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis.” Management Science 57, no. 5 (May 2011): 843–863.

Jeppesen, Lars Bo, and Karim R. Lakhani. “Marginality and Problem-Solving Effectiveness in Broadcast Search.” Organization Science 21 (September–October 2010): 1016–1033.

King, Andrew, and Karim R. Lakhani. “Using Open Innovation to Identify the Best Ideas.” MIT Sloan Management Review 55, no. 1 (Fall 2013): 41–48.

Boudreau, Kevin J., and Karim R. Lakhani. “Using the Crowd as an Innovation Partner.” Harvard Business Review 91, no. 4 (April 2013): 61–69.

Guinan, Eva C., Kevin J. Boudreau, and Karim R. Lakhani. “Experiments in Open Innovation at Harvard Medical School.” Art. 3. MIT Sloan Management Review 54, no. 3 (Spring, 2013): 45–52