NASA Using Online Competitions To Address Difficult Problems – Aviation Week & Space Technology
Originally posted in Aviation Week & Space Technology by Graham Warwick on March 4, 2013.
Solving a problem for NASA carries a cachet that ensures there is a strong response whenever the space agency posts a challenge to any of the online communities it uses to crowd-source new ideas.
A just-completed challenge offering $30,000 in prizes for ways to increase power from the International Space Station (ISS) solar arrays as it orbits saw a record 4,056 competitors register. The challenge sought a control algorithm for the rotary joints that would maximize power generation while minimizing shadowing of the longerons supporting the arrays. When the competition closed on Feb. 6, 459 competitors had submitted 2,185 solution attempts.
“NASA is a huge brand globally, and galactically,” says Rob Hughes, explaining the popularity of web-based competitions like the ISS Longeron Challenge that his company runs for the space agency. Hughes is president and COO of TopCoder, an online community of more than 460,000 algorithm and software designers worldwide that NASA taps to solve problems.
The ISS Longeron Challenge was just the latest in a series of competitions that have seen online problem-solving communities help NASA find solutions for challenges ranging from packing medical kits for long-duration space flights to mining terabytes of planetary imagery for interesting data.
And what motivates participants is not always the prize money, which often is small. There is kudos from peers for an elegant solution, the chance to build a resume that will attract the likes of Google, or simply the pleasure of solving a difficult problem. “We are giving everyone a chance to be a rocket scientist,” says Jennifer Gustetic, NASA program executive for challenges and competitions.
Unlike NASA’s Centennial Challenges, which are staged technology-demonstration events, the online problem-solving contests “leverage the power of the web to crowd-source ideas and solutions from people that may not play well together,” she says. “It is not necessary for them to come together in a physical event.”
Use of online challenges started at NASA Johnson Space Center, says Jason Crusan, director of the Advanced Exploration Systems division. “The life sciences group was interested in using challenges to solve problems with which they were rather stuck and running out of ideas, and looking for new solutions,” he says.
What emerged was a pilot project to run some 20 different challenges using online “open innovation” platforms TopCoder, InnoCentive and Yet2. “NASA could have built our own platform and community, as we did for the Centennial Challenges, but we chose to tap into a community that already existed,” says Crusan. “TopCoder has an existing software and algorithmic community and we wanted access to that.”
The pilot kicked off in July 2010 with a $23,000 challenge on TopCoder to develop a mathematical algorithm that would determine the optimum contents of medical kits for future manned space missions. “It was a classic packing algorithm,” says Crusan. The challenge drew more than 2,800 code submissions from almost 1,100 participants, with the top solution-providers coming from the U.K., Japan, Indonesia and Brazil.
Other challenges followed on InnoCentive, ranging from designing a zero-g laundry machine to developing a way of forecasting solar activity that poses a radiation risk to humans and hardware in space. “We received a solution that exceeded our requirements from a retired radio-frequency engineer in rural New Hampshire, which can forecast a solar event hours in advance with 85% accuracy,” says Gustetic
The online platforms used for the challenges ar2 “fundamentally different,” says Crusan. The NASA Tournament Lab (NTL) was established with Harvard Business School and TopCoder to run software-oriented challenges. The NASA Innovation Pavilion, which uses InnoCentive, focuses on ideas and approaches, such as the $20,000 challenge to come up with a new way of measuring the strain on Kevlar and Vectran straps without damaging inflatable space structures.
“The original goal was 20 challenges over two to three years. We are coming to the end of the pilot project this October and will be close to hitting 20,” he says, adding “We have learned quite a bit.” One lesson is that a prize with a large dollar value can be intimidating. “People may doubt they have a $1 million idea, but think they have a $10,000 idea, so you get higher participation.”
NASA’s online challenges are attracting interest and ideas from outside the traditional space community. “We did a bunch of pilots in 2007-09 with the InnoCentive crowd-sourcing platform. Over seven prizes we had 3,000 solvers participate,” Gustetic says. “Out of the 3,000, 81% had never registered to get a request for proposals or work with NASA. We are tapping into huge groups of problem-solvers.”
With slightly more than $1 million in funding over its life, the NTL does more than just run competitions, says Crusan. The project also has funded post-doctoral students at Harvard to study how best to run challenges.
“The research highlights general lessons on how to size the incentives and generate the right behavior,” says Hughes. “If you are looking for broad-ranging ideas and different viewpoints, you want a large group and as many submissions as possible. If you have a narrower, more focused requirement you want fewer participants because you need an optimized, high-quality solution.”
“It has been very interesting,” says Crusan. “We have run experiments on signaling, to see if it is just cash, or other incentives that get people to participate,” he says. For some it is status within the online community; for others it is building a resume to get a sought-after job at Google or the NASA Jet Propulsion Laboratory.
“TopCoder’s rating system is central to participation, particularly on the algorithmic side,” says Andy LaMara, program director. “Handles on the forum are color-coded to indicate the individual’s proficiency. This has an interesting effect. Red handles are at the very top, several standard deviations above their peers. When they weigh in, others listen,” he says. And if a TopCoder “red handle” applies to Google “they skip five of the nine interview steps.”
Sometimes NASA is looking more for ideas than algorithms, as illustrated by series of challenges involving the agency’s Planetary Data System (PDS), a 100-terabyte archive of images and data from 30 years of planetary exploration. “We knew we needed better algorithms to search the silos of data, and we knew what we wanted to search on, but what else could you want to search for?” asks Crusan.
“So we ran an ideation [idea-generation] challenge. Anybody can compete, and some like it better than creation,” he says. The challenges were aimed at high school students and teachers, looking for ideas for web-based applications and teaching tools to exploit the planetary data. “After that we ran more detailed challenges focused on implementation.”
A fundamental lesson from NASA’s experience with challenges is that problem decomposition is key, says Crusan. “There is value in decomposing a problem to frame it to a community of non-domain experts. That’s a valuable skill set, challenge or not,” he says. “Decomposition of the problem allows us to interest a wider audience, but takes longer than you would guess at the beginning.”
NASA understands challenges and platforms better than the other customers, says Hughes, and is bringing people into TopCoder that are new to the community, and to challenges. “In the ISS Challenge, three or four of the top 10 winners are new to the platform,” says Crusan. “We need to go back and ask them why they participate, and how much time they put in,” he says. “This is just as much about getting at the why— that’s part of government’s role.”
For Carlo Piovesan, a programmer in Italy and first-place finisher in the ISS Longeron Challenge, it was his second NTL problem. “I found them interesting and challenging,” he says, describing ISS Longeron as “very cool.” It took him two weeks to solve, and he says the NASA challenges are worthwhile. “I have tried a few new things, gone back to studying topics which I had forgotten and explored a lot of possible ideas before implementing the best ones.”
Others cited as reasons for participating the prize’s size, problem quality, competitor numbers and that it was a NASA competition. “You have to admit it has some ring to it,” says participant Peter Szilagyi. Hou Qiming entered after seeing the problem involved optimizing shadows. “I’m a professional of computing shadows—for the entertainment industry. I’m curious how far computer-game techniques can get you in space,” he says.
“We are definitely tapping into people that would not participate in small business awards, requests for proposals, grants, etc.,” says Crusan. “We are getting ideas that are not from the usual suspects, and the value is quite large. You have to go into a challenge with your eyes wide open, as you do not know where a solution will come from.”
Designing a good challenge takes effort. “If you’ve seen one prize, you’ve seen one prize. They are all very different,” says Gustetic, depending on whether the desired outcome is a broad range of ideas, a point solution, or “a skill that is gained by competing in an educational challenge or a network prize.”
How a challenge is scored is important. “You need to get to objective scoring to take out any biases and get a really unbiased diversity of solutions,” says Crusan. “Ideation challenges are the most subjective, but we still score them with the help of subject-matter experts. With implementation challenges you can get to very objective scoring, such as error rate, cycle time and power usage.”
ISS Longeron was “extremely objective,” he says, with final rankings based on the raw score of power generated. “I can definitely say that the biggest and most challenging work was done by the people preparing the contest,” says a participant.
Challenges designed to generate concepts and ideas, rather than algorithms, can involve “a lot of back-and-forth with the customer,” Crusan says. It helps to engage the solvers and “a post-challenge post-mortem is very valuable to the community,” adds LaMara.
What becomes of the intellectual property (IP) generated depends on the platform and the challenge. For standard challenges on InnoCentive and TopCoder, IP provisions are part of the submission process. Some winners will grant a government-purpose license while others prefer to transfer their solution to an open-source repository.
When NASA’s existing platform contracts come to an end, there will be new awards to continue running challenges and bring in new communities. The goal is to enable mission directorates to “order up a challenge like a laboratory process,” says Crusan. “We are slowly spreading the lessons learned. We have established a working group across the agency to set up the policies and methods. It is in its very early stages.”
NASA programs have shown a willingness to use solutions generated by the challenges, says Crusan. “If it’s a sound solution, they will use it. But we do make them commit up front, and that takes time.” Some solutions can be infused directly into programs, while others “help guide our internal investment,” he says. “We can use challenges to seed a program with top-quality ideas, as a technology accelerator.”
Tap on the icon in the digital edition of AW&ST for more about NASA and AFRL challenges on crowd-sourcing and problem-solving platforms, or go to AviationWeek.com/innovation.