Robonaut Challenge

This project has been completed.

The project was completed using 1 challenge and included 38 registrants from 16 different countries. In this challenge, competitors created an algorithm to control Robonaut and teach him how to interact with a taskboard. Robonaut learned how to recognize the state and location of several buttons and switches on the taskboard. Click here to watch an overview of the challenge.

To view the output of this project or to download the source code (open source), please click here.

Robonaut 2 is the first humanoid robot in space and was sent to the International Space Station (ISS) with the intention of taking over tasks too dangerous or too mundane for astronauts. There’s only one problem: Robonaut 2 needs to learn how to interact with the types of input devices the astronauts use on the ISS.

How do we fix that? To start, NASA has built several taskboards for Robonaut to play with.

The taskboards have a number of LEDs that turn on when the power switch is flipped or the buttons are pushed.

In these challenges, competitors will write algorithms to control Robonaut and teach him how to interact with the taskboard.

The first challenge is to teach Robonaut how to recognize the state and location of several buttons and switches on the taskboard. To discover the current state of the taskboards, Robonaut will need to look at it to figure out what LEDs are on/off and to locate them in “robot space”.  To do this, you will be given a set of imagery from Robonaut both here on earth, on the ISS, and in the simulator.  The camera system is slightly different for each system and has different lighting conditions as well.  Your algorithm must work for every application. Strong performance on the real imagery will translate best to software that works on the ISS.

For each test case you will be given two images – a “left eye” image and a “right eye” image – and a string array containing the names of the buttons/switches/LEDs you have to locate. In your return you have to define the button/switch’s state and (x,y) location in pixels relative to the upper left corner of the image you choose (“left eye” or “right eye”).

You will be judged on:

Accuracy

  • What is your false alarm rate vs. detection
  • How far from actual position is your computed position

Time

  • How long does it take to determine the state of the LEDs

 

Top