113% ROI on Large Enterprise Crowdsourcing Programs - Forrester TEIā„¢ Get the Study ×

Node-RED for Topcoder Challenges with IBM Bluemix & Watson

By wwwtc In Uncategorized

Posted July 28th, 2015

For the past week I’ve been obsessed with Node-RED. If you are not familiar with it, Node-RED is an open source IBM technology for “wiring together hardware devices, APIs and online services in new and interesting ways”. Since Node-RED is built on Node.js, this makes it ideal to run at the edge of the network on low-cost hardware such as an Arduino or Raspberry Pi as well as in the cloud.

Similar in methodology to Yahoo! Pipes, Node-RED initially started as a visual tool for wiring IoT but has developed into a service that can do all sorts of crazy things. Installation is simple but to make life easier, IBM Bluemix offers a Node-RED template so you can get a web app up and running in no time.

I went through the “create your first flows” tutorials but I wanted to make something more substantial utilizing some of the super sweet IBM Watson services. Here’s what I created, and in the end, it was ridiculously simple to build. A video of the flow in action is embedded at the bottom of this article.

  • Calls the topcoder RSS feed for JavaScript challenges every hour.
  • For each new challenge, call the challenge API to get its details.
  • The flow only processes challenges where the type is ‘Assembly Competition’ and the total prize money is greater than $1500.
  • Translate the name of the challenges and its requirements into Spanish using the Watson Machine Translation service.
  • Save the Spanish overview of the challenge to MongoDB.
  • Tweet the challenge name (in Spanish) and a link to this application so that the Spanish requirements can be viewed.

node-red

The flow tweets a link to the app (http://jeffdonthemic-node-red.mybluemix.net/challenge?id=:id) so that people can view the Spanish version of the challenge. Building the REST endpoint was almost laughable. Node-RED provides input and output node for http request/response, allowing for the creation of simple web services. My endpoint does the following:

  • Fetches the challenge from MongoDB using the challenge id passed.
  • Generates output HTML with the Spanish data using a mustache template.
  • Writes the template to the output stream displaying the Spanish challenge content in the browser.

node-red

IBM Bluemix Setup

Login to your IBM Bluemix console and create a new Cloud Foundry App using the Node-RED Starter boilerplate. Ensure you add a MongoDB service (I chose MongoLab) and the Watson Language Translation service. You’ll need the credentials from the Language Translation service when building the flow so click the “Show Credentials” link and keep them handy.

Building the Flow

Our application is simply built by finding a node in the left panel, dragging it onto the tab and connecting them together.

To get started, add a Feedparse node for the topcoder RSS feed and enter this URL for the feed URL.

node-red

The feedparse node passes a msg.topic for each entry so parse the ID of the challenge and build the msg.url to pass to the next node.

node-red

Make a GET request to the url passed in msg.url (instead of in the URL form field) and pass the response to the next node as a parsed JSON object.

node-red

Add a Switch node so that we only process flows where the challenge type is ‘Assembly Competition’.

node-red

Add a Function node that calculates the total prize amount and passes it along in the message.

node-red

Now we add another Switch node which continues the flow only if the total prize money is greater than $1500.

node-red

Add another all purpose Function node and prep the challenge data for translation. Add the challenge object to the payload and challenge name itself to the payload for the next node (Watson Translation).

node-red

Add the Watson Translation node and enter your credentials for the service. The node translates the name and passes it to the next node as its payload.

node-red

Add another function node and set the challenge name to the newly translated challenge name and add the requirement payload so that they can be translated.

node-red

Add another Watson Translation node and translate the requirements into Spanish.

node-red

Add a final Function node and prep the payload to be saved in MongoDB and tweeted. First we replace the English requirements with the newly translated Spanish ones and create two object for the outputs. We then wire these two outputs to their corresponding nodes.

node-red

Add a mongo output node to save the msg.payload object to the challenges collection. Connect the top output in the previous node to this one.

node-red

Lastly, add a twitter out node, connect the bottom output from the previous node to it and then authenticate with your twitter credentials.

node-red

If you deploy the workspace now it should run the Feedparse and automagically persist data to MongoDB and tweet… assuming everything is setup properly.

Building the Web Service

Now we need to build our web service so that when people click on the link in the tweet, it displays the challenge information in Spanish in their browser.

Add a new sheet to your workspace, drag an http input node onto it and setup the method and URL. When the /challenges endpoint receives a request it passes it to the next node.

node-red

Add a Function node that simply parses the challenge id from the URL string and adds it to the payload.

node-red

Then use a MongoDB node to find the challenge record by id in the challenges collection.

node-red

Add another Function node that simply uses the first record returned in the array from MongoDB and adds it to the msg.payload. This is done to make it easier to work with in the templating node.

node-red

Lastly, we want to make the HTML for the user (somewhat) pretty so we use a Template node and use mustache to format the msg.payload.

node-red

Then we simply use an http response node to send the HTML response back to the initial http request and we are done!

Results

Now when the flow receives new challenge entries in the RSS feed, it will run the flow then tweet the challenge info in Spanish and display the Spanish version of the challenge on the web page for users to view.

node-red

node-red