ico-arrow-big-left

96h Web Scraping Account Service API

Key Information

Register
Submit
The challenge is finished.
Show Deadlines

Challenge Overview

Competition Task Overview

The purpose of this challenge is to create a service that will visite a site via a provided proxy, grap the content from a certain page and return the data in JSON format.

More specifically, this challenge must cover the following:
1. The service should be built as REST service, let's call it the AccountService.
2. Programming language should be node.js
3. In this version, we are only scraping one site, but the goal is to be able to scrape multiple sites, so we want the module that does the actual scraping to be pluggable so that we can easily add more modules to scrape multiple sites at the same time. This will likely also require some configuration parameters so the AccountService can load multiple scraping modules at runtime.
4. The AccountService must monitor all errors and report the errors to an error service, the error service will also be REST based so you can just make the endpoint of the error service configurable in this challenge. Account service will post an array of error objects to the error service, each error object will have an title and a details field. Typical errors include errors in proxy, bank sites changed the layout and AccountService can no longer scrape the data, etc...
5. Please list out blockers / risks / issues you faced when building this, we want to know them all so we will have a better idea about how to handle them when we go with other bank sites.
6. The winner will be provided with the real error service to test compatibility and fix any issues found.
7. Provide 
REST API doc in postman format
8. Provide documentation or simple architecture doc that shows how next challenges will need to create new plugin for different bank sites.
9. Make sure code is well documented.

Technology Overview

Node.js
REST
Web Scraping
JSON
AWS

Documentation Provided

Register to see details about the proxy.

Final Submission Guidelines

Submission Deliverables

Below is an overview of the deliverables:
1. 
Full code that covers all requirements
2. A complete and detailed deployment document explaining how to deploy the application including configuration information.
3. 
REST API doc in postman format
4. Documentation or simple architecture doc that shows how next challenges will need to create new plugin for different bank sites.

Final Submission

For each member, the final submission should be uploaded via the challenge detail page on topcoder.com.

Reliability Rating and Bonus

For challenges that have a reliability bonus, the bonus depends on the reliability rating at the moment of registration for that project. A participant with no previous projects is considered to have no reliability rating, and therefore gets no bonus. Reliability bonus does not apply to Digital Run winnings. Since reliability rating is based on the past 15 projects, it can only have 15 discrete values.
Read more.

REVIEW STYLE:

Final Review:

Community Review Board
?

Approval:

User Sign-Off
?

CHALLENGE LINKS:

Review Scorecard

?