Topcoder’s Review Process

Topcoder Review Process

The Topcoder Review process helps ensure submissions that reach our customers are of the highest quality. Just like a manufacturing inspector puts each product through the wringer, our review board makes sure we are producing high quality output by individually inspecting every submission. Both our reviewers and submitters learn from the review process through lively discussions and feedback mechanisms which leads to higher quality output and improved skills.

The review board is made up of experienced members chosen from the community and trained in the process. Membership of the review board is voluntary but there are strict eligibility criteria which must be met for anyone to be granted membership.

Some reviews have a mix of review board members, project staff and/or the customer.

Points to Remember:

  • As the submitter, you have responsibilities within the review process
  • You are notified via email when review phases start and end
  • Delays can cause a reduction in payment

The Phases of Topcoder Review

After submitting to a development challenge with Topcoder Review, your submission will go through some variation of this process:

1. Screening

2. Scored Review

3. Appeals

4. Appeals Response

5. Aggregation

6. Final Fixes

Screening

A member of the review board, typically the primary reviewer (or screener), screens each submission to ensure it is non-trivial and is capable of meeting the requirements. The submission is judged against a scorecard, which is accessible to all competitors from the Challenge Details page. If it passes screening, the submission moves to Review.

Review Scoring

After the screening phase, reviewers assess the passing submissions based on a detailed review scorecard to determine if each solution is feasible and meets the requirements. Each submitter will have as many scorecards as there are reviewers assigned to the challenge. Challenges typically use 2-3 reviewers, thus each submitter will have at least 2 scorecards and at most 3 scorecards. Reviewers score each submission independently — they do not compare submissions. Scores are assigned and the submitter is notified via email when the review phase is over.

Appeals & Appeals Response

After review is complete, submitters have 24 hours to view and appeal the score. Reviewers leave comments on each depressed score in the scorecard, and competitors can dispute any score. However, competitors must have a very good reason to appeal, or it will be rejected by the reviewer during appeals response, which opens when the appeals phase ends and lasts for 12 hours. Read more about the Appeals process here.

Winner Determination & Aggregation

Once the appeals and appeals response phases are complete, the final scores are calculated and a winner is determined. Then, the primary reviewer compiles all of the scores of the winner’s scorecards which show incomplete or incorrect areas of the solution into a single scorecard. This is used for the next phase: Final Fixes.

Final Fixes

The winner is presented with a final scorecard that will contain items that are required and/or recommended before the submission can be deemed as complete or finalized. A submission can only be presented to the client for approval after it has been finalized. More information about the Final fixes process can be found here. Once the fixes are complete, the winner submits the final solution, the reviewer approves it, the payment is set and the challenge is complete!