March 5, 2019

Late Deliverables Tracking


In order to improve the quality and the reliability of the software development process the Topcoder tracks all lateness in the Online Review.

Each project in the Online Review consists of a set of phases each with its own timeline. Failure to meet the timeline for any of the phases leads to delay of the entire Online Review project which in turn delays the dependency projects. All cases when a member fails to meet the current timeline will be logged into the system for the further investigation. The lateness statistics will be used by the Topcoder to identify weak places in the process and to apply proper discipline measures to the late members if needed.

We start with the following types of the Online Review deliverables tracked and will extend the list as necessary:

  1. Specification Review

  2. Screening Scorecards

  3. Review Scorecards

  4. Test Cases (for component development only)

  5. Appeal Responses

  6. Aggregation

  7. Final Fix

  8. Final Review

Additionally, we track all cases when a final reviewer rejects the final fixes submitted by the winner and the project goes into another cycle of Final Fix and Final Review since this causes delays to the project as well.

How it works

Every time a member misses deadline in the Online Review a notification email is sent to the late member with the details of the lateness and a reminder to submit the deliverable. The notification email is also sent to the winners when their final fixes fail the final review. The email will contain a link to the page in the Online Review where the late members can view the details of the late deliverable and explain the reason why they missed the deadline. A screenshot with an example of this page is shown below.

There are situations in which a delay (or a rejected final fix) is justified and we are giving the members a way to explain why they missed the deadline. Members can type the explanation in the “Explanation” field above and then click the “Save Changes” button. The explanation can only be saved once, it won’t be possible to change it later. Explanations must be written in an objective, verifiable manner. For example, “There were ten submissions to screen in only two hours.” and “Final reviewer brought up new issues during the final review” are perfectly valid reasons. “A dog ate my laptop” is not.

There are only 24 hours after the deadline to submit the explanation, if the late member does not submit the explanation in 24 hours the late record is considered closed.

Once the explanation is saved a notification email will be sent to the Project Manager and he will respond to the late deliverable shortly. Once the PM responds you will receive a notification email with the details and a link to the same page. You can view the PM’s response in the “Response” field above. Additionally, the “Lateness Justified” status shows whether the explanation has been accepted by the PM.

Explaining late deliverables is an optional step and members can ignore it if they feel it was their fault. PM will not respond to the late deliverables that have not been explained and these cases will be considered as “not justified”.

Request an extension in advance
It is always best to request an extension in advance rather than being late and explain later. If you ever need more time to finish your work in the Online Review please contact the manager before missing the deadline.

Special cases

There are several special cases that need to be explained in details:

  1. Compensated deadline
    It often happens that the Appeals phase completes prematurely and the Appeals Response starts early. Since the Appeals Response phase is short (12 hours) reviewers can easily miss the deadline thinking they still have enough time. This problem is addressed in the following way: if a phase B depends on a phase A and phase A ends prematurely and phase B is less than 24 hours long the late record will only be added when the members miss the original B’s deadline, i.e. the deadline that would have been if the phase A had ended as scheduled. In the example above the phase A is the Appeals phase and the phase B is the Appeals Response phase (which depends on the Appeals phase) and it is less than 24 hours long.

  2. Review timeline extension on registration
    Reviewers often sign up after the Review phase has already started or even is already late. To provide them enough time to finish the review the timeline is automatically extended when reviewers sign up in the following way:

    1. For the Specification Reviewer the Specification Review phase is extended to give at least 2 hours from the moment of registration.

    2. For the Screener the Screening phase is extended to give at least 6 hours from the moment of registration.

    3. For the Reviewer the Review phase is extended to give at least 48 hours from the moment of registration.

As an Online Review contest progresses its phases rarely end exactly as per the original schedule. The only exception is the Registration and Submission phases which always have fixed duration. The other phases always close as soon as all the necessary deliverables are submitted (e.g. the Review phase closes when all reviewers submit the scorecards). Except for the cases explained in the “Compensated Deadline” section above it is the reviewers responsibility to watch the timeline changes regularly and be available throughout the whole project.

Late Deliverables statistic

The Online Review provides functionality to search and view all lateness for all past projects. The Late Deliverables Search page allows to filter by project category, project status, deliverable type, deadline date etc.

You can use this page to view your past statistic and the details of all of your late deliverables. The page will only let you view the late deliverables you have access to.


When members miss the deadline they delay the entire project. Besides making the other members wait it also puts the project in a risk of not meeting the final timeline. All late members who fail to justify their lateness are subject to penalty from warning to payment cut or suspension from the Review Board.

The payment for the late members in the Online Review is reduced based on their total delay. The total member’s delay for a project is the sum of all his/her unjustified late records for this project. The payment will be reduced by 5% if there’s a non-zero delay plus 1% for each full hour.

Additionally, each unjustified rejected final fix reduces the winner’s payment by 5%.

The maximum possible penalty is 50%.

Group 9
Group 9