TopCoder would like to introduce changes to the current review process to both improve the quality of the reviews being performed and better reward those reviewers that excel at their job.  The cornerstone of this proposal is to introduce the concept of a Super Review Board (SRB) that would oversee review work within a specific contest category (such as component design or component development, for example).

 The Super Review Board

 At a high level, the sole duty of the SRB is to improve review quality.  In more detail, the SRB is responsible for:

• Improving the reviewer rating formula.
• Assigning reviewers to review positions.
• Auditing reviewers and reviews.

Details follow for these responsibilities:

Reviewer Rating Formula

We intend to use a formula to rate every reviewer in the review board.  This formula will be used to break ties when deciding whom among a group of reviewers interested in a project will review it, with the intent that the best qualified reviewer should get the job.

A sample formula could be (rating / 10) *  (min(#ratings, 10) / 10) * log(#reviews).  A similar (slightly tweaked to deal with corner cases) formula produced the following sample top 15 ranking of our design review board:

handle	rating	#ratings	#reviews	reviewer rating
nhzp339	2490	20	349	633
duner	2409	10	133	511
Wendell	2081	25	186	472
Pops	2572	81	67	469
adic	2201	45	122	459
aubergineanode	2516	61	64	454
Standlove	1938	108	192	442
fastprogrammer	2091	26	118	433
Ghostar	1785	143	209	414
kyky	2705	50	27	387
argolite	1542	201	309	383
Mafy	1667	20	135	355
urtks	2130	24	44	350
DanLazar	1510	30	183	341
hardstone	1833	14	69	337

The reviewer rating formula needs to be fine tuned and perfected until we can assert with confidence that a reviewer A is better for a particular review position than reviewer
B if and only if reviewer A's rating is higher than reviewer B.  This may require data we don't currently track (such as feedback from reviewer audits), and this may require us to have more specific ratings, such as a per-technology or per-complexity rating.Since the formula may not be precise enough to guarantee that the above property will hold for any two reviewers, we propose a tiering system.  This tiering system could be set up as a series of rating ranges that define each tier (for example, defining tier 1 as rating 400+ and tier 2 as rating 300-400), or as a series of tier sizes (for example, defining tier 1 as having the top 5 reviewers and tier 2 having the next 10 reviewers).  We will consider two reviewers within the same tier as being equivalent for a job, whereas a reviewer from a lower numbered tier will be considered better for the job than a reviewer from a higher numbered tier.It will be the SRB's job to propose and verify improvements to the rating formula.  The SRB may propose improvements or it may put together a set of requirements and goals for a marathon match to then come up with the particular formula.

Reviewer Assignment

Projects will have a reviewer registration phase as well as a submitter registration phase.  During this phase, any number of reviewers will be allowed to sign up for any specific piece of work.  Potentially, they will be allowed to specify preferences for the jobs they sign up for (for example, saying they would prefer to review component A, and if they can't get that position they are also interested in component B).

Project Managers and Architects, as well as the component management team, will be able to tweak pricing and registration timelines to attract better talent if they determine that the current signups aren't adequate for the project.  This would allow small, easy components to be done cheaply by lower rated reviewers, while the more complex components will be priced higher to attract the higher rated reviewers.  This is analogous to the market system that we use for submitter signups.

Once registration is closed, the SRB (initially, and down the road an automated system) will assign reviewers to projects based on a set of well published criteria:

• A conflict between two reviewers of equal tier will be resolved randomly.
• A conflict between two reviewers of different tiers will be resolved in the following way:
  • If the lower numbered tier reviewer hasn't been assigned a review yet, they get the position.
  • If the difference in tiers between the reviewers is 2 or higher, the lower tier reviewer can sign up for a second review before the higher tier reviewer gets a first review.
  • Every tier beyond 2 allows the lower tier reviewer an additional signup.

It is important to clarify that the tiering system is not really designed to spread work around the review board: it is designed to allocate the job to the best interested reviewer.  Correct pricing and registration timelines will be equally important in ensuring proper resource allocation.

Reviewer Audits

The SRB will be responsible for performing random audits on reviews.  They will initially be tasked with selecting random reviews to audit, but this job will need to eventually be handled by an automated system.

Audits would also be triggered by certain system events.  For example, any time a reviewer's score was very different from the rest of the reviewer scores (by more than a number of standard deviations, for example) then that review would be audited.  If the review passes its audit, then the other two reviews it differed from would be audited.

Other triggers could be an inordinate number of appeals or appeal petitions, or an architect or PM could manually trigger an audit based on a perceived poor review.

During an audit, the SRB would be looking for the following potential issues in a review, for example:

• Misinterpreting the requirements.
• Missing forum clarifications.
• Failing to properly justify a decision.

Super Review Board Compensation

The SRB will receive compensation in two ways:

• Each audit will be paid for.
• The SRB will be paid a bonus if certain metrics improve on a quarterly basis.  This bonus would be divided based on the involvement each SRB member had during the quarter.  The metrics would include:
  • Amount of triggered audits for the quarter.
  • Average review timelines.

Compensation would be structured so that the bonuses outweigh the audit payments the SRB would get if review quality remained low.  This way we will make it in the SRB's best interest to keep numbers like triggered audits low by improving overall review quality, as their maximum profit would come from doing random audits and collecting performance bonuses.