The reviewer should be able to take each point in the requirements document and find the associated implementation in the code. The reviewer should ignore the quality aspects of the implementation at this point, and only verify whether a requirement is addressed or not. Other questions of the scorecard give you an opportunity to express your concerns about the quality of the implementation.

Rating 1 – Missing requirement(s) prevent the implementation from fulfilling the goal of the requirements.
Rating 2 – The implementation addresses all requirements to solve the problem for which the component was requested, but is missing major features from the requirements document.
Rating 3 – The implementation addresses all of the  requirements and potentially includes minor additional functionality. The implementation solves the problem for which the component was requested but either does not provide enhancements on top of the requirements, or the added functionality does not alter the customer’s experience with the component.
Rating 4 – The implementation addresses all requirements and provides new and substantially useful and pertinent functionality

The reviewer should check if the technologies and components specific to the component are used and used properly. For example, if a singleton pattern is required--Is it implemented correctly? In a thread safe manner? Are TopCoder components or other third party libraries used properly? Are other technologies used or implemented properly (e.g. XML, JDBC, EJB, servlet, JSP)?

Rating 1 - Components and technologies are not used, or used improperly affecting the functionality of the component or making the component unusable.
Rating 2 - Some components and technologies are not used, which that slightly influences the performance or results in excessive amounts of code. Same score applies if some components and technologies are not used as the best practice dictates, but the final goal is achieved nevertheless.
Rating 3 - Components and technologies are used properly.
Rating 4 - Components and technologies are used properly beyond what is outlined in the requirements. The implementation is not a simple copy/paste of the code provided in the spec. It is clear that the developer has done some research of his/her own (or already possessed the knowledge).

The reviewer should look at the presence and quality of the required algorithms. All algorithms must be implemented as described in the algorithms section. Improvements are allowed and encouraged, as long they are properly explained and documented. They do not necessarily have to be approved in the forums, because they can be considered a competitive advantage, but they should be at least approved by the PM in a private mail. Fixing mistakes is not considered an improvement. If an algorithm has mistakes in the requirements or it is not appropriate, then it is up to the developer to fix it or work with the PM on it.

Rating 1 - Some algorithms are missing or are implemented so poorly that the component does not work in the way it was intended to.
Rating 2 - Some algorithms are not implemented as the requirements specify, and this slightly affects the performance of the component or the elegance of the implementation.
Rating 3 - The algorithms are implemented properly.
Rating 4 - All algorithms are implemented properly and efficiently and the developer brings improvements beyond those described in the requirements. Direct translations of the pseudocode from the spec (or even copy/paste) should not be awarded with this score.

Class structure and method names make usage of the API apparent to a new user. Each method has a singular purpose and causes minimal side-effects. Convenience methods should be carefully chosen to simplify the overall API. Excessive use of convenience methods should be avoided.

The reviewer should look at the indicated demo class in the test source tree to see the developer's vision of how the component will be used. The reviewer should then analyze how the API addresses the requirements and how easy it is to use the API to accomplish the requirements.

Rating 1 – The API demonstrates an obvious lack of consistency, cohesion, or is generally hard to use. Tasks (as defined by the requirements) take multiple API calls to accomplish what should be done atomically.
Rating 2 – The API is not immediately obvious; one cannot understand how to accomplish the required tasks by looking at the class diagram and reading the demo code (provide the name(s) of the requirement task(s) not easily performed using the API).
Rating 3 – The API is obvious (by looking at the Javadocs/MSDN Docs and the demo code) but could have been made simpler or more powerful/flexible (provide specific modifications to the API that would make it easier to use).
Rating 4 – The API is consistent, powerful, flexible, and is easy to understand. The design of the API is driven by a single concept, making the API easy to learn. The common tasks (as defined by the requirements) are easy to perform using the API

The reviewer should determine whether the classes in the implementation attempt have too large of a scope (it tries to do too many tasks) or has too small of a scope (the task was broken down into too many classes). A good clue to a violation of this is whether the name of the class adequately encompasses the API of the class. If so, then the class is probably properly scoped (unless the name is too generic or too high-level).

Rating 1  Multiple unrelated groups of related classes violate scope and must be refactored into smaller or larger classes. Describe in your comments the purpose of refactoring for each class or a group of related classes, and state its tangible benefits to the component.
Rating 2  A single group of related classes violates scope and must be refactored into smaller or larger classes. Describe in your comments the purpose of refactoring, and state its tangible benefits to the component.
Rating 3  The classes seem fine but at least one class could potentially benefit from being refactored. Describe in your comments the proposed refactoring, and explain what makes the refactored solution preferable to the developer’s solution.
Rating 4  All classes are scoped well.

The reviewer should make a determination whether classes in the implementation could be grouped into separate packages. This category generally applies to complex components or components that allow for functionality to be plugged in (where the functionality should justifiably exist in a separate package).

Rating 1 – The component is sufficiently complex or uses pluggable classes that can justifiably be separated into sub-packages and was not.
Rating 2 – The developer separated some classes into separate sub-packages but either missed some classes or missed other sub-packages that could be justified based on the developer’s approach. For example, when one set of features was moved into a separate package, but another similar set was left in the main package.
Rating 3 – The developer created an unnecessary sub-package or the component could potentially benefit from an additional sub-package The reviewer should provide their reasoning when suggesting a sub package be eliminated or added.
Rating 4 – The developer appropriately used sub-packages or the component did not need any sub-package

Have internally used classes been defined incorrectly as public? Has a public class improperly exposed protected or private members? Is the Serializable attribute used in lieu of implementing the ISerializable interface for standard serialization of .Net classes? It is strongly advised that variables have a private scope and only be accessible via get/set methods or C#'s property variables. Additionally, the effect of the final, virtual and other similar modifiers on class extendibility should be considered in this question. Each class and method should be designed for extension or marked as final.

Rating 1 - The implementation demonstrates obvious lack of understanding or ignores modifiers fully.
Rating 2 - The implementation incorrectly assigns a majority (>49%) of the modifiers or scope.
Rating 3 - The implementation incorrectly assigns a minority (<49%) of the modifiers or scope.
Rating 4 - The implementation appropriately uses modifiers.

.Net Custom exceptions follow the Microsoft Design Guidelines on defining custom exceptions and have the appropriate constructors and are marked as Serializable.

Has the implementation appropriately reused existing system exceptions and appropriately defined custom exceptions for the rest? The reviewer should also imagine they were using this component in an application. If a custom exception is thrown, does it give enough information to the application to react appropriately? Example: an inappropriate custom exception would be some high level exception that is thrown for multiple, unrelated, reasons.

Please note that TopCoder Software has decided that java components should not throw NullPointerException.  When null is illegally passed to a method which doesn’t accept it, Java components should throw IllegalArgumentException and C# components should throw ArgumentNullException.

Rating 1 – System exceptions were re-used where a specific custom exception should have been defined. Appropriate custom exceptions were not defined or a single generic exception is used throughout the component for every custom reason.
Rating 2 – Appropriate custom exceptions were defined but for multiple, unrelated reasons or system exceptions were reused inappropriately.
Rating 3 – Custom exceptions were used appropriately for multiple, related reasons but do not give specific reason details beyond the included text or the implementation inappropriately identified a null argument exception.
Rating 4 – System exceptions were reused appropriately. Custom exceptions were used appropriately and give reason details beyond simple text. Custom exceptions were used appropriately and for a single specific reason alone. Custom exceptions were not appropriate for this component.

Are the type assignments correct and what you'd expect?  Are the method signatures as descriptive as possible?  Are method parameters and return value object types statically identified, or is the user expected to cast to the appropriate type?  Please note that TopCoder Software has made determination to prefer statically defined parameter and return values rather than to require the component user to cast objects to the appropriate type.  When Collections are used, are they used appropriately?  Was the best (in terms of performance and overhead) Collection implementation used for the component's internal representation?

Rating 1 – A majority (>49%) of types are used incorrectly (not including array to collection stuff). That is, a string used where an integer should be etc.
Rating 2 – At least one type is used incorrectly (same limitation as Rating 1).
Rating 3 – The types are appropriate and arrays are used where collections should have been.
Rating 4 – The types are appropriate and the collection interfaces are used where appropriate (i.e. arrays are used when justified). Method parameters and return types are statically identify

Does the name provide a hint as to what the function of the entity is? Does the name follow system language and TopCoder standards (capitalization, underscores, etc.)?

Rating 1 – The name of one or more entities is totally misleading or incorrect for the entity’s function. Note: the developer may challenge this if they can give sufficient justification as to why the name is appropriate. Only ‘public’ names should be eligible for this rating.
Rating 2 – One or more names violate either system language or TopCoder standards. Only ‘public’ names should be eligible for this rating.
Rating 3 – All names follow the system language and TopCoder standards, and the names cover the functionality of the entities; however another name may more fully describe the functionality. All names (private or public) are eligible for this rating.
Rating 4 – All names follow the system language and TopCoder standards and fully describe the functionality of the entity. All names (private or public) are eligible for this rating.

Please note that ‘public’ is defined as:

- Public or protected class names
- Public variables or protected variables on public or protected classes
- Public methods or protected methods on public or protected

Does the developer duplicate functionality that is already provided by the language? Are properties, delegates, interfaces, inheritance, networking, etc. used appropriately? Does the implementation pay attention to releasing the non-used resources, for example by using the .Net Dispose pattern?

Rating 1 – Language features are consistently ignored.
Rating 2 – Language features are inconsistently ignored or consistently misused.
Rating 3 – Language features are incorporated consistently and occasionally misused or abused.
Rating 4 – Language features are used correctly

The reviewer should look here for variable and object types that are not the best choice (from a performance or memory consumption point of view). For example, using a Vector instead of an array. Interfaces should be used for attribute types instead of the concrete implementation classes where possible (e.g., List instead of ArrayList). Efficiency is a primary concern here. For example, HashMaps should be used instead of TreeMaps, unless key order is important.

Rating 1 - Sub-efficient object types are consistently used and this could potentially affect the performance of the component.
Rating 2 - Some less efficient object types are used but this is not a pattern of misuse throughout the implementation.
Rating 3 - Concrete classes are used instead of interfaces as variable types.
Rating 4 - No better object types can be found and interfaces are used everywhere as variable types.

The reviewer should look for pieces of code that are not needed. Such code includes emulating standard Java functionality (such as array copying), code that can be easily simplified, unused variables, assignments to variables that are not used afterwards.

Rating 1 - Useless code is present in many places and that seems to be a pattern in the implementation.
Rating 2 - There are some pieces of code that could be simplified with little modification.
Rating 3 - Some useless code appears by accident.
Rating 4 - No useless code can be found.

The reviewer should score here how well the developer managed to avoid code redundancy. There should be no duplicated pieces of code. The developer should factor out code into helper classes or private methods and implement overloads if possible. This factorisation should not go to the opposite extreme either. The end result should not be a class with many small helper methods that do very little. Every method used to factor code should have well defined functionality and should fit naturally in the implementation.

Rating 1 - The code is consistently duplicated. Large pieces of common code are copied in different places and there seems to be no effort to avoid such situations.
Rating 2 - Some duplication exists but there are signs that the developer paid some attention to redundancy.
Rating 3 - No code redundancy exists.
Rating 4 - No code redundancy exists and the developer uses elegant techniques to avoid it.

The reviewer is supposed to score here the clarity, the elegance and the efficiency of the code. The code should not be more complicated than it needs to be for what it does. It should not use complicated `if’ constructs that make it hard to follow. There should not be huge pieces of code. Where needed, private methods should be used to reduce the complexity. The code should be as efficient as possible and different optimisation techniques could be used to achieve that (e.g., caching information, breaking out of loops once the result is obtained). However, the optimisations should be used wisely without going to the opposite extreme where the optimisations affect the clarity and the elegance of the code, without bringing significant benefits.

Rating 1 - Complicated code is written for simple tasks to such a degree that the code can no longer be followed. Unnecessary complications and sub-efficient code is a pattern in this implementation. The code is neither clear nor efficient.
Rating 2 - The code clarity and efficiency could benefit from some improvements but it is obvious that the developer was concerned with these aspects. The same score should be given if the balance between clarity and efficiency is not right.
Rating 3 - The code is clear and efficient where needed. This score should be given if there aren’t any obvious places that need improvement but the developer did not show any special skills in the written code.
Rating 4 - The code is clear and easy to understand. Necessary optimisations are used but without going to the extreme of sacrificing the code clarity for insignificant gains. The developer has shown obvious skills in writing clear and efficient code.

The reviewer should check whether the exceptions provide useful error messages.The spelling of the messages should be checked, too.

Rating 1 - No messages at all or the messages are so poorly written that they aren’t very useful.
Rating 2 - Consistent errors in messages are present but the messages are still comprehendable.
Rating 3 - Some minor typos or some isolated situations in which the messages are missing.
Rating 4 - Meaningful and correct error messages are provided for each thrown exception.

The reviewer should check here the code style used by the developer. Tools such as Checkstyle should be used to properly check it. In addition, there should be no tabs, proper tags (author, version) and copyright section. Because there are tools to check the code style automatically, nothing less than perfect from the code style point of view should be tolerated.

Rating 1 - Consistent code violations are present to such a degree that the code looks bad.
Rating 2 - Many code violations are present, but when looking at the code, it still seems professional.
Rating 3 - Some minor code violations are present here and there.
Rating 4 - The code style is perfect and every element is present.

The reviewer should look at this from an application developer standpoint. Class documentation should contain a description of the class and its purpose. The documentation should emphasize how the class will be used and also if it is subject to future extensions (subclassing, for example). Class documentation should contain code examples of class usage where this is the case.

Rating 1 - Documentation is missing for one or more classes or it proves to be consistently unhelpful. The documentation provided by the developer does not clarify how most of the classes will be used, by providing simple comments that do not bring additional value to the name of the class.  
Rating 2 - Documentation exists for each class in the component but it provides little help to the developer; it is missing details regarding the way a class is intended to be used.
Rating 3 - Documentation is complete and adequate to use the component but requires further minor clarifications in several situations.
Rating 4 - Documentation is complete and provides full details to the application developer using this component. The developer will likely not require any other details on how the component will be used.

The reviewer should look at this from an application developer standpoint. Method documentation should contain a detailed description of the method which can include: intent of method, pre and post conditions, side effects, dependencies, implementation notes, who should be calling this method, whether the method should or should not be overridden, where to invoke super when overriding, control flow or state dependencies that need to exist before calling this method.

All required tags defining parameters, return type and exceptions should be present and correctly describe method details.

Method documentation should contain code examples where appropriate.

Rating 1 - Documentation is consistently missing or consistently unhelpful. A majority of the methods either lack documentation or do not provide required details.
Rating 2 - Documentation is missing on at least one method or provides little help to the developer.
Rating 3 - Documentation is complete and adequate to use the component but may require further clarifications.
Rating 4 - Documentation is complete and provides full details to the application developer using this component. The developer should not require any additional information on how the component will be used.

The reviewer should look at this from an application developer standpoint. Variable documentation should contain a description of the variable role and purpose, implementation notes and whether it is intended to be used in subclasses. Variable documentation should exist for all variables in the implementation, regardless of the visibility of the variable.

Rating 1 - Documentation is consistently missing or consistently unhelpful. A majority of the variables either miss documentation or do not provide required details.
Rating 2 - Documentation is missing on at least one variable or provides little help to the developer.
Rating 3 - Documentation is complete and adequate to use the component but may require further clarifications.
Rating 4 - Documentation is complete and provides full details to the application developer using this component. The developer should not require any additional information on how the component will be used.

The reviewer should look at this from a future component maintenance developer standpoint. This issue also ensures that there are no useless comments that pollute source code and make it hard to understand. At a minimum, the developer should provide meaningful comments for blocks of code in methods, describing what it is trying to do, the assumptions made and the way it is done.

Rating 1 - Documentation is consistently missing or consistently not helpful.
Rating 2 - Documentation is missing in at least one important algorithm / code block or consistently provides little help to the developer. Documentation contains useless comments that do not provide any value to the implementation, but only make it harder to read / understand.
Rating 3 - Documentation is complete and adequate to understand the implementation, but there may be several places where it can further be enhanced with useful details, or simplified by removing useless comments.
Rating 4 - Documentation is clear, complete and concise, helping in clarifying implementation decisions and details.

That means the documentation must be perfect from a syntactical point of view. There should be no missing tags, no additional tags, not even for private members or inner classes. Tools such as javadoc, Checkstyle or features of the modern IDEs should be used to check it.

No - The documentation misses documentation tags or generates warnings and errors for different other reasons.
Yes - The documentation is correct and complete.

Rating 1 - The developer made so many grammatical or spelling errors that one cannot understand what the developer is trying to relate.
Rating 2 - The developer made a number of grammatical or spelling errors but the overall knowledge transfer can be made with multiple readings.
Rating 3 - The developer made some grammatical or spelling errors but the points were quickly understandable.
Rating 4 - The developer was concise and made no grammatical or spelling errors.

The reviewer should verify if for each public method and constructor defined by the component, the developer provided test methods. The reviewer should focus on the existence of the test methods, instead of focusing on their quality. The Test Case containing the demonstration of the component will not be considered when counting existing test methods.

Rating 1 - Most of the test methods are missing or the existing test methods incorrectly test the functionality of the component.
Rating 2 - The submission does not contain test methods for trivial functionality of the component, such as set / get methods or exception constructors. However, this does not appear to be a pattern throughout the unit testing.
Rating 3 - There is at least one test method for each public method and constructor.
Rating 4 - There are several test methods for each public method and constructor.

The reviewer should look at this section from the point of view of a client and identify if the provided demonstration suits the needs of such a client, by helping the client understand how the component is used.

Rating 1 - No demonstration of the component usage is provided, or the one provided is not helpful or does not refer to important basic functionality of the component. Basic functionality is that functionality that is essential to the component.
Rating 2 - The provided demonstration properly describes the usage of the basic functionality of the component, but fails to present a demonstration of additional non-basic functionality.
Rating 3 - The provided demonstration properly addresses the usage of the component.
Rating 4 - The provided demonstration is clear, helpful and addresses all the needs of a client. All basic and additional functionality of the component is described, and properly documented.

Unit tests evaluate every public method in the component to ensure that valid and invalid inputs are handled properly and as defined in the documentation. The reviewer should focus on verifying the correctness and the quality of the test methods provided by the developer. The existing test methods should correctly test the functionality of the component as detailed in the documentation.

The Unit Test Cases consider using real data and appropriately configure the system as it is expected to be in the real world. For example, if a component implies using a database, the Unit Test Cases should provide test methods that interact with the database, instead of providing mock-ups of data.

Rating 1 - The unit tests do not correctly and thoroughly test the basic functionality of the component. The developer provided tests that address only valid or invalid arguments, or do not test the component using real values and system configuration. Basic functionality is that functionality that is essential to the component.
Rating 2 - The unit tests do not thoroughly test the additional functionality of the component. The developer provided tests that address only valid or invalid arguments, or does not test the component using real values and system configuration.
Rating 3 - The unit tests correctly and thoroughly test the functionality of the component. Basic and additional functionality of the component are addressed and tested using real values and system configuration, as it is expected to be run on the client’s computer.
Rating 4 - The submission is properly tested using different types of tests that address different aspects of each tested method or constructor. For example, if an algorithm has time constraints it should be tested not only for accuracy but also to see if it respects the constraints.  When testing for accuracy, an algorithm should be presented with different sets of data.

The reviewer should check if the Unit Test Cases wisely configure the test environment using setup and teardown methods, without duplicating the configuration with each test method. Also, the configuration should address only required items, and should not contain more configurations than needed. The reviewer should also look for extra initializations that affect the performance of unit testing.

The unit test methods should not contain configuration details that are system specific. They must be stored in separate configuration files and loaded when needed.

Rating 1 - The test environment is not configured using setup and teardown methods, but instead the configuration is repeated over and over for a majority of the test methods, or the test methods contain configuration details that should have been stored in configuration files.
Rating 2 - In several places, setup and teardown contain configuration and initialization sections that either do not address all entities of the environment, or address more entities than needed, or affect the performance of the unit testing.
Rating 3 - Setup and teardown contain configuration and initialization sections that either do not address all entities of the environment, or address more entities than needed. The missed or added entities should represent minor details, such as repeatedly initializing the same variable without affecting the performance or the integrity of the unit testing.
Rating 4 - Setup and teardown are used wisely for configuring the test environment, and configuration details that are system specific are stored in configuration files.

The reviewer should verify that all test files and configuration files used in Unit Testing exist and are located in the /test_files directory.

No - The developer either provided invalid files or did not store them in the appropriate /test_files directory.
Yes - All files used in the Unit Testing exist in the /test_files directory or there are no test files or configuration files used.

The system is not affected after unit testing of the component, meaning that no temporary files are left on the file system, no network / database connections are left open, no files / streams are left open.

No - Temporary files are left on the file system or database / network connections are not closed after testing is complete.
Yes - Temporary files are removed from the file system after testing is complete and temporary files are created only when needed.

The reviewer should look at this from a future component maintenance developer standpoint. Unit tests documentation should contain details about what is tested (which method / constructor), the expected behaviour and the arguments used. The quality of the documentation has to be the same as for the component code.

Rating 1 - Documentation is consistently missing or consistently not helpful.
Rating 2 - Documentation is missing on at least one test method or consistently provides little help to the maintenance developer.
Rating 3 - Documentation is complete and adequate to use the component but may require further clarifications.
Rating 4 - Documentation is complete and provides full details to the developer maintaining this component. The developer should not require any additional information on how the testing of the component will be maintained or enhanced.

The evaluation criteria include how much work needs to be done to complete it, whether the submission deserves to pass or not, and it’s level of quality compared to an average submission that meets all requirements.

Rating 1 - The submission is lacking in many areas. In order to complete it, much work needs to be done.
Rating 2 - The submission is lacking in many areas but the component still works and it is possible to test it. The problems it has should be fixable without much effort.
Rating 3 - The submission meets all requirements but it doesn’t excel in any way.  The fixes that need to be done should not be significant. The submission should be of average quality.
Rating 4 - The submission excels in many areas. All requirements are met and some are exceeded. In order to complete the component, only minor items need to be fixed. This submission is clearly above average.