Objective

The objective of this page is to establish architectural best practices for TopCoder component persistence, to be reflected in future components and to be factored into existing components.

Status

Working Draft

Requirements

Before establishing specific best practices, it is necessary to determine the requirements to be satisfied by persistence systems compliant with those best practices. Only in that context can the recommended practices be appropriately evaluated. This section explains the requirements that the best practices aim to satisfy.

Business Entity Persistence

The best practices will be aimed at persistence requirements for components dealing in individual business entities. They will be suitable for use with substantially all generic components having such persistence requirements, across business domains, but they are not intended to support bulk data transfer such as performed in large batch processes or data warehousing.
Business entities are defined by an architect or logic component designer, and do not have any a priori relationship to the structure of a persistent representation. In particular, they do not necessarily map directly onto tables in a relational database. Although an individual entity might map to one row in a particular database table, it might also be mapped to one or several rows of each of several tables, or to a subset of the columns of a single row of one table.

Consistency across Components

The best practices will establish a consistent programming model for persistence across all compliant components. It is acceptable that existing components may need to be updated to implement the practices.

Pluggable Persistence Implementations

The best practices are specifically charged with decoupling logic-oriented components from particular persistence implementations. They will enable persistence mechanisms appropriate to a component's usage context to be inserted or changed without modification of component or program source code.

Support for Enterprise Application Requirements

Components based on the best practices will support the requirements of enterprise-scale applications and programming models and runtime environments appropriate to such applications.

Applicable to Java and .Net

The best practices must be suitable for both Java and .Net, the primary platforms used by TopCoder applications. Where necessary, particular recommendations may specify different cosmetic details for the two platforms, but the structures and behaviors consistent with best practices will be substantially the same across platforms.

Performance

The best practices will aim to be compatible with good performance, but performance is generally a characteristic of specific persistence implementations, and therefore is out of scope.

Requirements Discussion

The individual and combined implications of the requirements bear some consideration, as they are not all obvious.

Design by Contract

The expectations that persistence systems compliant with the recommendations herein should be pluggable with respect to individual components, and that they be appropriate for use in the context of generic reusable components, lead directly to the conclusion that a design-by-contract approach is essential. Components with persistence requirements will need to specify expectations for and constraints on persistence implementations, and to rely on implementations to comply in all details. Contracts must be sufficiently specific to allow their participants to rely on them alone, without knowledge of implementation details. Design by contract is in general a fundamental aspect of the TopCoder application and component methodologies, so this is more of a reaffirmation of our reliance on that approach than a novel requirement.
It is acknowledged that a generic approach may preclude some conceivable usage scenarios that otherwise could be accommodated by a component or application designed around a specific persistence mechanism. The number of such scenarios will be minimized inasmuch as doing so is consistent with the requirements.

Programming Model

The requirements to support enterprise environments and applications, and to be consistently useful across components imply that the best practices must account for issues such as multithreading, transaction management, remoting, clustered application environments, and a wide variety of persistence mechanisms.

Simplicity

In order for contracts specified by the best practices to indeed be widely useful for multiple generic components and programming models, with various persistence mechanisms, on two different platforms, it will be necessary to keep them as simple as practicable.

Recommendations

This document recommends both a high-level architectural pattern for components having persistence requirements, and detailed patterns for persistence-related component interfaces. It is strongly recommended that components implement these patterns to realize their persistence requirements.

Persistence should be provided via DAOs and DTOs

Data Access Objects (DAOs) provide creation, retrieval, update, and deletion services related to particular business entities. Each DAO interface is paired with one corresponding Data Transfer Object (DTO) interface describing the objects that serve as vehicles for the persistent data.

• Components that require persistent storage of individual business entities will define appropriate DAO and DTO interfaces and use the DAO / DTO pattern to satisfy their persistence requirements.
• Each DAO implementation will be a separate component. Typically, each DAO implementation will provide its own DTO implementations.
• DAO and DTO interfaces will belong to the client component, so that DAO components depend on the client components to which they provide services, not the other way around. The client therefore doesn't depend on any particular DAO implementation component, and distinct DAO implementations depend only on the client, not on each other.
• Components will follow TopCoder configuration best practices Configuration Management to provide for configuration of the DAO implementation that a component instance will use.
  DTO architectural standards
• DTO interfaces will provide only property getter methods, and only for those properties relevant to the problem domain to which the component's data applies.
• DTO class and method names will comply with TopCoder code conventions for the relevant language.
• DTOs that have large, unstructured objects among their properties (i.e. properties that might be represented in a relational database as BLOBs or CLOBs) will present them to consumers in the form of byte arrays or character arrays, as appropriate.
• DTO implementations will be serializable to support remoting.
• DTOs representing complex business entities may contain other DTOs representing subordinate or related entities. Where such a design is possible, it may or may not be suitable; it is the architect's responsibility to decide.
• DTOs representing complex business entities may contain objects representing subordinate or related data that do not constitute independent entities. Where such a design is possible, it may or may not be suitable; it is the architect's responsibility to decide.
• DTO interfaces will not expose or depend on details of any specific persistence technology.
• Multiple related DTO interfaces should be defined to represent the same underlying entity where doing so provides stronger guarantees of program correctness.
  • This is a matter of persistence logic, not business logic. For example, if the business entity has an artificial key, and the DAO interface specifies that implementations choose a value for that key upon entity creation, then a DTO interface should be defined that does not provide access to the key field, for use as the declared type of the argument to entity creation methods.
  • This system is not intended to require multiple interfaces where entity properties may be undefined according to the applicable business rules.
  • DTOs representing the same entities should have appropriate inheritance relationships.
• DTO users must not assume any particular semantics for DTO hash code computation or value equality comparisons without certain, extrinsic knowledge of the DTO implementation class. Neither the default semantics of the Object class nor any particular property-based semantics are guaranteed.
  

DAO architectural standards

• DAO interfaces will provide only creation, retrieval, updating, and deletion methods for a single logical entity, which entity is represented by a corresponding DTO.
• A DAO interface will normally define at least one method of each category (creation, etc.), but
• DAO interfaces will define the minimum methods needed to satisfy component requirements.
• All parameters to DAO methods will be of serializable* types.
• DAO interface method signatures will adhere to the patterns described below.
• DAO implementations will not manage transactions internally. Those that operate in a transactional context, or may do, will be configured appropriately by the application (not by their companion client components). To facilitate control by the application, implementations will normally make use of an indirect mechanism for obtaining an underlying persistence resource; for instance, a database-oriented DAO might rely on the DB Connection Factory component or some similar facility to obtain database connections.
  DAO entity creation method patterns
• DAOs' entity creation methods will be named "create" (as adjusted for applicable code conventions; in C# the name would be "Create").
• Entity creation methods will accept a DTO representing the entity to create as their single argument
• An entity creation method may provide for automatically assigning artificial keys; in this case, the DTO interface used as its parameter type will not provide property getter methods for the fields that the DAO is expected to assign.
• An entity creation method will return a DTO instance representing the entity as persisted in the database. Fields of the returned DTO will match the corresponding fields of the argument DTO, and where any domain data is automatically created by the DAO, those created values will be reflected as well.
• If a create method is unable to successfully create a new persistence entry corresponding to the provided DTO argument, then it throws an exception as described below.
• DTOs provided as arguments to a DAO create method belong to the persistence consumer; the DAO will not modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.
• DTOs provided as return values from a DAO create method belong to the persistence consumer; the DAO will not subsequently modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.

DAO entity retrieval method patterns

DAO interfaces will almost always provide entity retrieval methods of one or both of two types: "get" methods which retrieve exactly one (specific) entity with the implicit assumption that such an entity is present, and "find" methods which retrieve zero or more entities matching specified criteria.

• DAO "get" methods will be named "getBy<Key>" (as adjusted for code conventions), where "<Key>" represents the name(s) of the primary or alternate key field(s)
  • "get" methods will accept one or more parameters jointly containing the value of the key field(s) identifying the entity to retrieve. Multiple parameters are used only in the case of a compound key.
  • "get" methods return a DTO containing the entity retrieved
  • "get" methods throw an exception (as described below) if no entity matching the key is present or if there is a persistence error
  • DTOs provided as return values from a DAO "get" method belong to the persistence consumer; the DAO will not subsequently modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.
• DAO "find" methods will be named "findBy<Criteria>" (as adjusted for code conventions) where "<Criteria>" describes the criteria to be used to select entities to return.
  • DAO interfaces may optionally define a "find" method "findAll" that takes no parameters and returns all persistent entities available at the time of invocation
  • DAO "find" methods will return an array of DTO objects or a generic collection (if the component can require a language version with generics) of DTO objects. All elements of the array or collection will be DTO objects matching the criteria (none will be null), and all entities matching the criteria will be returned
    • Example: Person[] findByLastName(String lastName);
    • Example Collection<Invoice> findOlderThan(Date limitDate);
  • The order of the elements returned by a "find" method may be specified by the DAO interface, but otherwise no particular order may be assumed
  • DAO "find" methods will return a zero-length array or an empty collection if no entities match the criteria; this is not an error case
  • DAO "find" methods will throw exceptions (as defined below) in the case of an error in the underlying persistence mechanism
  • DTOs provided among the elements returned by a DAO "find" method belong to the persistence consumer; the DAO will not subsequently modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.
  • Collections and arrays returned by DAO "find" methods belong to the persistence consumer; the DAO will not subsequently modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.
  • Collections returned by a DAO "find" method cannot safely be assumed to be modifiable.
    

DAO entity update method patterns

• DAO interfaces will generally provide exactly one method for entity update, named "update" (as adjusted for code conventions).
• The "update" method will accept a DTO representing the updated values, and will return void; successful completion indicates that the entity represented by the DTO was updated to match the provided values. If the entity cannot be identified from the provided data or does not exist in persistence, or if the persistence implementation fails then an exception is thrown as described below.
• if the updated version of an entity is not sufficient to identify the entity to update (for example, if a key field is modified) then the "update" method may not be used; instead, the original entity must be deleted, then the updated version added.
• DTOs provided as arguments to a DAO "update" method belong to the persistence consumer; the DAO will not modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.
  DAO entity deletion method patterns
• DAOs' entity deletion methods will have names beginning with "delete" (as adjusted for code conventions). There may be multiple deletion methods, including methods that delete multiple entities.
• If present, a method named "delete" and accepting a single DTO is expected to delete from persistence exactly one entity represented by the DTO.
  • The method will return a boolean value indicating whether or not an entity was deleted.
  • It is not an error for there to be no corresponding entity in persistence; the method simply returns false in that case.
  • It is an error if multiple entities are deleted by one invocation of this method; in that case an exception is thrown as described below. It is the responsibility of the application to roll back the transaction in which such a deletion occurs, if it wishes to do so.
  • If the DTO provided references other DTOs of any type then it is expected that the entities corresponding to those DTOs will not be deleted as a result of invoking the method.
• Deletion methods whose names have the form "delete<Criteria>" may be present instead of or in addition to one simply named "delete". These methods will choose the entities to delete according to the criteria described by their names. Such methods return an integer representing the number of entities deleted
  • Example: int deleteByLastName(String lastName);
  • Example int deleteOlderThan(Date limitDate);
  • Example int deleteAll();
• If a deletion method fails because of persistence mechanism failure (e.g. an SQLException in the case of database persistence) then the deletion method will throw an exception as described below.
• DTOs provided as arguments to a DAO "delete" method belong to the persistence consumer; the DAO will not modify them, synchronize on them, or provide them to any other object that would thereby presume to have conflicting ownership of them.
  

Exceptions

Errors generated by DAOs' underlying persistence implementations will be wrapped in generic persistence exceptions (that do not by their class expose details of the persistence implementation) and rethrown. A generic Persistence Exceptions component will be produced to provide common, reusable persistence exceptions for this purpose. DAO interfaces will declare appropriate exceptions from that component, or will declare exceptions that extend those of the Persistence Exceptions component if necessary. Other exceptions will be thrown as required by TopCoder standards, such as IllegalArgumentException where method arguments are illegal, including when arguments that must not be null are null.

Open issues

• Full code examples
• Requirements for the Persistence Exceptions component
  • Which exceptions
  • Checked vs. unchecked
  • Carrying what data (if any)?
• Can there be indexed finder methods? (I.e. to items n - m matching the criteria, relative to some defined order)
• Rewrite / reformat into suitable white paper form
• Transaction management - further details
• Interface naming scheme (e.g. do the interface names end in DAO / DTO? Essence / core DTO interface names)
• Setter/constructor validation in the default DTO implementations
• Value of providing default implementation for DTO versus letting persistence component provide it
  

Definitions

DAO interface: a programmatic service-provider interface embodying a contract for business entity persistence. Such interfaces use DTO interfaces to define the objects that will transport entity data, thereby incorporating the contracts embodied by those interfaces into their own contract. It is preferable that the contract defined by a DAO interface be represented programmatically to the greatest extent possible, but it is acceptable for behavioral guarantees to be specified only via documentation where there is no suitable programmatic representation.

data access object (DAO) : an object whose class implements a DAO interface. DAO classes are expected to faithfully implement the contract defined their DAO interface, including aspects defined non-programmatically, and a design-by-contract approach demands that they not make assumptions about client behavior (such as the classes of argument objects) that are not founded on that contract. Sometimes this term is used loosely, in the sense of a DAO class or its entire component.

DTO interface: a programmatic interface embodying a contract for transporting business entity data, and especially (but not exclusively) for transporting such data between a DAO and its clients. It is a matter of debate whether DTO interfaces may or should provide methods for modifying entity properties. It is preferable that the contract defined by a DTO interface be represented programmatically to the greatest extent possible, but it is acceptable for behavioral guarantees to be specified only via documentation where there is no suitable programmatic representation.

Data transfer object (DTO): an object whose class implements a DTO interface. DTO classes are expected to faithfully implement the contract defined by their DTO interface, including aspects defined non-programmatically, and a design-by-contract approach demands that they not make assumptions about client behavior (such as the classes of argument objects) that are not founded on that contract. Sometimes this term is used loosely, in the sense of a DTO class.