IBM - SiBM Watson Confusion Matrix Visualization Tool Design Concepts Challenge

Challenge Summary:

Welcome to the IBM - SiBM Watson Confusion Matrix Visualization Tool Design Concepts Challenge!  In this challenge we are looking for your help to create screen designs for our new Watson Confusion Matrix Visualization Tool, which will be leveraged by IBM Watson Service customers and partners.  The user interface will provide an intuitive and easy-to-use dashboard that will help our end users with the complexities of using our cognitive services, such as the Natural Language Classifier. 

For this challenge we are going to focus on only one Watson Service (hosted on Bluemix): the Natural Language Classifier, which is used to classify text. In the future, additional Watson Services will be added to the Confusion Matrix Visualization Tool. While the details of these other services are out-of-scope for this challenge, please make sure your screen designs are setup to house additional services in the future and is not just focused on the "Natural Language Classifier" tool.
 
At the end of this challenge, we are looking to see intuitive and easy to use screen design concepts that will help us build an app with a great UI/UX!  We look forward to seeing your creative screen designs!

Round 1

Round 1 will focus on the following screens for the new application (note: please feel free to include any additional features you feel will be helpful):

1) Landing Page
2) Tool Configuration (Only Bluemix config is needed for Round 1)
4) Project Configuration
5) Project Dashboard
6) Training Data View
7) Test Data View
9) History View

NOTE: If anything is unclear, please ask for clarification on the forum and we will be happy to help!

Round 2

In Round 2, you will need to include all Round 1 screens (updated based on checkpoint feedback).  For Round 2, you must also include all the remaining functionality described in the project documentation, including the following screens:

1) Landing Page
2) Tool Configuration (Both Bluemix config and Persistence are required for Round 2)
3) Create Project
4) Project Configuration
5) Project Dashboard
6) Training Data View
7) Test Data View
8) Test Run View
9) History View
0) Launch Icon

NOTE: If anything is unclear, please ask for clarification on the forum and we will be happy to help!

Challenge Description:

The purpose of this challenge is to design screens for our new IBM Watson Service Training Management Tool.  The goal is to create the best possible user interface and system for cognitive system developers and analysts who require a tool to manage the complexities of content management, training and historical evaluation.

Project Background Information:

The primary goal of this project is to produce a user-friendly, extensible tool that provides content management, transformation, training and testing functionality. The idea is that the tool will be workstation-based similar to what is offered by the Electron platform (http://electron.atom.io). While we like the idea of an Electron-based solution, we are open to additional considerations.

Natural Language Classifier Details:
- Natural Language Classifier is a multi-label classifier that accepts short text (<= 1024 characters) for classification. The general process one follows when working with the NLC is:
- Define training data (our input will be an Excel spreadsheet or CSV)
- Define Blind test data (Excel, CSV)
- Create a classifier with the training data (creates a trained immutable classifier)
- Evaluate performance using blind test data
- Repeat until desired performance is obtained

We want our customer to create a project that identifies the training data and blind test data. It is ok not to have the blind test data initially, but it needs to be available before testing! Training data should be evaluated against best practices and the application should  warn the user if the training data does not meet the best practices. The tool may allow the user to edit the code inside the tool or require the user to re-import an updated spreadsheet. Training and testing data should be accessible from the user’s file system – but may be stored in the cloud or other data source once imported.

Conceptually, an intermediate step will transform the training content into a form that is importable into NLC (CSV). This step will allow for training data to be stored in a variety of formats, then transformed into CSV for classifier creation and training.

When the user is ready to create a classifier, the transformed data is sent to the relevant NLC API to create a classifier. Once the classifier is created, it is ready to be tested. The implication here is an API integration with the Cloud Foundry APIs. There are two ways to accomplish CF integration. First, one may require the user to have the Cloud Foundry CLI tools installed on their workstation and make calls to the CF commands to create a service. The other option is to directly integrate with the Cloud Foundry platform with a language-specific library.

Testing the effectiveness of the training data is accomplished by sending the blind test data to the classifier. The results are compared to the expected results. This information is then displayed to the user in the form of a confusion matrix. The confusion matrix is the minimally necessary design element to display. However, designs that account for additional useful metrics and metric summaries will be extremely welcomed.

NLC Example:
Stage 1: Instead of logging into Bluemix or using the cloud foundry cli, there are SDKs available for Java and node that allow one to execute these steps programmatically. It would be great if the new tool would account for that (user the user's credentials, pick space to create a service in, create a service, etc).
Stage 2: Contains the sample data, use tool to create a classifier
Stage 3: is classify text...and how one would use testing data to create evaluation metrics on the classifier performance
'Delete': Once a classifier is created, it becomes immutable, so if feedback from the training cycle necessitates updated training data, a new classifier must also be created...and the old one would be preferably removed.

Training Process:
Training runs for a project should be recalled for historical evaluation of different training data and evaluation runs.

Design Guidelines:

- The screens designs will be for a new app: the Watson Confusion Matrix Visualization Tool.
- Screen size: This app will be designed to be a desktop app (1280px). The app will primarily be used on a desktop.
- All screens and functionality described in this challenge spec and the wireframes must be included in your submission.
- Platform: We like the idea of an electron-based solution (http://electron.atom.io/), but we are open to additional considerations.

Challenge Requirements:

The goal of this challenge is to create user-friendly screen designs for an extensible tool that provides content management, transformation, training and testing functionality. The idea is that the tool will be workstation-based similar to what is offered by the electron platform. While we like the idea of an electron-based solution, we are open to additional considerations.

Note: The wireframe mockup included shows the desired functionality, but the client is flexible on the layout.  The left bar for example may be useful on some screens, but that real estate can probably be better used in the project screens.

The following screens are required (at minimum):

0) Launch Icon:
- Please make sure to design a simple app launch icon for this new application (64x64px).

1) Landing Page
Wireframe: 01-Landing-Page.jpg (note: we would like the Landing Page design to be different than the layout in the wireframe, please use this wireframe just to see some basic requirements)
Description: First, the user opens the tools and is presented with options to create projects or view their existing projects. The goal is to create a project that manages the relevant data, output and training for a specific service.  The following elements are needed:
- Should have recent projects, search box, and list of projects with sortable/filterable values
- Button to create new projects of different types of services (NLC and “Future Functionality” for example)
- Allow the user to select an existing project (and search through existing projects)
- Delete a project
- Ability to define a testing project for evaluation of classifier systems

2) Tool Configuration
Wireframe: 02a-Configuration-Page-Bluemix.jpg, 02b-Configuration-Page-Persistence.jpg
Description: Selecting 'config' allows the user to configure their Bluemix settings, as well as their persistence mechanism.

3) Create Project
Wireframe: 03a-Create-New-Project.jpg, 03b-Create-Project-New-NLC-Service-Dialog.jpg
Description: This screen allows the user to create a new project. Should include a way to create a new service as well (for example, create a new NLC Service).  Projects are a way to “Elegantly group together training and testing data, ability to define transformations, and identify target service”.  The goals of this page are to:
- Support different types of input for each project type
- Identify training data
- Identify test data
- Identify target service
- Include project description
- Ability to add a note to a project (include note text and timestamp) 

4) Project Dashboard
(no wireframe provided)
Description: This is the main screen for a project.  It should have a high-level dashboard with lists of training data, test data, and test run data, and easy navigation to project configuration, training data view, test data view, Test run view, and the history view.  
The Project Name, Project Description, Service used (with icon), link to notes (with icon), and other relevant metadata should be on this page.
Note:  The left menu bar in the wireframe is optional in these screens.  

5) Project Configuration
Wireframe: 04-Project-Configuration.jpg
Description: Allows the user to view / configure the project.

6) Training Data View
Wireframe: 05-Training-Data.jpg
Description: Add new training data to the project.

7) Test Data View
Wireframe: 06-Test-Data.jpg
Description: The main goal is to understand the performance of the trained classifier(s) given a set of test data. This page will allow the user to import new testing data and use the "run test" button to select a classifier (or create a new classifier with a version of training data) and test it with the current testing data.  Results should be displayed that include metrics such as:
- Confusion Matrix
- Recall and Precision measurements
- For example, with NLC the user might see a chart of confidence level over time (each run X confidence level).  if you mouseover you can see the date run and training / test data used.
- Each run should be tracked.

8) Test Run View
(no wireframe provided)
Description: The main goal is to show the results of each test run.  The data and results should be clickable so the user can drill down into the data and export it.  Results should be displayed that include metrics such as:
- Test Run Number
- Timestamp
- Test Data Version
- Training Data Version
- Results
- Notes Icon (notes popup where the user can add timestamped notes)
- Status (Complete, In progress)

9) History View:
Wireframe: 07-History.jpg
Description: Allow the user to view historical training results over time.  This should use all of the data provided in the Test Run View.
Challenge:  We’re not exactly sure how the History Tab will be implemented and would love to see your ideas!  Perhaps the user needs to select a testing data version to see a history of that versus training data versions? Versioning makes things interesting. We’re looking for your ideas for a good user experience.
- Every test run will be a data point charted across multiple classes
- Can explore confidence over time for multiple classes (or one class if you filter)
- Negative trends for some classes can be good if they aren’t a good fit for the training data, so some classes will trend positively and others will trend negatively
- There can be 200 different classes we are tracking confidence for - so adding dynamic filtering by class would be worth exploring
- Explore other charts only showing deltas of results (EX:  show just the improvement amount of 5 classes over 10 test runs)

IMPORTANT: Please see the attachments / wireframes to understand more details about each page!  We’re open to your creative ideas to improve the UI/UX in the wireframes as long as all the required functionality is included.

Helpful Links:

- Natural Language Classifier (http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/doc/nl-classifier/)
- NLC Tutorial with Sample Data: https://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/doc/nl-classifier/get_start.shtml
- Classification: (http://blog.dato.com/how-to-evaluate-machine-learning-models-part-2a-classification-metrics)

IBM Design Guidelines:

All submissions MUST follow IBM Design Language, please refer to the client website: http://www.ibm.com/design/language/index.shtml
-- Layout: http://www.ibm.com/design/language/framework/visual/layout.shtml
-- Typography: http://www.ibm.com/design/language/framework/visual/typography.shtml
-- Iconography: http://www.ibm.com/design/language/framework/visual/iconography.shtml
-- Color: http://www.ibm.com/design/language/framework/visual/color.shtml
-- Interaction:  http://www.ibm.com/design/language/framework/interaction/introduction.shtml
-- App Inspiration: http://www.ibm.com/design/language/inspiration.shtml
-- Click around Client Design Brand site to get familiar with more client design brand that not stated on this challenge spec.
-- Creativity is also welcome!

Accessibility Guidelines:

- Make sure when you are planning your concepts you are also designing for accessibility.
- Since this application will be very visual (colors) - think about color blindness and the need for contrast/labels.
- You will notice the color accessibility suggestions within the IBM Design Language.
- Accessibility checklist: http://www-03.ibm.com/able/guidelines/web/ibm508wcag.html

Documentation Provided:

- Wireframes: Contains the basic ideas for screens, content, and the flow of the application. Please feel free to expand on these wireframes or add additional elements you feel will be helpful.  These are just a starting point - we want to see your ideas to come up with the best UX possible.
- sample_nlc_training.csv: how you define training data (this format)
- NLC Example.txt: NLC Example

Target Audience:

- All community members including IBMers and IBM Partners.

Judging Criteria:

- Overall Design and User Experience (UX)
- How well you plan the user experience and capture your ideas visually.
- Cleanliness of your graphics and design
- How well you capture the requirements in this challenge spec and from the wireframes.

Submission & Source Files:

Preview Image
Please create your preview image as one (1) 1024x1024px JPG or PNG file in RGB color mode at 72dpi and place a screenshot of your submission within it.

Submission File
- Submit JPG/PNG for your submission files.

Source Files
- All original source files of the submitted design. Files should be created in Adobe Photoshop and saved as layered PSD file, or Adobe Illustrator as a layered AI file.

Final Fixes
- As part of the final fixes phase you may be asked to modify your graphics (sizes or colors) or modify overall colors. We may ask you to update your design or graphics based on checkpoint feedback.
-  See more information about Final Fixes.