Amazon S3 NodeJS Dataloader

Task Overview

The goal of this challenge is to build an Amazon S3 data loader that can be hosted and run on heroku.

Task Requirements

General Requirements

• The application should will be using express.js web framework, with bootstrap in frontend, awssum amazon s3 node module to interact with amaon s3, and hosted in heroku.
• Expected application structure should be as follow :
  • package.json
  • app.js - contains the logic
  • .gitignore - should include files that should be ignored, including .env, node_modules, ..etc
  • env-sample - this file is sample of .env file
  • public - contains public files
  • view - contains the view pages
  • node_modules
  • README.md

Frontend Requirements

The application should have the following pages and flow :

• Login page - will contain the following fields
  • route : /login
  • "Data Source" : (required) a drop down list with S3 as the only option for now.
  • "AWS Access Key ID" : (required) input field
  • "AWS Secret Key" : (required) input field
  • "Bucket Name" : (optional) Amazon S3 Bucket Name
  • For successful login :
    • If user specified valid bucket name then take user to "Bucket Details" page
    • If user specified invalid bucket name or did not provide bucket name, then take user to "All Buckets" listing page.
• "All Buckets" Listing page
  • route : /buckets
  • this page list the authenticated user's buckets
    • Columns will include bucket name and creation date
  • selecting or clicking on a bucket name will take user to bucket details page
• "Bucket Details" page
  • route : /bucket/:bucket-name/?:object-path
  • this page list all objects (folder and files) in selected bucket
  • the table should include : object name, size (if avaiable), and last modified date
    • If the object is folder then clicking/selecting it will take user to same page with objects under that selected folder.
  • Selecting a file will take user to "File Viewer" details page.
• "File Viewer" details page :
  • route : /bucket/:bucket-name/:object-path/:file-name
  • we are only limiting the viewer to support only a comma seperated csv file.
  • The csv file will be displayed in a responsive table, the csv file's first row will represents table header, rest are data rows.
• "Upload File" page :
  • route : /bucket/upload
  • This page will be used to upload csv file to S3
  • It will contains
    • a form for uploading file, and it should only be limited to uploading csv files.
    • drop down list to pick the bucket to upload the file to
  • We will support only uploading single file at a time.
  • It will show upload loading indicator.
  • Cancel upload button should be added to allow user to cancel upload.
• Navigation
  • There should be general navigation contains the following (Only appears for Logged in users):
    • Home - take user to bucket listing page
    • Upload File
    • Logout
  • breadcrumb and proper titles should be used in each nested page that we navigate to it from other pages
• Error/Info messages :
  • Properly use bootstrap alert boxes to show user errors and success messages.
  • Messages should be either dismissed when clicking on them or when user navigate between pages, or when user make an action in the page itself (i.e. Clicking on upload button).
  • Error messages must be user friendly, make sure to convert error code messages from s3 to friendly user messages.

Technical Requirements

• Configuration file should read any sensitive data from environment variables (.env file).
• Login functionality - we assume that sometimes user will make mistake when inputting bucket name in login form, so when authenticating user don't pass bucket name, the flow should be :
  • If bucket exists and user has access to it, take user to bucket details page.
  • If bucket does not exist, then take user to "All buckets" page and display error message returned from checking the bucket.
  • On successful authentication
    • Store the aws credentials in session encrypted
    • validate that bucket exists https://github.com/awssum/awssum-amazon-s3#checkbucket
  • Authenticating user as in this example.
    • If authentication failed, return user back to login page with proper error returned from amazon s3.
• All buckets functionality
  • You will use List bucket method to list all buckets that user have access to.
• Bucket details functionality
  • You will use List Objects method to retrieve objects under selected bucket.
  • There is a possibility that a bucket would contain more than 1000 objects, so you need to support pagination by adding "view more" button at the bottom that load more content via ajax call. You will be able to determine if there is more data to load by checking isTruncated attribute from the last request made. If it is included then you display "view more", if not, then hide it (this applies in both ajax and non-ajax requests).
• File Viewer functionality
  • You will use Get Object method to retrieve object.
  • We will only support csv files, so before retrieving the object we need to validate that object is a csv by checking it's metadat using get object metdata method. (you will check content-type)
  • If the file is not csv then return user to bucket details page with error that selected file is not viewable in this version.
  • If the file is csv then parse first row as the column names, and rest as data rows.
  • The data will be rendered in File Viewer page on a table, and we need to render FULL rows at a time with pagination capability via ajax.
    • This part is very tricky, we need to only stream part of the file directly from S3 each time we need to view more data so you need to use 'Range' capability to retrieve part of the files. So we assume you can retrieve 10kb of the file in first attempt, and parse the file and count number of FULL rows, send them to frontend and set the next range start byte to be the last byte value from the last full row retrieved.
    • In next "view more" calls, it will use the next range start byte we passed to frontend + 10kbyte of byte to retieve next rows.
• Upload file functinality :
  • We want to support large file upload (i.e > 100MB), so we need to use Mutipart Upload, check the node module for the methods used to support multipart.
  • Check the file being uploaded, don't allow upload if it is not csv file.
  • We want to show user the progress of upload, so properly utilize the methods to send user an upload progress status.
  • Use abort multipart upload to properly abort uploading the file when user decide to cancel upload.
• Decrypt aws credentials helper
  • After successful login, we will keep aws credentials on session so we don't ask user to login each time we need to make a request to s3.
  • Middleware will decrypt the aws credentials if exists on session.

Amazon S3 Credentials

We will provide credentials in challenge forums for testing purpose.

Deliverables

• Full application implementing the functionality outlined above as .git repo.
• README file to configure and deploy the application on heroku, use https://help.github.com/articles/github-flavored-markdown

Final Submission Guidelines

.