ico-arrow-big-left

Customized Earth Project - Chemical / Aerosol Data Rendering Service

Key Information

Register
Submit
The challenge is finished.
Show Deadlines

Challenge Overview

Overview

Our client is developing an application that will be used in the HBGDki initiative (Healthy Birth, Growth, and Development knowledge integration). The application provides several different modes of analysis / data exploration, one of which will use a version of the Earth project globe for data visualization. The globe is now embedded into their larger application. We are working with the client to add additional capabilities to the Earth map. To date we have the basic globe working and embedded in the client's Frame. We have also added a new data set - "studies" to the globe. In addition to that, we recently ran a challenge to figure out efficient and effective data management and storage practice for our application and added support for rendering live weather data inside the Globe (similar to what you can see in the original Earth Project linked above).  In this challenge we'll extend our application to use live chemical/aerosol data and thereby activate the Chem and Particulates modes for the Globe.
 

Challenge Objectives

As mentioned in overview, in this challenge you'll be extending the existing code base to bring it closer to the original Globe that is used by Earth project. Specifically, having successfully implemented live weather data rendering and an efficient strategy for data storage, we now want to make the globe in our application feature-rich. The end goal would be to closely match or even expand (in some cases) the  features supported by the Globe in the original Earth Project. We will start by extending our Globe to support use of live Chemical Data (corresponding to the mode “Chem” in the Globe) and Aerosol Data (corresponding to the mode “Particulates” in the Globe).
 

There are 3 main goals in this challenge.

1. Extend/Setup data gathering services
 

In a previous challenge, we set up a service for gathering weather related data. You are welcome to extend that service or use it as a reference to set up the service required in this challenge.

Please note: During our discussion with the community in the previous challenge, we figured out that different types of data will require different setups for data related services. If you find that you can include everything within a single service, please discuss this is forums. Even in that scenario, we would like everything to be modular and self-contained so that any issue with one type of dataset does not affect any other renderings.

Following are the general requirement of the service: The service should run on the backend and should be easy to configure (endpoints, schedule for retrieval, ...etc). The scheduling capability should be flexible and allow for configuration either every XX days/hours/mins/seconds or at specific days/hours/mins/seconds. There should be libraries out there for this and we're free to discuss in the forums if needed.

Again, you can reuse the technique used for gathering weather data service as much as possible. If you want to adopt a different method, feel free to do so after discussing in forums. Please note that we expect some differences to emerge in various aspects due to the differing nature of datasets.


Gather Chemical / Aerosol data
 

We need to gather chemical/aerosol data to display the animated Chem and Particulates data on the globe. You can see an example of non-static, updating data on a production version at https://earth.nullschool.net.
The details of the resource and process to collect the data is provided in forums. Please see the document: Chemical_Aerosol Data Details.pdf

Please ask in forums as early as possible if something is not clear related to data.

Please note: The data needs to be stored as GeoJSON format in MongoDb as mentioned below. The above dataset is available in nc4 format. The data detail document provides some information on how to parse and convert such data into other formats. But it may not be complete and you will require to research a little bit about converting the provided data in GeoJSON format. Again, please discuss in forums if something is not clear.
 

Additional Notes:
This data should be gathered and stored in mongoDB on a scheduled basis.
The default schedule should be every three hours. But this should be configurable and allow the user to change the scheduling as needed.
The data should be stored as GeoJSON (we understand this is supported by Mongo). The actual data will be in TopoJSON when retrieved, so a conversion will need to take place when an API call is made to retrieve data. When "importing" new data, please take care that data isn't duplicated - if the new dataset overlaps the previous data (or if the same data is retrieved twice), this should be handled correctly so that the database contains only a single correct set of data (expanding over time). This needs to be validated during the import process (and not rely only on scheduling).
The service should also include the configurable, optional, ability to purge "data older than…" when new data is retrieved.

 
2. Adopt REST services for this new data access
 

The previous challenge has already implemented REST services associated with Weather data. You need to extend/provide a REST interface to the data stored in the database. We made it a requirement in last challenge to keep it flexible so additional end points can be added. If you find that this is not implemented flexibly or you would like to do it differently, please discuss in forums. We want to maintain the current services and reuse them as much as possible. We also want the application to be as flexible and configurable as possible so we can add additional endpoints at any time.


Additional Notes:

Provide access to the data needed to show the Chems and Particulates animation. You'll need to understand how the Globe displays this data in order to determine the exact REST endpoints needed. Please take note of the ability to "shift" the time in the menu (on the "Control" line).
Convert the data to TopoJSON before returning it to the frontend.
We only require retrieval endpoints for this challenge (e.g. we don't need REST endpoints for adding, updating, or deleting data - as this will all occur on the backend via the service mentioned above). Note - if it makes sense to do so, feel free to create those endpoints anyway for use by the back-end service which retrieves data (although I'll leave this up to you).
 

3. Chemical/ Aerosol data and overlay animation

Update the Globe in the application to use the new "live" / "real" data being collected by the service you implemented in #2.

Note that the Globe is capable of displaying several different types of data. This is primarily controlled by selecting a choice on the "Mode" line in the menu. The Chemical  display is selected by choosing "Chems” in that menu. The Aerosol data display is selected by choosing “Particulates” in that menu. Please make sure that all features of the "Chem" mode and “Particulates” mode of the Globe are working correctly with the new data.
When in "Chem" mode, you are able to select from several Overlays for the Globe - COsc, CO2sc, SO2sm. When in “Particulates” mode, these overlays will change to: DUex, SO4ex.
Ensure that each of these overlays is also being generated from the correct data pulled from the database.
Please see the nullschool demo as a reference while working on this.
 

Additional Notes and Requirements

- You can ignore the formatting issues on the left hand menu for now (client is working on these separately. Also note that these don't occur in Firefox if you'd like to see it cleanly).
We're working within a Node / Express based frame provided by the client. Please keep changes to their framework to a minimum and try to compartmentalize the Globe code, services, and DB access into it's own files as much as possible / practical. Reviewers please note: the client code (the outer frame) will not be part of the review process and scores should not be impacted by it.
There was a discussion in one of the previous challenge regarding the d3js version being used. The client is considering updating this, however they are relying on specifics on this version in other areas of their application. For now, we need to keep with the version being used.


Resources

Github Earth Project - The original base project

Natural Earth - vector data used in the Globe

Final Submission Guidelines

- Include a zip file containing your submission.
- Update the existing deployment guide in markdown format.
- Include a vidoe demonstration of your solution.

Reliability Rating and Bonus

For challenges that have a reliability bonus, the bonus depends on the reliability rating at the moment of registration for that project. A participant with no previous projects is considered to have no reliability rating, and therefore gets no bonus. Reliability bonus does not apply to Digital Run winnings. Since reliability rating is based on the past 15 projects, it can only have 15 discrete values.
Read more.

ELIGIBLE EVENTS:

2017 TopCoder(R) Open

REVIEW STYLE:

Final Review:

Community Review Board
?

Approval:

User Sign-Off
?

CHALLENGE LINKS:

Review Scorecard

?