November 18, 2019 TCO19 Dev Final Recap
I can’t believe it’s been one year since the last TCO. The TCO19 development final started at 12:00 pm, on Nov 15th, 2019 as scheduled. Ten finalists from around the world gathered together to compete for the glory of TCO final champion in a short four hours.
In this challenge, the finalists were asked to create a data scraper that will run as a CLI app and that will be used/called on at pre-scheduled intervals to scrape data from the NASA AVAIL API, then create or update a dynamic sitemap that search engines can consume to index the data from the AVAIL web page.
To avoid introducing traffic to the actual NASA AVAIL site, a mock version of the AVAIL API was provided to the developers. The NASA AVAIL API document is https://images.nasa.gov/docs/images.nasa.gov_api_docs.pdf
From the API document we can see the API has some limitations. So, all finalists had to come up with an efficient algorithm to collect as much of the data as possible and save it in the Sitemap API, which led the competition to be more challenging.
The scoring criteria of this challenge is a bit different from the usual. Every submission will be judged in the following three aspects:
- The scraper efficiency: the time the scraper is running. The maximum allowed run time is 2 minutes. Any scraper that takes more than 2 minutes to execute will be terminated.
- The ability of the scraper to capture all the data available.
- The ability of the scraper to detect duplicate data. A penalty will be applied in cases of duplicate data.
So our finalists had to focus on not only the functionality implementation, but also efficiency and duplication removal. It looks a bit similar to the competitive programming contest, as time complexity (efficiency) and correctness of the results are both important.
Around one hour after the final challenge kicked off, several members got points in the leaderboard and then the leaderboard was fluctuating. Even later when too many scraper calls made a lot of traffic to the mock API and caused some technical issues, all finalists were still enthusiastic and kept optimizing their submissions.
The intense four and a half hour final challenge completed successfully. Well done all finalists. The final results will be revealed during the award ceremony on Nov 16th. Good luck to you all and let’s see who will be the TCO19 Dev champion!