ico-arrow-big-left

Hercules TV Web Apps - BBC and HLS scraper updates

Key Information

Register
Submit
The challenge is finished.
Show Deadlines

Challenge Overview

A previous challenge has implemented a set of REST APIs for handling video assets, including storing them and managing them (create, retrieve, update, delete).  This challenge will add HLS validation to the scrapers that read HLS / M3U8 streams:

* NewsMax TV 
* CBS News
* Fox News

It will also update the BBC News scraper to pull MPEG DASH / MPD streams instead of HLS.

Existing API

The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.

HLS Validator

Some of the streams don't currently fit with the HLS standard, particularly the CBS News Live TV stream - it doesn't seem to match the M3U8 / HLS spec exactly.  We need to add some basic validation to the scrapers that parse out HLS streams to ensure the streams we add to the data store are valid.

Implementation

When we are scraping HLS streams for Fox News, News Max TV, or CBS News, we want to validate the HLS streams.  The implementation is at your discretion, but here are some possibile implementation tips:

* https://github.com/erikwt/hlsvalidator
* https://www.npmjs.com/package/m3u8parse
* https://developer.apple.com/library/ios/technotes/tn2235/_index.html

Configurable

The validation should be able to be turned on or off via a configuration flag that can be set in the environment and using "heroku config:set", named "VALIDATE_HLS_STREAMS".  The configuration value will either be true or false.

BBC and MPEG/DASH

The client has asked if we can update the BBC scraper to parse out MPEG/DASH streams instead of HLS.  These streams have an "mpd" extension, like:

http://vod-dash-ww-live.edgesuite.net/usp/auth/vod/piff_abr_full_sd/b6998b-p04721k8/vf_p04721k8_968b9322-7a5c-4d9a-8585-f7f3c45302f4.ism/pc_hd_abr_v2_dash_master.mpd?__gda__=1473241500_48f91b7587c1544692bbd0ef56d814e9���

This challenge needs to ensure that the BBC scraper parses out the MPD / DASH stream URLs when available.  It can default back to HLS when no DASH videos are found.
 
README

Make sure the README is updated with verification information about the new features and configuration information so they can be easily added.

Unit tests

As with the other scrapers, unit tests are required for these new chanages.

Heroku deploy

Make sure the Heroku deployment information is up-to-date and that you keep the package.json up to date as well.  Don't expect the deployment to be anything other than "npm install" / "npm start" locally and "git push heroku master" for Heroku deployment.

Submission format

Your submission should be provided as a Git patch file against the commit hash mentioned in the forum.  MAKE SURE TO TEST YOUR PATCH FILE!
 
 

 

Final Submission Guidelines

Please see above

Reliability Rating and Bonus

For challenges that have a reliability bonus, the bonus depends on the reliability rating at the moment of registration for that project. A participant with no previous projects is considered to have no reliability rating, and therefore gets no bonus. Reliability bonus does not apply to Digital Run winnings. Since reliability rating is based on the past 15 projects, it can only have 15 discrete values.
Read more.

ELIGIBLE EVENTS:

2017 TopCoder(R) Open

REVIEW STYLE:

Final Review:

Community Review Board
?

Approval:

User Sign-Off
?

CHALLENGE LINKS:

Review Scorecard

?