Prize

1st place - $1500

2nd place - $750

3rd place - $325

Background

Over the last few months, a series of challenges have been run to generate a series of high quality financial forecasts for a consumer Broadband and Mobile provider, ‘Sandesh’.  These challenges have been known as ‘CFO Forecasting’ in various formats.  As a result, a high quality, high accuracy series of algorithmic forecasts have been produced for a number of financial target variables.

 

This challenge is being rerun to generate a similar high quality forecast for Sandesh Brand 4, a ‘sister brand’ owned by the same client, offering a similar suite of Broadband and Mobile products to the Small and Medium Enterprise market (‘SME’). 

Challenge Objective

The objective of this challenge is to generate the highest accuracy predictions possible for the 10 financial/volume variables outlined below. The accuracy of the forecast must at least improve on the Threshold target quoted for each variable/product.

(1) The model should be tailored to a 12-month forecast horizon but must be extendable beyond this time period. The accuracy of a prediction will be evaluated using MAPE (Mean Average Percentage Error) on the privatised data set.

(2) For robustness calculation do the forecast in iteration as explained in section: Quantitative Scoring point 2.

(3) The Prediction window for MAPE calculation is Oct19 – March20. Training should not be done on the prediction window data points.

Business Context

Sandesh Brand 4 (‘SME’) has two categories of products – Mobile and Broadband – which have different sets of variables to be forecasted for each of the underlying products.

(1) Mobile SME

    (a) Products:

        (i) Leopard – handset, SIM card, and Airtime contract sold together

        (ii) Panther – only SIM card, and Airtime contract sold together

        (iii) Hyena – mobile broadband service

    (b) Target Variables:

        (i) Leavers – the number of subscribers per individual product who terminated service with the brand during that month

        (ii) Gross Adds – the number of new subscribers to each individual product joining the brand during a month

        (iii) Closing Base – the number of subscribers to an individual product at the end of each month

        (iv) Total Revenue – the revenue generated by the subscriber base per month for the ‘airtime and data’ service for all Mobile products within Sandesh Brand 4. The variable is therefore the aggregate of Leopard, Panther, Hyena as well as the additional SME Mobile product offerings not included in the challenge. As such, it should be predicted as a single variable

Challenge Thresholds and Targets

 

 

Financial Year Modeling:

 

Sandesh reports its financial year from April - March.  This may contribute to seasonality based on financial year, and quarters (Jun, Sep, Dec, and Mar), rather than calendar year.

 

Anonymised and Privatised Dataset:

 

‘Z-score’ is used to privatise the real data.

 

For all the variables, following is the formula used to privatise the data:

            zi = (xi – μ) / σ

 

where zi = z-score of the ith value for the given variable

            xi  = actual value

            μ = mean of the given variable

Final Submission Guidelines

Submission Format

Your submission must include the following items

• One prediction file over the forecast time period (Oct19 – March20). We will evaluate the results quantitatively (See below)

  • Please use Time Period, Generic LookupKeys as the column names. 

  • The values in Time Period column are something like 2019-08

  • The values in each Generic LookupKey column are the predicted values, i.e., floating numbers.

  • The final spreadsheet has a Nx(M+1) shape, where N is the number of time periods and M is the number of variables that we want to predict in this challenge. “+1” is for the Time Period column.

• Six prediction files for the robustness purpose following the same format correspond to 6 sliding windows (mentioned in the robustness definition).

• A report about your model, including data analysis, model details, local cross validation results, and variable importance.

• A deployment instruction about how to install required libs and how to run.

Expected in Submission

1. Working Python code which works on the different sets of data in the same format

2. Report with clear explanation of all the steps taken to solve the challenge (refer section “Challenge Details”) and on how to run the code

3. No hardcoding (e.g., column names, possible values of each column, ...) in the code is allowed. We will run the code on some different datasets

4. All models in one code with clear inline comments

5. Flexibility to extend the code to forecast for additional months

Quantitative Scoring

First, MAPE on Prediction Window

 

Given two values, one ground truth value (gt) and one predicted value (pred), we define the relative error as:

 

            MAPE(gt, pred) = |gt - pred| / gt

 

We then compute the raw_score(gt, pred) as

           

            raw_score(gt, pred) = max{ 0, 1 - MAPE(gt, pred) }

 

That is, if the relative error exceeds 100%, you will receive a zero score in this case.

 

The final MAPE score for each variable is computed based on the average of raw_score, and then multiplied by 100.

 

Final score = 100 * average( raw_score(gt, pred) )

 

MAPE scores will be 50% of the total scoring.

 

You will also receive a score between 0 and 1 for all the thresholds and targets that you achieve.  Each threshold will be worth 0.03 points and each target will be worth 0.03 points.  If you achieve the target for a particular variable you’ll get the threshold points as well so you’ll receive 0.07 points for that variable. Your points for all the variables will be added together.

 

Second, Robustness

 

Once model building is done, robustness of the model is to be calculated. For this we need to do evaluation on a rolling forecasting from origin. See the below image for understanding. (This is just a sample image in actual, dates will be changed for the training and forecasting horizon)

1. Every horizontal line represents one iteration.

2. Blue windows are the training period and Orange windows are Forecasting period.

3. One separate directory to be created with the name “robust” to store the forecast for all the iterations.

4. Forecast to be saved in a .csv file with the name submission affixed with the iteration number.

5. Separate python function/module to be created, which will call the model to generate the forecast for different periods as explained in the above image. This is required for code modularity.

6. The function/module for the robustness, should have below input parameters.

   1. Start date of the training window of the iteration 1. (April2019)

   2. End Date of the training window of the iteration 1. (March-2020)

   3. Forecast period i.e number of months to forecasts. (12 months)

   4. Number of iteration. (12 iterations to be done)

   5. For subsequent iteration Train/Forecast start and end month should be automatically calculated based on the input given in step A and B as shown in the above image.

7. While running this module, you should use the final model based on training data till September 2019. For an example if it’s an ARIMA model then p,d,q values should be the same throughout all the iteration. If it’s a LSTM then hyper param such as epochs, number of LSTM units, look back/look forward etc should be the same as your final model built for date range mentioned in point 8. All the iterations should run on the same parameter configuration.

8. Robustness will be calculated based on MAPE over 12 iterations.

Judging Criteria

Your solution will be evaluated in a hybrid of quantitative and qualitative way.

• Effectiveness (80%)

  • We will evaluate your forecasts by comparing it to the ground truth data. Please check the “Quantitative Scoring” section for details.

  • The smaller MAPE the better.

  • Please review the targets and thresholds above as these will be included in the scoring.

  • Along with MAPE, we will calculate the robustness(at our end) based on the forecast generated for 12 iterations.(as explained in Quantitative Scoring point 2 section )

• Clarity (10%)

  • The model is clearly described, with reasonable justifications about the choice.

• Reproducibility (10%)

  • The results must be reproducible. We understand that there might be some randomness for ML models, but please try your best to keep the results the same or at least similar across different runs.