1st place - $800
2nd place - $400
3rd place - $200
We currently have 4 working algorithmic based models for a forecast model, and potentially a fifth. All five models were derived using the same data sets but each using a different model technique to derive their respective solution.
Different approaches were used to undertake Feature Analysis on various models with mixed results and certainly limited insight / explainability. The purpose of this exercise is three fold:
Interpretation Model: Complete model-agnostic Feature Analysis to compare and contrast the Feature Importance of the various models. The results will be used to generate Business Insight to improve confidence in the model(s) through explainability.
Next Steps: Steer refinement of the next phases of the development of the forecast model by validating which models should form the basis for further iterations and steering further sourcing of potentially valuable datasets to improve accuracy in subsequent phases.
Benchmark: Create a benchmark for the project from which the project can build and track progress and refinement of the course of the project.
In this challenge, we have 5 target variables for each Product.
Note: Derived Feature vs. Original Variable
Note that, in each model, the "derived features" might be different than the "original variables". For example, you may see a feature like “The previous/delta value of the variable X”. We would like to see importance analysis at the “original variable” level, instead of the “derived feature” level. We would like to see some conclusion like "variable X is critically impactful (positively or negatively) to variable Y".
In this challenge, we require you to do the following things.
For all models and target variable listed below, complete SHAP analysis to sufficient depth of variables including standard ‘scatter’ plots, mean absolute Shapley ranking, and partial dependence plots (PDP) for key variables. Extra value will be attributed to PDP’s that bring our significant second order impacts.
Provide an explanation of the key variables, particularly if dataset has been significantly transformed during the modeling process
Propose and complete alternative model-agnostic Feature Importance Analysis if it is deemed to generate a better interpretation model.
SHAP Analysis / plots are required. An example of SHAP plot is as below.
Example SHAP Summary Plot
Example SHAP Feature Importance Plot
Example SHAP Dependence plots - please include for all Target variables
Example SHAP Dependence plot - with second order impact
Example SHAP explanation force plots - please create these for all predicted values for the 5 Target variables and 3 products.