Thread by Mark Tenenholtz
- Tweet
- Mar 7, 2023
- #Statistics #Scientificmethod
Thread
Time-series forecasting is becoming a lost art.
It’s the engine behind the $5,000,000,000,000 retail industry in the US.
And yet, research progress is slow and the tricks of the trade are locked away.
Here’s what you need to know:
It’s the engine behind the $5,000,000,000,000 retail industry in the US.
And yet, research progress is slow and the tricks of the trade are locked away.
Here’s what you need to know:
1. EDA
The best way to start any ML problem.
Here are some things to look at in time series data:
• Seasonality (monthly, daily, etc.)
• Trends (big for proper evaluation)
• Autocorrelation
• Diff. between older and newer series
• Tons of raw samples
The best way to start any ML problem.
Here are some things to look at in time series data:
• Seasonality (monthly, daily, etc.)
• Trends (big for proper evaluation)
• Autocorrelation
• Diff. between older and newer series
• Tons of raw samples
Don’t forget to overlay other variables than just historical factors, though.
Often what appears to be a random fluctuation has an obvious cause.
Price shifts, advertising campaigns, etc.
Often what appears to be a random fluctuation has an obvious cause.
Price shifts, advertising campaigns, etc.
2. Benchmarking
Is an MAE of 96.05 good?
What about a MAPE of 36%?
There’s no way to know unless you create a dumb benchmark.
Predicting the mean is decent, but keep in mind that even a naive forecaster could outperform that usually.
Is an MAE of 96.05 good?
What about a MAPE of 36%?
There’s no way to know unless you create a dumb benchmark.
Predicting the mean is decent, but keep in mind that even a naive forecaster could outperform that usually.
Try things that better consider the domain, such as a rolling mean, predicting the value seen in the last day of available data, etc.
Then, use the performance of that baseline to benchmark your models down the line.
Then, use the performance of that baseline to benchmark your models down the line.
3. Domain expertise
If available, ALWAYS engage with someone that understands the underlying operation and dynamics at play from the business side.
There can be spikes caused by collection issues, strikes, out of stocks, you name it.
A 5 minute convo can save you 50 hours.
If available, ALWAYS engage with someone that understands the underlying operation and dynamics at play from the business side.
There can be spikes caused by collection issues, strikes, out of stocks, you name it.
A 5 minute convo can save you 50 hours.
4. Model evaluation
Any answer other than roll-forward/expanding window cross-validation is wrong for 99.999% of problems.
Do NOT do any sort of random split on your samples that doesn’t ensure future values aren’t included.
That’s a death sentence for production.
Any answer other than roll-forward/expanding window cross-validation is wrong for 99.999% of problems.
Do NOT do any sort of random split on your samples that doesn’t ensure future values aren’t included.
That’s a death sentence for production.
And don’t forget evaluation metrics.
MAE and RMSE are fine.
MAPE/WAPE have their place.
But if you aren’t using scaled metrics like RMSSE and MASE, you probably aren’t capturing your model performance very well.
MAE and RMSE are fine.
MAPE/WAPE have their place.
But if you aren’t using scaled metrics like RMSSE and MASE, you probably aren’t capturing your model performance very well.
Even better: deep dive the underlying business operations so that you can directly convert your forecast accuracy into $$$ uplift for your business.
5. Now, we're gonna get into the fun stuff: modeling techniques!
“State-of-the-art” largely depends on business use-case. I’ll give something for everyone here.
“State-of-the-art” largely depends on business use-case. I’ll give something for everyone here.
Statistical models
In low-data environments, it’s hard to beat good ol’ ARIMA and ETS.
But that’s not the end of the story. There have been some innovations in the form of the Theta model, STR, and many others
In low-data environments, it’s hard to beat good ol’ ARIMA and ETS.
But that’s not the end of the story. There have been some innovations in the form of the Theta model, STR, and many others
LightGBM/XGBoost
If your company has any decent amount of data, you probably won’t beat this.
Combine rolling/lag features, grouped aggregations, and all of the external variables you can get your hands on and there’s a chance you can slap one model onto your whole enterprise.
If your company has any decent amount of data, you probably won’t beat this.
Combine rolling/lag features, grouped aggregations, and all of the external variables you can get your hands on and there’s a chance you can slap one model onto your whole enterprise.
LSTMs/GRUs/Transformers
If you have a MASSIVE amount of data and spend enough time tuning the model, you can beat LightGBM/XGBoost sometimes.
The same feature engineering techniques work, but keep in mind that these models are much more finicky about the scale of the data.
If you have a MASSIVE amount of data and spend enough time tuning the model, you can beat LightGBM/XGBoost sometimes.
The same feature engineering techniques work, but keep in mind that these models are much more finicky about the scale of the data.
6. Combining predictions
Time-series data changes its behavior all the time, so ensembling multiple models usually gives much more robust results.
Once you’ve found your best single model, try all of the other ones on the previous list and ensemble some of them for diversity!
Time-series data changes its behavior all the time, so ensembling multiple models usually gives much more robust results.
Once you’ve found your best single model, try all of the other ones on the previous list and ensemble some of them for diversity!
Hierarchical Forecasting is also incredibly powerful for making your forecasts more coherent.
Not only will it make your forecasts and different levels agree with each other, but it can also give a nice accuracy boost!
Not only will it make your forecasts and different levels agree with each other, but it can also give a nice accuracy boost!
7. Production
The main concern you'll have is data drift. Make sure you have a monitoring system that you check in with on a regular basis.
Personally, I love adversarial validation to identify data drift. It's one of the most powerful and underused techniques in ML.
The main concern you'll have is data drift. Make sure you have a monitoring system that you check in with on a regular basis.
Personally, I love adversarial validation to identify data drift. It's one of the most powerful and underused techniques in ML.
Topic TL;DR
1. Domain-specific EDA
2. Domain-specific benchmarking
3. Stakeholder relationships for anomalies
4. Model evaluation
5. Pick your model from data volume
6. Combining predictions
7. Monitoring in production
1. Domain-specific EDA
2. Domain-specific benchmarking
3. Stakeholder relationships for anomalies
4. Model evaluation
5. Pick your model from data volume
6. Combining predictions
7. Monitoring in production
I’m teaching a course on this that will go over all of these topics in detail.
(And it starts next Monday!)
Use the code FORECASTINGML10 for 10% off! corise.com/go/forcasting-with-machine-learning-EBFH4
(And it starts next Monday!)
Use the code FORECASTINGML10 for 10% off! corise.com/go/forcasting-with-machine-learning-EBFH4
