Predictive Analytics: Knowing When to Update Your Model

As much as you may not like it, your predictive analytics job is not over when your model goes live. Successful deployment of the model in production is no time to relax. You'll need to closely monitor its accuracy and performance over time. A model tends to degrade over time (some faster than others); and a new infusion of energy is required from time to time to keep that model up and running. To stay successful, a model must be revisited and re-evaluated in light of new data and changing circumstances.

If conditions change so they no longer fit the model's original training, then you'll have to retrain the model to meet the new conditions. Such demanding new conditions include

• An overall change in the business objective
• The adoption of — and migration to — new and more powerful technology
• The emergence of new trends in the marketplace
• Evidence that the competition is catching up

Your strategic plan should include staying alert for any such emergent need to refresh your model and take it to the next level, but updating your model should be an ongoing process anyway. You'll keep on tweaking inputs and outputs, incorporating new data streams, retraining the model for the new conditions and continuously refining its outputs. Keep these goals in mind:

• Stay on top of changing conditions by retraining and testing the model regularly; enhance it whenever necessary.
• Monitor your model's accuracy to catch any degradation in its performance over time.
• Automate the monitoring of your model by developing customized applications that report and track the model's performance.

Automation of monitoring, or having other team members involved, would alleviate any concerns a data scientist may have over the model’s performance and can improve the use of everyone’s time.

Automated monitoring saves time and helps you avoid errors in tracking the model's performance.