Of Internet.
And welcome to part one of my series on Time series forecasting.
Over the next couple videos, I am going to show you many of the different ways we can use data to predict
the future.
And because you guys are so interested in the coronavirus, everything's going to be based off of the
coronavirus.
Or at least this first video.
Will and I have a lot to do, so let's get into it.
All right.
So here I am in Jupyter Notebooks, and in the description underneath the video you can find all tutorials
on all this stuff.
So numpy pandas I'm using for data matplotlib for plotting.
I am going to use statsmodels for all of the awesome scientific calculations it can make for us.
Seaborn and Plotly are going to be used for plotting, of course.
And then we have beautiful soup for going and manipulating URL data Json for manipulating it as well.
And more on Pandas.
This whole entire notebook is available also in the description 100% for free on GitHub.
All right, so now we're getting into the meat of it and basically you're going to choose different
models for forecasting the future depending upon the data you have.
And what we're going to do in this tutorial is use the Holt-winters method.
And mainly I'm using it because I already talked about it in my time series video.
You don't need to watch that right now.
Everything's going to be covered here.
Basically what we're going to do is we're going to train our model on the first group of data.
So this is going to be referred to as our training data, and we'll use the first 80% of the data.
And then we're going to test the model using the last part of our data, meaning the leftover 20%.
And all along throughout this process, we'll be modeling using real known true results.
And it's important to understand that whenever you're forecasting in the future, you should use the
same length of time that you are going to test your model against.
All right.
So I'm going to be importing a whole bunch of Covid 19 data.
This is a CSV file and I talked about it in my very last video.
And if you want to see the type of data that you have inside of this CSV file, you can just type in
columns and it's going to show you all the different ways we're going to be able to analyze data.
But what I really want to focus in on in this tutorial is new deaths per million.
So that's what we'll model out and make predictions about.
First thing that I want to do is create a numpy array that is going to have a list of all of the countries
inside of it.
So this is location is going to reference the countries and unique is going to give me only the country
name one time.
Then what I'm going to do is I am going to get indexes specific for the United States is who I'm going
to be analyzing.
You can go off and do whatever country you want.
However, just by plugging in a different country name here instead of the United States for location.
So I'm saying that I want just United States data here and then I'll say index.
Now what I can do is get data frame location and the column data for our country name using our indexer
that I have here.
So I'll have location and we'll use the indexer to match up against the United States.
And then I'm going to say that I want date and new deaths per million.
I'm going to come in and delete out any empty values that I have inside of this with Drop n A, I'm
going to say that I want my index to be whatever the date is for the data.
So date and then in place is going to save that inside of our data frame that we have here.
I'm then also going to say that I want to drop anything that isn't based off of new deaths per million.
And how we do that is say columns difference and then new deaths per million is what we want to go down
to the next line.
And again, you have to do in place to save this change that you're making to the data frame.
And now I can come in here and plot this data just by calling plot and figure size, and I'll make this
12 by six because that looks pretty good.
And I spelled columns wrong, so.
Let's fix that.
And there you can see is our plot.
So basically, this is all the data that we have available to us as of right now.
So what I want to do is use this data, or at least the first 80%, create a model of the future, then
test it against what really happened and then make predictions in the future.
Thereafter.
I can also come in here and plot out a moving average by saying rolling and based this off of a 30 day
period.
And again, we're specifically looking for new deaths per million miles.
I'll just copy this and then we can plot that out also.
So you can see after the first 30 days, we're then able to come in here and print an average of everything
that precedes.
So pretty cool.
Now to use Holt-winters to make different predictions here, we are going to have to set the frequency
for our data and this is going to be daily.
So as frequency and set this to D for day, we can come in and see how many different data points we
have just by saying info.
And we're going to do this to make sure that we get the first 80 to train and then the last 20% for
testing.
So we have 241 entries.
So what we're going to do is say training data frame is going to be equal to the full data frame.
And I'm specifically targeting the location from 100 or up to one.
It's actually 191.
So this is what will train our model on.
And then we will have a test data frame that we will test against.
And so this is going to go at 191 up to the end.
All right.
So now we're ready to come in here and actually model this.
If you don't remember, the Holt-winters method is going to model based off of three aspects of a time
series being the average, the trend and the seasonality.
If you want to know more specific like a lot of specifics about what exactly it's doing, look at part
four of my Time series tutorial series.
And it it just totally focuses in on it.
But basically what it is doing is it's going to try to predict the future provided the series is seasonal
and repetitive over time and this is how we import it.
And then we need to define how our model is going to be set up.
So I'm just going to go and get exponential smoothing, throw it inside of there and this is going to
be my training data frame.
So again, going to use new deaths per million and paste that in there.
And then I'm going to say trend and I'm going to use the you either have the multiplicative or additive
and normally multiplicative is going to be used whenever you're seeing exponential growth in your data.
So, for example, this is not an example of that.
This is sort of flatlining and so forth.
This would be an example of sort of exponential growth.
So if our chart looked like this and it just kept going up in this direction, then we would use multiplicative.
But since it isn't growing, then we are going to use something that is additive.
And when we're talking about seasonality or basically talking about the frequencies going up and down
like that and trend a trend is this moving average is a trend.
So if we would be going up like this, that would be a trend line.
All right.
So there is the different parts that we're going to be using.
So that's the difference between using additive and multiplicative.
And seasonal is also going to be additive.
And then we need to decide how many periods we want.
I am going to focus in on 14, 14 days.
And the main reason is because that provides me with the best results and that's something you have
to play with to see which works out better for you.
You would think seven days would work, but actually 14 works better with this specific model.
And now what we're going to be able to do is make predictions 50 days into the future for the coronavirus.
So you're going to know at the end of this tutorial what the coronavirus death rate is going to look
like in 50 days based off of this model.
And we can come in here and say prediction and you can see exactly and this is deaths per million what
we have here data wise.
But we're going to plot this out so you can actually see it.
Now, what I want to do is go and do that.
So I'm going to say training data frame new deaths.
Per million and plot and figure size is equal to and I'm going to use 12 by six again.
And if we run it, you can see right here new deaths per million based off of our training data and
exactly what that looks like in blue.
Then after that, I'm going to go and show our test data.
So change this to test DF and we can just leave this plot like this to plot it on the same chart.
So now you can see how the testing data meets up with our training data.
And then what we can do is come in and make our prediction and it's going to be a prediction based off
of the testing data that we have.
So I'm going to call this prediction.
Let's just go and get rid of all of this right here, prediction plot.
And I'm going to say that I'm specifically interested in very specific dates.
So it's going to be 2020 dash zero nine, dash zero 9 to 2020 dash 1028.
And the reason why I did that is to zoom in.
So you're going to see here that we have our in green is actually going to be our testing data.
Well, here if I do, let's get rid of this just so you can see the whole thing.
All right.
So I'm going to run it.
This is the entire period.
And you can see right at this point, the green is going to be my prediction while the orange is the
actual.
And you can see how close it is.
And this prediction is based off of the blue data, not off of the orange data.
So this is how accurate it was at predicting the future, using the blue data.
And then we're comparing the predictions versus the actual results.
And then what I did here was I actually came in and zoomed in like this so that we could see it more
closely.
So as you can see, it's fairly accurate in regards to predicting the future.
But you might say to yourself, well, how accurate is it and how, you know, can we come in here and
actually test this?
Well, I'm going to use scikit learn here.
So this is how you would install it.
And this is the first way that we're going to evaluate our prediction, which is to use the mean absolute
error.
And I'm going to look at a whole bunch of different ways of evaluating forecasts.
And basically what the mean absolute error does is it takes the difference between our predictions and
the true results and it finds the average of the difference between them.
And the major negative for it is that a few major errors made pretty dramatically skew our results.
So we're going to use a bunch of different ways to test it and you can install scikit learn with either
Conda typing this into your terminal or with Pip typing this into your terminal or command line to install
it.
All right, so let's come in here and let's test some stuff.
Well, why don't I actually just show you a function?
I'm going to make it so we can come in and say y one and y two and axis is going to be equal to zero
by default.
And I need to convert this into numpy arrays to be able to work with this data and this situation.
Anyway.
So to do that I have to go to numpy like that and then do that with Y two as well.
And this is just going to be passed in the test data and the prediction data.
That's what those are.
And then the axis, this part right here, if it's equal to zero, it's going to be testing per column.
If it was one, it would be testing per row.
And if I typed in none, it would give me a total average overall.
All right.
So after I convert these data frames into numpy, I can go and say NP mean in and then numpy absolute
value y one numpy minus y two numpy and axis is equal to whatever the axis value is.
And then I can come in here and call this test.
DF is what's passed in along with the prediction.
And then I could say none and I forgot to change this into Y two and there it is.
And there you can see is the difference in average versus the test versus the prediction.
Well, we can just come in here and have scikit go and perform this for us, so let's just go and use
that instead so we can go and get rid of all of this and just have a simple command that we can issue.
And that is mean, absolute error and then passed in test DF and.
Prediction.
And you can see this is the difference between our test data and our prediction data.
Another option for testing our predictions versus the test is the mean squared error.
And what it does is it takes the differences like before and finds the mean.
But since we square the results, large errors are squared, which can dramatically show those errors
in the model, which is both a positive as well as a negative because we'll be aware of the errors,
but they're going to mess up our results.
And the main negative is that the units squared are going to distort our results.
So that's kind of a plus or minus.
But I'm going to show you with the root mean squared how we can correct all that as well.
Now, if you wanted to write a function for how this actually works, we would get both of our data
frames and again, we would get an axis passed in and I need to convert this into a numpy array just
like before.
And we're going to get more and more accurate with these models as we continue this series.
And here what I'll do is say return y one numpy minus y two, and then we will square it and then we'll
take the mean of that axis and whatever axis value they throw in.
And then we can test this with our test data as well as with our prediction data just like before.
And none.
And there you can see the result we got there, but we already have mean squared error, so why don't
we just use it?
All right, So grab it, throw it inside of there.
I just really want you to understand how this stuff works.
And that's the reason why I'm showing you actually writing the functions.
And if we run this, you can see here is our mean squared error.
All right, So let's go and test one more way of testing our results, and that is the root mean squared
error.
And for this, I'm just going to come in and say square root and then grab this guy right here.
So we're just going to take the results from this and then go and get the square root.
So pretty simple and run it.
And there you can see is the difference between the test as well as the prediction.
And that leaves us with only one thing to do, and that is to predict the future.
So we're actually going to go and predict 100 days into the future based off of all of our data using
our model we created.
So to do that, we're going to use exponential smoothing.
Let me just grab this, come down here, paste it in, and this time we're going to base it off of the
entire data frame.
Still additive, still additive.
We're still going to use 14.
And then if we decide we want to predict 100 days into the future here, I can say this is my Covid
forecast and Covid model forecast and 100 days, and then we can print both the original as well as
our prediction tacked right on to the end of it.
And we can go and grab our forecast.
And now you're going to know what to expect for Covid 19, 100 days in the future.
Oops.
Make sure you put figure size equal and there you go.
So based off of using all of the data that we have so far, this is what the Holt-winters method predicts
for the deaths per million with the coronavirus into the future.
All right.
So there is your first step into being able to predict the future.
As this tutorial series continues, We're going to be manipulating time and all kinds of really interesting,
fascinating, different ways.
If you don't want to miss that, click your notification bell and like always, please leave your questions
and comments down below.
Otherwise till next time.