Well, hello, Internet and welcome to part five of my Time series Forecasting Tutorial series.
In this tutorial we've been talking a lot about Arima.
This time I'm actually going to show you the fantastic predictions it can make, and I'm going to cover
a whole bunch more and I have a lot to do, so let's get into it.
All right.
So like always, everything's available in the description on GitHub.
And these are exactly the same imports as always.
So now it's time to make some predictions with Arima.
What I did was I went and actually saved my Pittsburgh weather temperature information into a CSV file
and we're going to be using this over the course of this video.
But I'm going to need to import a couple new libraries because I'm going to be doing a lot more things.
So I'm going to statsmodels TSA Arima model and I'm going to import Arima and Arma and Arima results.
We might not get to every one of these in this video.
I am going to of course have more videos coming because there is so much to cover and are my results.
Okay.
What else are we going to need from again?
Statsmodels graphics, TSA plots and import plot, ACF and plot pacf.
And you may need to do Pip install in your terminal or your command line because we're using Pyramid
Arima in this tutorial.
So that's what you would type in.
But I already have it installed.
So then you're going to go here and say import auto Arima and there you go.
All right, so let's go and take this Pittsburgh weather information.
Oh, let's take this guy.
Just throw it up there.
All right?
And here you can see everything plotted out.
And this is on a monthly basis.
And you can very clearly see or now you can better see, I changed fig size here.
You can very clearly see that this is most definitely seasonal data.
All right.
So that's very important to remember.
Now, what Auto.arima is going to do for us is it is going to tell us what model we should use.
All right.
So I'm going to throw my Pittsburgh weather information in there and I'm going to mark seasonal as true
and trace also as true.
Trace is just going to go and give me all of the pieces of data as it's checking them.
All right.
And there's a little error here.
Don't worry about it.
And if we come the whole way down inside of here, you can see this is going to be the requested model
that we use.
I'm going to use Arima, though, with the same exact settings in this situation and I'll talk more
about seasonal and Rhema later.
Okay.
So this is the model we're going to use.
Now, what I want to do here first is I want to split the data into a training as well as a testing
set like I have done many times before.
Of course, we're going to want to know what the size is.
So that comes out to 480 and then we will say train df is equal to p df I lock and then we go and put
80% multiply 0.8 times 480 and you get 384.
So those will be the data points that we train on.
And then we will test against real world data points to see how accurate we are.
And then we say 384 like that.
Okay.
And that will give us the last 20%.
So what I want to do now is I want to create my model and I'm going to use Arima and train DF and the
order I am going to use is exactly what it told me to use, which is 202.
Then I need to fit my model so mod fit to get my results and then I can go and get the summarized forecast,
which is going to be the same as Auto.arima.
So res summary.
And if I run that, you can see exactly what we have right here.
And I've talked many times about what all of those things mean and now what I want to do is go and get
the predicted values to see how accurate my results are.
So to do that, I'm going to go length of my training data and then end is going to be.
The length of my training data, plus the length of my testing data minus one.
And I can make my prediction now.
So I'll say prediction is equal to I can get rid of that.
So that doesn't distract us.
And the prediction is going to be the results call predict start to end.
And let's say we want to rename this Arima prediction and then what we can do is come in and plot our
prediction.
So I'm going to say get our data that we have plot.
I'm going to set my fig size equal to 16 by eight.
And let's say that we want a legend.
And then after that I will print my prediction or plot my prediction and let's also Mark Legend true
for that.
And as you can see, if we go and plot our real data or our training data and then we actually make
our prediction, you can see here in Orange, that is the Arima prediction.
It is pretty accurate.
And you can go in there and zoom in and look at it and so forth and so on, but pretty impressive stuff.
And that brings us to Stationarity.
Now I've received a ton of questions about Stationarity, and I'm going to go in here and analyze everything
and show you a whole bunch of different ways to analyze stationarity and such.
But basically stationary data is going to have a constant slope or rate of change however you want to
refer to it.
And the reason you're going to want to use them is because they are easy to analyze and they are very
often required in many aspects of time series analysis.
And the main thing to remember is that for a data set to be stationary, the data must have a constant
mean as well as variance.
And we can test this by splitting the data and then testing for mean and variance.
So I'm going to say D set is going to be equal to and go and get my weather data.
Then what I want to do is I want to come in and get my midpoint.
So I'm just going to say split rounds, length of this data set divided by two, and then I can say
D set one is equal to and D set and split it right down the middle.
And then I'm basically going to get the other half, throw that in there, change this to D, set two
and then get rid of this and put it at the end instead.
And then now what I'm going to be able to do is go and get my mean for the first part.
So I'm going to say D set one mean and then we'll do the same thing to see if the means and variances
match up and just do that and then throw that there and we'll change this to variance one and then change
this to a variance.
And this is my Pittsburgh weather data when I'm playing with here.
And the variance for this is two and then this is also two.
And you can see right here that the means for this data set are a very close as well as the variance
is very close.
Not perfect, but this is definitely data that looks stationary and data that is largely stationary
in general.
And the Dickey-fuller test from before is going to show I have a test statistic.
Let's go and run this.
Well, I guess I have to go get that.
Da da da da.
Throw this here, throw that there and run it.
Okay, so there's our dickey folder and like I said, I'll make sure it's in the GitHub repository.
And after I run it, you can see right here that I have a test statistic of negative six.
And of course the more negative this value is, the more likely the data is to be stationary.
And if we come up here and actually go and get the size for this and run it, you can see the size comes
out to 480 and a dickey-fuller lookup table at sample size of 500 says that we can be certain 95% that
our data is stationary.
If the test statistic is less than -3.98 in all situations, well that's a negative six.
So while not perfectly stationary, it is very in 95% plus percent of the times.
It is going to work for us.
Now let's go and get some data that that would not be true for.
So let's just go and comment this out and let's go and get our flight data.
So I'm going to go and get Flight DF and I'm going to come in here and plug this in as well.
There we are.
There's all of our flight data and if I go and plot that out, you can see right here it is most definitely
not stationary and I can go and run an ADF test on it as well.
Paste that in there and get my flight data.
And most definitely whenever we come to the test statistic that comes out as 0.81 and also it has a
P value that is over 0.05, which shows that this most definitely is not stationary data, even though
we can look at it and see that it isn't stationary.
So let's come in here and analyze how we can make data stationary or more stationary.
So what I'm going to do is I am going to use pandas to differance the data frame and make the data stationary.
So let's throw this in here.
So I'll go p df dot df and drop n A which is going to get rid of any empty spaces and then I'll say
diff diff and plot and then also ADF to see if anything has really changed here dramatically Anyway,
diff DF and you can see the results that we have here.
So let's come back up inside of here and uncomment this and let's comment this flight data.
And you can see right here, our test statistic is a negative six.
You can see here it is a negative one.
However, you can see the P value is super tiny, -18 versus negative seven.
All right.
So by going and taking the difference, we were able to make our data more, more stationary.
And you can see that it doesn't really look that much different.
Well, let's go and change the fig size on it.
And there it is.
And here's the other plot.
So this is much smoother what we have here while this is more jagged.
All right.
So what it's trying to do is get the variance more in tuned, basically.
All right.
So let's take a look at our flight data.
And what we're going to do here is we are just going to set it up so that we get a stationary mean.
So I'm just going to call this diff to underscore DF is equal to flight DF diff again and drop an A
again and we can go diff to df dot plot throw fig size inside of here like that and then we can run
an ADF test on it.
Diff to underscore df comment this out.
Come up here, comment this out and uncomment this and throw fig size in here so that you can see exactly
what changed, which is basically the mean changes.
Okay, so previously we have this.
Then after performing the difference, now we have this.
All right?
So you can see we have a average mean across the whole thing.
But very often when I make videos like this, one of the number one questions I get is how does that
work?
How does Arima figure out that to make that Pittsburgh weather data plot two comma zero two?
How does it figure that out?
So what I'm going to do here is I'm going to show you.
All right.
So kind of show you I'm going to show you exactly what it's it's thinking.
So what I want to do here first is check to see that if there is a seasonal component to my weather
data.
So I'm going to go seasonal decompose.
This is our weather data and the model is additive.
It's definitely not multiplicative.
And then we can come in and get the results of that and plot it.
And here you can see our results most definitely seasonal.
I mean, there's no doubt this is seasonal data.
All right.
So let's comment that out then.
What I'm going to do is I'm going to go to Arima P DF and this time, even though it says it's seasonal,
I'm going to mark it as false just so I can show you what it's doing.
And I'm going to say trace is equal to true and then say we want a summary and you can see it's generating
all those different values for us here.
Okay, so what does it do?
It comes down here and it tells us in this situation, even though I told it wasn't seasonal, it went
and figured it out on its own.
But it gave us our best model as 405 right there.
So that is what it told us.
So how does it know that is true?
Let's come down here and let's think about that.
So the first question we might have is how is integration determined?
Well, if we come in here and we run dickey-fuller on this boom, in this situation, we see our P value
is under 0.05.
So whenever it's figuring this out, it immediately says, well, we don't need to do anything with
difference.
And hence you can see with the results right here, it did nothing with difference.
All right.
So that's how it figures that out.
So then what we need to do is we need to determine which is more significant autoregression or our mean
average.
Well, how we can do that is I talked about ACF and Pacf in the past.
We can throw this here, we can go PDF and I'm going to say lags is equal to 60 and copy this, paste
it inside of here and then just change this to P and run it.
All right, so here are our results.
Now if Pacf is going to have a sharp cut off, then we are going to add a AR to our model.
And there are between 2 and 5, and that's basically what we are going to then test for.
And that's exactly what Auto.arima does.
It checks 2 to 5 values and then it looks for our best results.
So you may say, well, how exactly do we then determine what our Mar weighting should be?
Well, in the situation in which there was a slow decline, let me go and show you what that looks like,
because we don't have that situation here.
So what I'm going to do is I'm going to go and get my Apple data Apple DF and lags and set that for
30 right like this and let's go semicolons in here so it doesn't double print these well in this situation
here's Apple and you can see here a slow decline now in a situation like that it is suggesting that
we use a AR over Mar, but this is not going to be true with our temperature data.
And what that implies to us is that we need to test for mar values that are greater than AR.
And basically there's just a lot of guessing here and that is basically the reason why we use Auto.arima.
So hopefully you guys see exactly the power of Auto.arima and the accuracy of it.
You have a pretty good idea of how it's making these calculations and hopefully you understand stationary
data more and the importance of it and how to convert your data sets to a stationary format.
And like always, please leave your questions and comments down below otherwise till next time.