Well, hello, Internet and welcome to part two of my Time series forecasting tutorial.
In this part of the tutorial, we're going to focus in on the Arima model.
Now Arima stands for Autoregressive Integrated Moving Average and it is the basis for many other models.
And this is all about predicting the future based off of past events.
And what it's going to try to do is focus on trying to fit the data as well as possible by examining
differences between values instead of the values themselves.
And as you're going to see, Arima works very well when data values have a clear trend and seasonality.
And we can only make predictions, however, based off of the data we have and any outside effects not
in the data can't be used to make predictions.
So for example, we could make predictions on stock prices, but since we don't know when a recession
may occur or some other dramatic event that can't be modeled, and there's going to be another thing
called Sarima, which is based off of seasonal.
And we're going to talk a whole lot more about it at a later video.
And as you're going to see, the A R is going to deal with auto regressions, the I going to deal with
integrated issues and Ma stands for moving average.
And what I'm going to try to do in this video is make you completely understand auto regressions.
Then we're going to get a little bit more into the integrated part of Arima and more on it later and
then moving averages will be covered later.
So I'm sort of breaking this down because it can become extremely confusing.
So I want to slowly ease you into the process.
And along the way I want to teach you a whole bunch of things, like how to grab dates in specific ways,
how to set frequencies, how to handle whenever you have missing data and a whole bunch of other different
things.
And I have a lot to do.
So let's get into it.
Okay.
So I'm going to start off with a very simple regression example.
What this is going to be based off of is whether tip amounts are going to increase based off of the
size of the bill, and everything that's available here is available in a link in the description on
GitHub for free.
So what I'm going to do is I'm going to go and get a seaborne data set which I have used in past tutorials
and it's just a whole bunch of tip data based off of sales.
So we are going to size this figure that we are going to plot out and I'll make this 12 by six because
that works.
Now if I come in and I say I want to plot this tip data, what it's going to do is create a regression
line.
It's going to be based off of the total bill amount and Y is going to be based off of the amount of
tip.
And this data is going to come from tips, which is what we have just imported.
And if I run that, you will see what is called a regression line.
All right.
So basically the way these are created is what we do is we take all these different points, we ignore
some of them, and we try to move the points that are underneath up and then the points that are above
down.
And by doing so, we are able to create an average which is called a regression line.
All right.
So that's what it is.
It's basically averaging out all the points as we go up the x axis and find an a happy average in between
all of them based off of all of the previous results.
Now, the difference with Arima is we are going to be dealing with time series on the X axis.
So what I want to do and I'm going to cover some things where Arima works really well and I'm going
to cover some situations where Arima doesn't work quite so well just so you can sort of see an example.
And I know there's a lot of interest in stock market stuff, so I'm going to go and show you what happens
when you apply Arima to the stock market.
So what we're going to do here first is I am going to pull in some weather data for Pittsburgh, which
is where I live, and it is just a CSV file which is also available on GitHub.
And along the way I'm going to sort of cover a whole bunch of things that can cause confusion.
Now, if I want my index to be based off of the date column in my CSV file, that's how I do it.
And parse dates is going to be set for true.
Then I'll be able to automatically get the date column and place it in there and it will be used as
an index for all of what I'm going to be doing afterwards.
I'm going to go now and there's going to be two.
Columns on this CSV file that I do not want.
And this is how you drop columns.
So if I do not want temperature minimum and I do not want precipitation information, well that is how
we go in and get rid of that data.
And if I want to come in and look at what we got, this is what we got.
And guess what?
Oh, boy.
There's a whole bunch of nan values there.
So we are going to definitely want to solve those problems.
Now, what I want to do is I want to get just weather data since 2010.
So how do I go about doing that?
Well, I go and get our data frame equal to our data frame.
And I'm going to say that I want to look for indexes in which they are greater than 2010 and this will
be January 1st.
All right.
So I got that information.
Another thing that comes up constantly is you have to set your frequency and this is how you do that.
So data frame as frequency.
And I'm going to set it to days because that's what I have now.
If I come in here and go pit DF and index, you're going to see frequency is set to day right there.
All right.
So that is good information to know.
Believe me, when you're starting off, this can cause confusion.
Well, what about situations in which I have empty data?
How do I handle that?
Well, what works out very often is to fill in any empty data with the data value that preceded it.
So that's what I shall do.
I'll make this equal to F fill, and now we won't have any empty data.
And you may say, Well, what if I want to know if I have any empty data values?
Well, this is how you check that values dot any and run it and it comes back as false.
So now we know we don't have any garbage values inside of there.
So that's fantastic.
Also to know now what I'm going to do is I'm going to use Seaborn to plot this out.
I'm sort of using different plotting options just so you guys can see different things.
All right.
So there you are.
I'm using a it says dark grid.
It's basically a gray.
And I want to add automatic date time converters for this.
And how you can do that is like this.
So we're going to say register matplotlib, which is what we are using to do our plotting.
All right.
And what else?
I'm also going to set out my default figure size, which is going to be the area.
Whoops, that's dot RC.
This is going to be the area where we go and we print our plots.
And you do this with fig size equal to and I'm going to use 12 by six.
What else do we want to do?
Well, let's say that I want to come in here now and I want to plot my daily max temperatures.
I'm going to pass in my figure, my axes and plot dot sub plots.
And then how I can go in and print is I can say axis is equal to pit df and plot with set axes and run
it.
And you can see all of the max temperatures for every year since 2010 up to 2020.
In Pittsburgh.
All right, so the good stuff, what I want to do now is I want to create our model using our entire
data set.
So I'm going to come in.
And how we do that is auto reg and pit DF and I'm using the temperature max amount or column, and I'm
going to say that I want 30 days to be used and then I'm going to fit to this model and I'm going to
get more into different options that are available when creating these models as the tutorials continue.
See, I'm sort of trying to ease you into this rather than just throwing a whole bunch of things at
you and you just say, What does all this mean?
Well, now what I need to do is I need to define my data that I want to use for training and the data
I want to use for testing.
So to do so, I need to know how many pieces of data I'm working with.
So that's 3651.
Okay, so that's good.
So I want to train with the first 80% and then I want to test my model against the last 20%.
So that will tell me if my model is any good or not.
And I can just tell you right now that it's not going to be perfect because we are not dealing with
data that is perfect for this type of study.
But I want to just show you what it looks like, because I think the only way to learn is to jump in
there and just play around a lot.
And I'm going to give you all the tools you need to go in and do so.
So this is basically the first 80% of the data and this is going to give me everything thereafter to
the end of the data available for testing purposes.
So now what I want to do is define my training model and auto and by saying train DF and I'm interested
in the temperature, Max, and let's say I want to do like 730 days worth of data, I can just go and
throw fit inside of here.
And you're going to be able to define here that I want to use White's covariance estimator and how do
we do that is and covariance is just a measure of how much different data points converge, how much
they move apart from each other or how far apart they are and how we do this is h c o All right.
And we can go and plot that.
And as you can see is our empty grid, but we will fill it up.
Now, what I need to do is define where my starting point is going to be to start making my predictions.
So my predictions are going to begin where the training data ended.
So that will give me the upper value.
And then I'm going to say that I want it to end where the training data began and plus the length of
the testing data area.
And that will be minus one because it starts at zero.
And now that I got all that set up, I can start making my predictions and we can see how good our predictions
are based off of actually comparing it to real life data.
And we say predict and we say start at the defined start and end at the defined ending point.
I'm going to mark this as dynamic, true, and now we can plot and see what's up.
So we're going to say axes at and I want to plot out the testing data which I will be comparing my results
to, and I will also be plotting out my prediction so that we can see how that compares to the actual
result.
And if I run it, you can see here blue is the actual results and orange is my prediction, as you can
see, because everything is so random, it is hard to make an extremely accurate prediction.
But you can see still that it is fairly accurate.
And as we add in additional parameters, as this tutorial series continues, we're going to get more
and more accurate.
Not perfect, because Arema isn't specifically aimed at this type of data I will show you as the tutorial
continues what type of things it does work great with.
But I just wanted to start off with things like weather data and stock market data so that I can really
get you interested in it and moving forward.
Now, since we are talking about the stock market, why don't I come in here?
I'm going to Another thing you can do here is I can just come in and say copy cells and then I can say
edit paste cells below because we're going to be doing very similar things with Apple stock.
All right.
So I'm going to come in here and I'm going to go and get some apple price data just to show you how
Arema works in comparison to apple price data.
Our stock price data.
I'm going to change this to a apple DF like this.
And then just to save time.
Well, let's see.
I don't need to drop temperatures and things like that from this so I can go and get rid of all of this
information.
So that's gone.
I am still going to be basing all of my stock information on the day, so I'm just going to come in
here.
And I'm going to change this to Apple DF instead of pit DF and am I going to fill missing values?
I'm going to say that I want to do that.
So I'm going to say, get rid of this and get rid of this.
I don't need to check if it's empty.
I already know that it isn't.
I can leave the plotting be exactly the same.
I don't need this guy right here.
This is going to have to be changed to Apple DF And this is going to be changed to price because that's
the name of the column for our CSV file.
With all that information, this can all remain exactly the same.
I'm going to need to find out how big the Apple data is.
So let's throw that there and let's go and comment that out.
Run it.
So I have 345 days worth of data on Apple.
What does that mean?
Well, we can come in here, change this to Apple DF, change this to apple DF change this to well I
need 80% of 345 and that is going to be 276.
So change that to 276 and then change this to 276.
Okay, so all that should work.
Great.
Now this is going to be Apple again.
Let's just go and copy this whole thing right here.
We're looking at apple price changes and let's say that I want to use 100 days.
Everything else here can stay the same.
This can stay the same.
Get rid of this.
Get rid of that.
I think everything else is fine.
And let's go and do the test versus the prediction.
And you can see right here that we came in and the blue is going to be the actual price and the orange
is going to be our prediction.
So this is focused in just on that specific data, let's say I would like to pull and plot out all of
our data, though, that we have on Apple.
So I'm just going to come in and go Apple DF dot plot a x equal a x run that and I accidentally typed
in Apple instead of Apple.
And you can see now here is all of our price data and you're going to see that the actual test data
or the actual data is in orange and the green data is based off of my prediction.
So yes, it's off, but is it off by really all that much?
Really not.
Not that bad.
So this just gives you a I told you that Arema is going to work perfectly with something as completely
erratic and crazy as stock prices, but you can see that it still gives you a rough estimate.
But what would happen if we would decide we wanted to make predictions 60 days into the future in regards
to Apple stock price?
Well, we could do so.
We could say train, underscore, model, predict and start is going to be the end and the end is going
to be equal to the end.
Plus 60 days.
And again, dynamic is equal to true.
And we can now take this and go and plot our forecast into the future and see what we see.
So is Apple price going to go up or down from this point in time?
And our prediction is that it is indeed going to dip, but then it's going to rise even more.
All right.
And I don't know if that's what happened.
You could go check that on your own to see if in 2015, about a week into February, if Apple's price
was anywhere near, let's say, 122 or something like that.
Okay.
So there are three examples of using regressions and how Arima works with different types of data and
different levels of accuracy depending upon how chaotic the data can be.
And now what I want to do is I want to talk about the integrated part of Arima, the AI that is right
here.
Basically, integrated integration is going to focus on replacing values with the differences between
previous values and Differencing is the computing of the differences between the consecutive values
and the whole reason we're doing this.
The goal is to make the Time series stationary, and a stationary series has a somewhat constant mean
as well as variance.
So what I want to do to finish this tutorial off is to make sure you're completely 100% aware of what
I mean by a stationary series.
Because to have.
Arema work properly.
Your data must be transformed to stationary form so that it can be best evaluated.
And we're going to use Ad Fuller to come in here and verify if data does fit that stationary goal that
we are looking for.
So I'm going to use my pit data again, my Pittsburgh weather data, so I might as well just come here
and copy it.
There it is.
So let's go.
Copy this.
Come down here.
All right, so there is our Pittsburgh weather data.
Now, let's say that I would like to come in here and flatten it out a bit so that I can get a smoother
sort of chart.
Well, I can say pit df dot rolling, and I'm going to say the window is going to be based off of 60,
which would be two months.
And I want to get the mean of this and specifically for t max temperature.
Max, Another thing I'd like to do is let's say that I would like to focus in just on the dates between
1985 and the year 2000.
How do I do that?
Well, I can mask off dates and then return everything that that is masked off in a brand new data frame.
So I can say mask is equal to see.
I told you I was going to be covering a lot more than just the basics of a moment.
I'd be actually teaching you how to use this software rather than just throwing things at you and hoping
that it looks pretty or it impresses you.
I actually.
I'm doing my best to teach you.
Okay.
So we said from 1985 to 2000.
Well, and I'm going to say data frame to index less than or equal to and 2000.
All right.
So hopefully that makes sense.
All right.
And then what I can do is just say DF two is equal to DF two and apply our mask just like that.
And I can come in here and plot it just so you can see that it worked right like that.
And if we look down here, we can see that, yes indeed, it starts in 1985 and goes to the year 2000.
All right.
So good stuff.
Now, what are we going to do?
Well, I want to also go and get some US birth rate data so that I can show you an example of non stationary
data.
I want to show you a couple different things.
So let's do this PD and read CSV and I have a whole bunch of information on different age groups and
so forth and so on.
And when they have children and this is called US birth rates CSV and I'm going to go and get index
column.
Well, this is going to be actually slightly different.
It's going to be based off the year.
So I want to show you a whole bunch of different data, though.
All right.
And let's change this to year and I can come in and plot this out if I'd like.
Usbr DF And let's say that we want to chart out the number of children born by women who are between
the age of 30 and 34, There you are.
Well, I have to get rid of this plot first.
All right.
So that that makes more sense.
Okay.
So you can see that since 1940, in the United States, women on average used to have about 85 kids
per thousand between the ages of 30 and 34.
And it went down dramatically and then it went back up over 100.
All right.
So kind of interesting.
Let's go and get our Apple stock price data.
This is a different one.
This is not Apple to throw that in there.
To show you another non stationary example, read CSV.
Actually, let's use the data that we had before and let's have it be Apple to CSV.
So it's exactly the same file.
I don't want to cause any more confusion.
We can come in and do Apple plot and that most definitely is not stationary and I'm going to show you
how to prove that different data isn't stationary.
So how we are going to do that is what we're going to use is what is called the augmented Dickey-fuller
test.
And what it does is it tells us whether our results are stationary or not.
So I'm actually going to create it inside of here ADF test and it's going to be passed a Time series.
And let's just print this out so it looks.
Nice and professional.
Dickey Fuller test.
DF test is going to be equal to and how you call it is with add folder you pass in the time series you
want it to analyze and auto lag we are going to set for AIC.
And what AIC does is it aims to minimize corresponding information and then we'll say if output is equal
to PD series DF test and we're going to grab four data values.
So that's how we do that.
And we're going to index these with what is being returned, which is the test statistic.
And I'll explain what all of these mean.
So test statistic P value.
This is the important thing and I'll explain to you why.
Talked about P value in the past.
Also in my statistics tutorial, I'm also going to have the number of lags.
I'll get more into lags in the next tutorial series.
Basically all lag is is a comparison to data that lags previous to it.
So you could say one previous data point and all you do is get the previous piece of data or five data
points ago lag, you know, it comes after it and number of observations used.
Okay.
And then we can print out all of this information.
DF And then I'll explain to you exactly how it can be used to tell if data is stationary or not.
All right.
So let's go and let's go ADF test.
And the very first thing I'm going to throw inside of it is going to be my weather data for Pittsburgh.
Whoops, I forgot to put brackets here.
So throw a bracket there.
Throw a bracket down here.
There we go.
All right.
So we went and we ran our weather data here.
And I just want to focus really in on the p value.
Now, if a P value whenever you run dickey-fuller on it, is very small.
What that means is that there is a very low probability that our results are random, which means that
this is pretty seasonal data.
And if you look at it versus this is Apple obviously not seasonal.
So let's go and look at this data, which is our Pittsburgh data.
You can see that's definitely seasonal, up and down, up and down, up and down, just as you would
expect with the rising and falling of temperatures.
But what happens whenever we would go and use non-stationary data like I've referred to numerous times
with the Apple stock price, Well, let's just throw it in here.
So DF Apple and price is what we're looking for.
There we go.
And in this situation you can see that our P value is not small.
All right?
So that means it's not stationary, which we already know.
All right.
So there you go.
That is an introduction to the Arima model.
And we're going to be getting more into it as the tutorial continues.
We're getting into advanced stuff now.
Hopefully you guys like it.
If you did leave a comment down below and like always, please leave your questions and comments down
below otherwise till next time.