what is going on everyone and welcome to
part 14 of our Python with panels for
data analysis tutorial series in this
part we're going to be talking about is
adding a few more days sets we're just
going to kind of run through them real
quickly as far as what we want to add on
top of what we're already kind of
grabbing and I think maybe what I'll do
is I'll move this mortgage rate down
here underneath
hpi woods underneath hpi benchmark and
then we can add a few more here now this
process now you can use this basic
process to add as many data frames as
you want biggest thing is you want to
make sure that you've got data that's
end of month if the granularity is any
less than a month like its yearly or
something like that you're gonna be
forfeiting a lot of accuracy and I early
screening larity of data and a lot of
data points which the machine learning
we'd like to have as many data points as
possible opportunity expecting a certain
extent of course more than like a
hundred thousand rows of data we
wouldn't really want that now we've got
mortgage 30 year and that covers
basically the interest rate and that
means we've got like two more major
factors here but really just one major
factor and that major factor would be
the economy the overall situation
basically so there's kind of two major
measurements of the overall economy one
would be the gdp so the gross mission
domestic product the other one would be
like the stock market
ok now theory the idea of of housing is
or real estate is to be you know you
would have stocks bonds and real estate
would be like a diverse portfolio maybe
throwing some precious metals or
something like that but that would be
the idea right but a lot so a lot of
people consider real estate to be on
diverse in the whole point of diversity
is so you have usually it's more like
negative covariance but negatively
correlated to is kind of important
because you basically want to be
when your diverse you have a slow but
steady increase in revenue every year
whereas if you're not divert say you're
only invest in the S&P 500 you go
through as much volatility is the S&P
500 stock market you know dow jones
whatever goes through so the reason why
people get diversify died is to smooth
out market fluctuations and also
decreased risk say you know the housing
market couldn't theory go down forever I
suppose her or whatever or at least for
a very long time so anyway moving along
whoa move go ahead and grab those two
metrics now in theory again the sp500
shouldn't be as valuable as her should
be not necessarily negatively correlated
but have low correlation like almost you
want to have you don't necessarily what
happens things that are like negatively
correlated but if they had zero
correlation that'd be great
anyway enough yet yip-yap yip-yap plus
chitchat anyway
Nicki ack anyway here we go so fine
sp500 data so this one the let's go
ahead and let's grab basically we want
this one comes with a bunch of columns I
think what we'll do you can promise you
could type it out if you want but we can
also just grab this data there's really
nothing to be gained
I don't think from us not copying and
pasting the data from the tutorial not
really sure why you would need to type
it out so i'm not gonna make everybody
type it out so just let me link in the
description and you can go to that code
copy and paste and at least I'll run
over it anyway so this was to grab the
S&P 500 data the S&P 500 data has open
high low close adjusted close so we want
the adjusted close i don't think it
matters with the S&P 500 index I don't
think there's ever a time when there was
like a split there but adjusted close
generally like for example Apple did
stock split
not too long ago and what happens when
they do a stock split is basically the
price got so high that to buy one share
you need like a thousand dollars they
don't really like that so what they do
is they say okay every
right now is actually two shares in to
buy a new shares half the price or
boards now for shares and to buy a new
shares twenty-five percent of what it
used to be so anyway stock splits
generally do not are adjusted for
through history a lot of times so you
have what are called adjusted clothes or
even sometimes you'll have adjusted hot
open high low close
that's the whole point that anyway then
we sample the month because you actually
get daily data for the S&P 500 and then
we're resampling adjusted close that
name of that column to the sp500 just so
it makes sense to us cool in place true
next up this is GP data this is just
gives us gdp information for the United
States from 1975 here we do the percent
change of course of course with the
percent change up here too
then we resample the month we rename
that column from value to GDP say that
the data frame is GP boo done and then
we also grab unemployment this is kind
of my idea was just you know hey like
that would probably okay metric pretty
hard to get up a home mortgage if you're
not employed so anyway I thought that
might be useful but who knows I've never
heard anybody reference the unemployment
rate being very strongly negatively
correlated with the housing price index
for theory of people aren't able to get
houses thousand residex ball cause less
the man who knows
so these are just a few now again
ideally you might get you know a lot you
might be best to have you know fifty to
a hundred possible data points here and
we'll talk about why probably in the
next tutorial but for machine learning
purposes you might want you want to have
a lot now we still could talk about that
pairs trading wear anything that
diverges from the hpi where the hpi is
going up but that housing market is
going down you want to invest in that's
actually probably a pretty good strategy
and its really simple and so just don't
forget about simplicity
really does went out a lot in the
markets so a lot of people really break
their backs trying to come up with crazy
strategies and stuff and sometimes the
easiest strategy to is not even try to
gain the system just just buy and hold
her hand the most tried-and-true method
it doesn't take any of your time
it's great but for data analysis
obviously there's a lot of the reasons
why you do data analysis besides
investing right lot of data analysis is
great for business analysis and
analytics stuff like that so anyway
can't really buy and hold that
information so here's all the data now
what we can do is we can add it just
like we did here now so let's look for
example we could say sp500 equals this
sp500 data function we do this copy and
then we'll just say sp500 equals that
then we can go ahead and do the gdp now
with GP we're gonna say this is actually
usgp I should mean that function but
we'll keep it for now equals gdp data
and the reason for that is you might
actually find that you get each state's
gdp and that might have a very high
positive correlation of housing price
index and it might even be futuristic
predictor so keep that in mind break
because that obviously the more money
that specific state is bringing in that
means that's more money that's being
spread out amongst all the people
working in that state so in theory that
means more money for houses and that
would in theory raise up the market
ok so you might want to consider doing
something like that obviously you know
the the idea of data analysis and
actually like pandas and stuff like that
you'll find a lot of times if you ever
kind of dive into this field it
literally is like you're playing in the
sandbox you think of these various ideas
you play them out you see how to do you
know and there's there's a million
things that we can do here and I could
spend you know we could probably have a
100-hour long
tutorial series but part of the fun is
just kind of thinking about yourself for
figuring it out so i'll let you guys do
it but this is it you just basically do
the same thing you just make these
functions and you go for it
so anyway that's GP and then we can also
grab unemployment again unemployment can
be by state so again that could be a
negative correlation factor so it might
be wise to get both of those numbers
anyway so us unemployment let's say
equals us on plant
ok so now that we have all these values
right we're joining in 30 but actually
what we can do is we can join a list of
values so we're going to join in 30
we're also going to join us one employee
and let's spell unemployment right there
you coming paste US gdp will take that
and we also want to bring in the sp500
ok so now we have all that information
we can do all kinds of stuff really by
this point but we can take what we can
do correlation because we look at the
correlation factors so no longer
let's see this is a hpi let's just call
this let's just call this
hpi ok and then we'll do
hpi here let's see what we want two
things let's go ahead first we can print
out
hpi dot for that might be interesting
also let's pray let me just create let's
just print out all of HP I and then
we'll print out core i just want to look
and see what we're dealing with here try
to think we would have any not available
data from the other datasets but
whatever
ok let's see if we spot a not available
data year yeah so we have some not
available data for this cone here
there for that's from obviously after a
certain date it looks like we just don't
aren't getting that data so some of
these don't have data beginning a
certain date or after a certain date
ok so that's kind of one of your kin
constraints if you use various in these
datasets so some of them don't go back
to 1975 and others don't just go all the
way through 2000 you know 13 I think the
oldest sets or 2 2013 i'm filming in
2015
but if you find that you're onto a good
track all of this data is available for
purchase
ok so either through cuando or like with
this stuff actually you could get the
zillow API there's other API to for
housing data so don't just you know you
can use this free data really quickly
easily but if you actually find that
you're onto something
there's someone will sell you the data
if you can't find it any other way
so anyway yeah what's the GP just
doesn't start until later we don't know
when but maybe 2,000 or something and
then so what we could do is we already
know we can just drop all na so we could
do something like this we could say hpi
drop in a play in true and it takes a
little bit because we're pulling all
those functions so as you might be able
to guess what we ought to do is Wow
uh-huh that's the intricacy ok so this
day is that right it starts at nineteen
ninety so we're losing you know a decent
amount of time here about 15 years so
and we're losing that because of GP so
that's a perfect example of like well we
just don't have GP data going back to
nineteen seventy-five at least through
the day that we found right but i
guarantee you can find some gdp data
that goes back to nineteen seventy-five
dick the real crux the hard part will be
finding monthly gdp data going back that
far but i'm sure you can find it so
anyway so what drop na we're gonna do
that because I
we're gonna feed this through a machine
learning classifier and I'm not really
too interested in and replacing 15 years
with no negative nine thousand so
finally the last thing that we're going
to do in this specific tutorial is the
following we want to go ahead and HP i
dot to pickle hpi dot tickle alright
send that to pickle run it and make sure
it actually goes to the pickle but it
should and once you're done there now
you're ready to continue on the next
tutorial in the next tutorial will be
incorporating scikit-learn you can
already install it if you want but we'll
talk about it's just pipping install sk
learn and then and what that allows us
to do is we've got a bunch of pre-made
machine learning classifiers for us and
we just pump this data through very
easily you don't need to know a thing
about machine learning and
classification also in the next tutorial
actually you can follow very nice and
easy flow chart to figure out what class
fire you want to use so it's really easy
don't machine-learning scare you off
because anybody can do it i'm not a
machine-learning guru and i was able to
figure out and use machine learning and
like a day so it's really simple
so anyways that's what you guys have to
look forward to questions comments
concerns whatever leaving below
otherwise as always thanks for watching
thanks for all the support subscriptions
till next time