what is going on dudes and dudettes welcome to part 15 of our data analysis
with python and painting tutorial series in this part we're going to be talking
about how we can start structuring your day set on so we can feed through a
psychic learn machine learning classifier first of all at the end of
the previous tutorial we basically took all these data and we stuffed it in his
HP I . pickle that mean we're all set here now if you want to i would
recommend saving the code from part 14 that way if you want to add more stuff
to your pickle you can do that but of course you can always load in the pickle
add some new functions that pickle and then make a new pickle
okay you do it if you want but for now homage delete basically all this code
all the way up to this point
ok so in order to feed through machine learning classifier we gotta do one
major thing and with machine learning generally with supervised machine
learning anyway
well we have to do is I mean obviously we could also try filtering through and
supervise but anyway supervise you have features and you got your labels
ok so you've got feature sets usually select features would be you know the
state hpi state unemployment usgp sp500 these are features at the moment and
then the label or the classification is you know that current hpi value and our
goal is to at least in this scenario we want to predict changes in the hpi so
what we end up doing is we're going to take current hpi values compare current
to next month's hpi value or maybe the next two months the hpi value and either
HP I went up for hpi wind down if HP I went up we call that a 1 hp I went down
call that zero and then so that's the classification and then the feature sets
that made up that classification is that months you know features
ok so we feed through all those features for machine learning classifiers to say
okay machine learning classifier today we have such and such gdp we've got the
current hpi is X we've got unemployment rate at you know why and so I'm gonna go
down the list what's gonna happen next month are we gonna go up again and the
goal is to have relatively high accuracy ideally something over seventy percent
but obviously we'd like a tease or more obviously like a hundred percent but
we're not going to get it
uh-huh we want something is relatively accurate at predicting the housing price
index so first thing we have to do is you have to generate that label right
because right now we have the hpi and we can find out what the next hpi value is
but we don't currently have those numbers so we need to need to generate
that data you're still kind of fuzzy on that don't worry we'll and I think it'll
clear up for you as we start writing and coding all this so in order to do that
we're going to cover mapping functions and then we since we're going to map
these custom functions i'm also going to talk about the rolling apply because
those both are very similar in my book it basically if pandas doesn't do for
you what you want pandas to do you can run a quick function and make that
happen or you can write a quick like apply and do the rolling apply and that
kind of stuff so this is pan his way of letting you know how such an awesome
module but if they don't have the things that you need specifically it's fully
customizable so first of all we're loading our data loading our day is
pretty simple it's going to be housing underscore data equals PDR read under
store people I do like that one line to read and do a pickle is nice
hpi that pickle boom we read the pickling goes to housing data good to go
ok next side note i wonder if pandas can read every pickle or if that pickle must
be a data frame because otherwise you would never even import pickle right we
can get rid of can't believe that still around
I don't know how that still in there we use that for a long time anyway I wonder
if you could always use painters for like all pic
or something because using only one line to open up a pickle and read it and
stuff that's kind of nice
anyway what we're going to do now we've read the pickle and we have the housing
data now if you recall before when we did our little percent change thing we
wanted the percent change over time but when we're investing we don't really
care about percent change from the first value we care about what's the value now
and what's the percent change to the next value so what we're going to go
ahead and do is actually apply a percent change the entire data frame so we do
that by going like this housing underscore data equals housing
underscore data . PCT honest or change done let's print out housing data . head
just so we can kind of see where we are
Oh is saying no liar you're lying to me i know i have it there
what is this sorcery let's see do we say that hpi maybe I didn't seem as HP out
pickle I'm pretty sure I did
let me pull it up here we go whoops with wrong-way pn does tutorial apparently i
don't have it this is the code that we wrote hpi pickle
oh I must got an error that's really strange
I don't recall getting an error in that last video but maybe I cut it off before
I was complete making the pickle or something I'd ok
or maybe the air maybe it was just like off the screen
I don't know anyway ok there i have the pickle now that was odd i'm so
surprising catch that last videos anyway carrying on loading the pickle
there we go so the first value is not a number because obviously we can't
calculate percent change of nothing so okay but otherwise we've got all the
state HP is
our housing price indexes or indices and then we have although it is in the
benchmark unless it was truncated I don't think it was
that's okay we'll leave that out for now anyway and yes so we just now at this
point we want to drop the the not not a numbers basically and so we can either
do actually go back and see what that was like sometimes I've gotten number
before being infinity but at least that's weird in the past I've seen when
you do percent change it'll be okay at least we have some infinity values so
we've got not a number and negative infinity so we gotta handle for both of
those so first of all to you
housing underscore data drop in a in place equals true second what we're
going to do is we don't have no PI importance is bringing numpy for numpy
as NP coming back down what we're going to go ahead and do here is we're going
to say housing under store data . replace and we're going to split
actually where where students above not a number surrounding places number
replace and we're going to replace numpy . infinity as well as negative numpy
infinity replace that with NP . nan and then in place equals true you could also
that's really only way we're going to get away with as far as I now
okay then come take this pace that down here let's make sure we spread our
Comment make sure that worked up till now here it is
looks like it did so that's good the next thing that we're going to do now is
we want to we want to take the CR do not have
at us hpi very angry now let's go back to 14 then let's bring that old code
backup and CH there's hpi benchmark here so i'm going to copy that and apparently
we never had that as a part of the CHP I data equals read pickle hpi been she was
hpi benchmark but we never added that let's go ahead and copy this and I'm
going to add the benchmark guess we just didn't have it there so go ahead and run
this one more time say that in pics we do want to compare to the benchmark
right i mean the benchmarks we want we could have done we could do a state
though right you could you can run this information on just the state of Texas
or any other state but anyway so let's close this out now whole fully let's run
this
we're okay so United States pressure call that United States hpi but we'll
leave that for now anyway that's called united states anyway so now what we're
going to say is so housing data . head but will do so now we have all the data
that we want to get rid of that now we're going to say housing data and then
we're going to say the calm here is going to be us on our school us
underscore hpi underscore future this is the future usap i equals housing
underscored data . we're actually we have to do this united states united
united states
housing data United States and then we do this we do . shift- one now print
housing data . head and what it should do is shift that calm down for us one .
so so it's the future value so yeah so here right the current values 4019 and
we're seeing in fact maybe what we should do is so it's so much more clear
probably would be this and then take us into the future this the current values
United States paste so the future will be on the left the current will be on
the right so here's the current value and it's saying futures 4957 well sure
enough we can look at the future to see that so what did is it copies that calm
but shifted it down minus one so it's our way of a you know accessing that the
future number of course every time we'll just drop in a this way because the
final value would be another number now what we want to do is we want to
calculate the difference when I see is that bigger or lesser number so the way
that we can do that is with function mapping so we'll come onto the very top
and we're going to define a new function we're going to call this because that
that will be our label basically so define create underscore labels and then
we got curd valve cute API something like that so actually will maybe make
this crazy I hpi and then the question is if the future if the future hpi is
greater than the current hpi well that means we want to invest that was a good
thing so we'll say that's a 1 so we return a one else return 0 so if the
future api's drop we're going to say that label is 0 so everytime hpi goes up
we're going to take the current features and the label for those that those
current features is a one if it falls we take the current features and we say the
label for those features is 0
and then we feed that into our machine learning classifier so we have that
information
one problem that kinda can arise with this is as you can see in the graphs
mostly hpi goes up over time so you might have a really strong bias to
predicting up every single time so you want to look at that and take that into
account but even if you have that strong bias if hpi you really is going up all
the time then you're strong bias should pay out right so anyway now let's create
labels and how do we map this kind of functionality to an entire column
well what you do is and in fact let's when you print this right now we'll come
back to print stuff will say here housing underscore data label housing
data label equals and then what we say is with the list map and then in here
you first you want to put the function that you want to run we want to run
create labels and then after this you put all the parameters we have two
parameters so we just put two parameters to get five parameters you put in five
parameters doesn't matter so create labels and we have two parameters with
the current hpi in the future hpi well the current hpi is right here so the
current hpi is housing underscore data United States wills and then the next is
future hpi well we have that too that is us future hpi for us
hpi future was this this value paste
ok so it's kind of a long thing but basically you just do list of the map of
the function and all of the parameters after that has been created a new
console now we can
free housing underscore j dot head
yeah that should be fine you dropped in not available then you have your labels
so here we've got label1 that means it went up
ok so we can say I don't even see it oh here it is
so from 40 to 49 right or 100 40 vs 0049 that's up good
label1 then obviously here we went from the current see this would be a bro 3 45
to 60 but then we went down to 5118 so that's labeled zero right you can see
that's true from here as well and then again we went from 51 18 to 35 again
that's a drop so 0 and so on
so anyway our label is working that's how you can apply an entire function to
a data frame and then since we're on the topic of these custom functions being
applied so that's a custom function to the entire data frame and then what
about that rolling apply we were talking about before we can do is pretty quickly
so let's just do it up at the top
go ahead and run statistics import mean you're in Python 3 and greater you
should be able to do that and Python to you can't so if you're following along
it to shame on you anyway I'm getting good at making up these rhymes
anyway now we're going to decline moving average now of course i know that we can
do a moving average in pandas already but this is just me kind of showing you
the rolling apply in action so moving average and then moving average takes
values
that's all we have to pass through and then basically we return the mean of
values simple as that and really you wouldn't even need a function at all for
this you could actually rolling apply mean values but we're trying to show
that you can make your own functions that's what we're doing so then what you
would do at the very end here is you could say something like this you could
say housing underscore data and then we'll just stay
na apply example gonna save this past this tutorial better shown it equals P .
rolling underscore apply and we want to have the ruling apply be applied to what
data
well we can do the things we have been 30 so that we can do housing data and
then capital in 30 like that so that's what we want to apply the data to what
is the window that we want you want to 10 window and since now normally that's
all you need right with like a rolling mean for example you pass the data that
you want it to be applied to and in the window but in this case is a rolling
apply we have to specify what function we want this rolling apply to be done
with so then after that you say moving average and that's that so now we can
take this tastes actually heads not gonna work perfect right because it's
moving so anyways detail and there you go you got your rolling apply worked
okay so anyways a long story short that's mapping function to the entire
data frame which is I find mapping function is I use that way more than I
don't think I've ever legitimately used a rolling apply other than to show it as
an example but i'm sure that's just me I just haven't come across the reason for
it but i'm sure people have otherwise it probably wouldn't exist in pandas but
yeah so so those two things mapping functions and wrong apply basically
opens the door for you to customize pretty much everything in pandas like to
get any sort of data manipulation or calculation to be run on your data frame
done and I think it's really cool that shows some serious foresight on you know
the side of pandas so anyway that's pretty not nifty and a lot of times you
know there's prior way to make this happen without mapping function
ok the prize away I just don't know it and there's gonna be a lot of like
little stuff like this one is probably pretty easy you might be able to do a
one liner and have that be generated but sometimes you might have liked pretty
complex logic that you want to have mapped in the it's just easy just make a
function it's just a lot easier on you
anyway that's it for this tutorial in the next tutorial now that we have
labels we can really quickly create features because we already know how to
reference a list of columns so from there we got our features we gotta label
we can feed it through machine learning classifier very fast in the next
tutorial so stay tuned for that questions comments suggestions whatever
leaving below otherwise as always thanks for watching thanks for all the sports
subscriptions till next time