what's going on everybody welcome to a
data science slash data analysis
tutorial series with the Python
programming language and the pandas
library if you don't know what pandas is
at the you know most basic level it's
just a library that helps you interact
with row and column type data I think
the best way to learn something like
this is just to jump in to it so I'm not
really gonna explain much much more than
that so let's go ahead and get into it
what we're gonna be doing in this series
is just playing with a variety of data
sets so I'm just gonna hopefully get you
guys comfortable and familiar with
interacting with the pandas library by
no means can I teach you everything
there so if you just go to google and
type like pandas Python Docs or
something like that in fact you'll
probably land maybe even at the home
page or something like that but then if
you go to documentation and then go to
the API reference here is where
basically all the things you can do in
pandas is listed out for you so
definitely go there and just kind of go
through everything so first of all talk
a little more on that but the first
thing we should probably do too is open
a command prompt and let's install all
the things that we're gonna be using so
go ahead and do a pip make sure pip is
corresponding to hopefully Python 3 I'll
probably use F strings here at some
point
so hopefully 3.6 or greater I'm gonna be
using Python 3 points 7 and then pandas
the version will be 0.2 4.1 unless
there's been an update in like the last
day but if you're having some sort of
deprecation or warning issues or
whatever
check it out in 0.2 4.1 if you need to
and if you don't know how to do that you
could always say pip pip install panda's
equals equals 0.2 4.1 or something like
that but don't do that unless you're
having a problem you want to see if it's
you making a mistake or it's like a new
version so anyways we're gonna do a pip
install and hopefully spell install
correctly hand us we're gonna grab
matplotlib so pandas for data analysis
stuff matplotlib for some basic
visualization we're not going to make
this a matplotlib tutorial series by any
means but there's some cool quick things
we can do with matplotlib
so we will and then I'm gonna be using
Jupiter lab I've already got Jupiter lab
installed so I'm not actually gonna
trying to install it it wouldn't matter
anyway but that's the editor I'm going
to use it's just an interactive Python
notebook type of editor if you want to
use PyCharm or sublime text or whatever
the heck ideally you want to use doesn't
matter to me
I do like interactive Python notebooks
for data analysis because it lets you
kind of poke around but you can get
yourself a little screwed up and like
modify variable later and then get
things out of order and get a little
confused so if you don't want to do that
that's totally fine but I kind of like
them for data analytics so anyways at
least these two things in jupiter lab if
you if you want that as well
so while that's installing we'll just
talk about a couple of things involved
here and also let me check the version
just to make sure
panda yeah there it is 0.2 4.1 okay so
while that's installing let's talk a
little bit about how pandas works so
basically you're gonna read in anything
anything that has a sort of anything
that has a rows and columns sort of
structure or could have a rows and
column structure things like JSON for
example isn't quite rows and columns if
you were to look at it but that's what
it is or at least it can be not all JSON
is because you also want to have
everything have the same number of
columns but you can have JSON do that
same thing with like a Python dictionary
for example that's not even listed here
but that it there's a two dict and I'm
pretty sure from dict and then all kinds
of things like excel documents CSV SQL
even bigquery they've got HTML hdf5
documents so you can read in and then
the other cool thing is you can push out
so sometimes I've used pandas purely as
a converter so so just take data from
one format to another so SQL to JSON for
example I've done that because it's just
really quickly quick to convert it right
same thing like or especially like HTML
to SQL because like with JSON we
actually can do that pretty quickly and
easily with like a dictionary and then
JSON or whatever
but something more like from a table to
SQL or something like that if I just
want to do something really quickly when
I'm working I'll end up using that
Google bigquery is also pretty cool
because that's new I haven't actually
seen that I've never actually used
bigquery here and I wonder how good it
is like if it all loaded in batches or
not I actually don't even know so that's
really challenging and I'm just finding
out for the first time that this
supports bigquery so cool alright looks
like we've got everything installed so I
can quick quit yappin we are going to
use data sets here from Kaggle it's not
sponsored and supported in any way by
Kaggle it's just a cool place to get
datasets they have like a bajillion yeah
it's a real number by the way if you
just go to kaggle com make an account
come to data sets and then you can just
kind of poke around if you want there's
some there's 14,000 590 data sets right
now but there's always new ones coming
in we're gonna be using the avocado data
set first
if you didn't know avocados is actually
a fruit fascinating also most closely a
berry would not expect that imagine I
get in like a fruit salad and it's like
avocado or fruit flavored something in
itself a comet
so anyway we're gonna be using the
avocado prices data sets go ahead and
grab that download you have to have an
account to download it so I make an
account and then a lot of times on these
data set it's kind of cool they have
like some explanation a lot of times
this one doesn't really quite have the
greatest explanation ever but most of
them do and then this is just kind of
like a quick visualization and this is a
lot like what pandas is gonna look like
- this kind of distribution stuff would
that's also pretty cool but pandas
doesn't do that for us but it gives us
like a little table like this so that's
kind of neat let me think here oh the
other thing I would want to show you
guys is the kernels a lot datasets
people can write kernels which is
basically a Python notebook so you can
see what other people are doing with
those data sets and learn even more
coasts of the time they're going to be
using pandas and then other things on
top of pandas maybe at scikit-learn
which is what we'll use scikit-learn
towards the end XG boost is really
popular or maybe tensorflow care us that
kind of stuff so once you feel
comfortable and you really want to see
like how good are you or what kind
things are you missing in terms of doing
data analytics and then maybe later
doing like predictions stuff like that
you can always go to the competitions on
kaggle I think this is the best online
like in terms of online coding
competitions being applicable to the
real world this is the best place
because these are actual things that
companies want solve and these are
things that companies will ask you to
solve if you go work for them either as
an employee or contractor consultant
whatever ok so download the Kaggle
avacado data set I'm gonna put it in a
directory literally called data sets so
data sets there's my der my avocado so
now what I'm going to do is go ahead and
run Jupiter lab to run Jupiter lab you
literally just open up a command prompt
to go to the directory that you're
working in then just type Jupiter lab
that'll open up in your browser pick I'm
gonna go with notebook again though if
you want to use your own editor whatever
go for it man I don't care or lady I'm
not judging I'm actually at 10%
gender split so that's pretty exciting
sex split anyways
watch out comment I'm not reading the
comment section anymore after my
statements anyway import pandas as PG
this is just got a convention that we
denote pandas as PD that's just a
standard that people do and you're going
to see this in a lot of people's code
the next standard is DF generally if
you're gonna define a data frame it's
going to be called DF now if you're
gonna have conflicting names or
something like that you might say
avocado underscore DF or something like
that but most the time you'll just see
DF short for data frame and data frame
is just the object that is your columns
and rows there's also a thing called
series which is just like a single
column and then there's a thing called
panel which I almost never see anybody
making use of except in really more
advanced use cases and we were probably
we're never gonna use a panel I can't
say we're never in this series if I like
add stuff to it but at least in what
I've received for the series we won't
even touch panel but panel is just like
multiple data frames so it's like a
three dimensional right so rather it's
like columns rows and then depth anyway
DF equals PD dot
read underscore CSV so this is a
function we can read CSV and then later
we could do something like DF to CSV and
save it out as a CSV and we'll do stuff
like that in a little bit but for now
we're just gonna read CSV and data sets
slash vocano dot CSV and we'll just read
that in first so that works and we could
just like print out a data frame but as
you can see it always prints out just
like gobs of information in like as we
edit and modify this data frame and work
with it we want to be able to debug
often like you might say like you buy
print something out in Python but this
is a lot so so what you can instead do
is use dot head you'll see people using
this a lot and all that is it's just for
just so we can kind of see a quick
snippet and this will be the first n
rows default is 5 but you can always
change that so we could say 3 and then
boom you get that alternatively
sometimes you'll do some calculations
especially on like chronological data
where you'll use like a moving window
and head won't be good enough you'll
actually want to see like maybe the end
so you like the foot maybe but instead
we call this D F dot tail 2 or something
you can put anything you want in there
same same sort of thing and you can go
bigger than 5 2 if you want so so that
is just our height a way for you to
visualize so as you can see we've got
quite a few columns here we've got date
average price total volume this is like
how many avocados were sold that day
these are specific PL use I want to say
and then hopefully that's the right term
anyway
total bags I'm not really like I guess
bags I don't know
not all of this data is something I'm
totally curious about and then there's
type which will bite us in a little bit
and then I guess the region that came
from anyway just a quick simple data set
for us to play with and I really get to
to advanced into but anyway what we're
gonna do is now like like if you wanted
to reference a specific column you could
say something like this DF and then use
square brackets and then in quotes so
kind of like you would
a dictionary DF will go average price so
as you can see this just kind of prints
out to us
the average price and again the same
thing we can do is dot head just to get
a few just get a snippet of what we're
working with so now the next things that
we can start to do get a little fancier
now and if also let me show you one more
thing before we go on
so aside from using the brackets in the
quotes the other thing we could use is
this dot notation so you could say
average price dot head as well but
nobody does this don't do this I mean I
might have seen this someone use this
syntax like one time just know it exists
and you so like sometimes a column name
might actually be something that you
think is a panda's method or
functionality and it's not it's actually
the person's column name and you just
were none the wiser and that's kind of
why I think it's better to use this
terminology instead and I'll leave that
there but why it's better to use this
because this is totally clear anyone
reading your code understands average
price is is a column name because I'm at
it you could have this called like you
can have a called mean or something like
that where people are like thinking oh
it's an average it would be very
confusing so anyway use that one but no
this one exists unfortunately but don't
use it so the next thing that we could
do is go buy like a specific a specific
region so like in this case we can see
there's many regions we can what we can
do is like we can separate out a very
specific thing in these columns so
there's probably a bunch of regions and
later we could we could get a list of
like all the regions and we'll talk
about that but let's say we just wanted
just the Albany region only how could we
do that well we could say instead and so
now we're gonna have like the main DF
and then we're gonna make a new data
frame so now we'd probably like to say
something a little more like Albany DF
okay so that'll be our new data frame
we're gonna say that's the data frame
where the data frame region
is equal to Albany Albany so this looks
a little confusing but it's really not
it's just the main data frame where in
these brackets the DF region equals
Albany and that's it so now Albany DF
dot head and now you can see okay so
this is only where the region is Albany
now let me just make some more space
here just so I don't have to keep typing
at the bottom and scrolling down okay
so now the next thing is okay first of
all we got a couple of issues we've got
like useless over here and we can see
here the this this bold here that's
actually our index so like if we said
Albany underscore DF index Oh bandy
Albany there we go it's a worthless
index what is the index well the index
is kind of it's exactly what it sounds
like
what is this data indexed by so how do
each of these columns relate to each
other and hopefully the index is a
unique identifier kind of like you would
have like a unique identifier in any
database or something like that it
doesn't necessarily have to be but it
should so most of the time so in this
case what is the one thing that all of
these are kind of related by right these
are all avvocato this is avocado prices
and volumes and stuff blah blah blah
blah blah over time so they're all
indexed by date or they should be
indexed by date and actually it's pretty
clear that was the original like left
most meaningful column so that's that's
what probably was the intended index in
the first place so the problem is a CSV
doesn't have any sort of meaning about
index so like for example if we just
opened up this actual CSV file let me
just open it with not notepad in WordPad
I'll tell you that in here so we can see
here this is a CSV file this is the raw
data that we're loading
now it does have apparently this like
meaningless column number which is
pointless so so and we'll talk later
about how we can handle for that but
mostly we just want the date to be the
index we're not really worried about
extra columns so there's a bunch of
different ways as we like load data in
and later we can drop columns blah blah
blah blah blah but for now we're not
worried about that we'll handle that
later I just kind of want to show you
guys a quick rundown of ways we can
manipulate data with pandas so anyway we
want a meaningful index so what are we
gonna say we're gonna say Albany
underscore DF dot set index as date did
I tie it all been do you get okay so now
it's indexed by date now unfortunately
that looks like our date is I guess yeah
it's in just a weird order unfortunately
but anyway so we've set index as date
now the next thing I want to say in show
is what if we said okay now all been me
oh my gosh I can't I can't type Albany I
was trying really hard anyway
would do the head again and what what
just happened we lost the date so so
this is like your first exposure to
pandas in it's nonsense sometimes it
makes total sense but it's confusing the
first few times you deal with it
so with pandas a lot of these operations
what they do is actually return a new
data frame so let's go back up here so
so you can even see it happened right
here
we just said dot set index date and that
we didn't ask it to display Albany data
frame it just just print it didn't print
it out but it just displayed it because
that's what this set index does it
returns a data frame so so we have two
options when we want to perform most
things I will say some of the commands
well just by default be what's called in
place and then others don't have an off
to be called in place you actually have
to redefine so there's two things that
you can do though is something like set
index one way one thing you could say is
Albany DF equals Albany bro this is it
this is the worst Albany this is okay
Albany DF dot set index date dude maybe
I just shouldn't be doing a tutorial
today I don't know so now you can see
nothing got printed out for us nothing
got displayed so could because instead
well we just reassigned basically so now
Albany D F dot head I did it now it is
actually done so the next thing that I
could show you as well is your other
option was to do something like this you
could say instead Albany actually Albany
do you have to hit I'll let that stay
there but then I just want to show down
here an example of the other way you
could have done it
Albany D F dot set underscore index and
then you would set the indexes date and
then you would say in place equals true
so this will actually modify the Albany
dataframe in place so you don't have to
do the reassignment
but we've already set the indexes date
so that's not gonna do anything for us
so so we're all done now the next thing
I want to do is show you graphing so one
thing we can do is just immediately
graph this data it's gonna look horrible
because it's clearly like not quite in
like decent order so anyway what we
could say is Albany underscore DF dot
plot and we can immediately plot things
and in fact this will be correct I'm
tempted to just do it I just kind of
want to see what it does shoot I forget
the
there's like a little short you can just
there's a shortcut that you can run in
it's like the percent thing I'll get it
right for the next tutorial
anyway there's a thing you can do in
Jupiter notebooks that will just display
the graph immediately in that case if
you just see something like this just
like run it again and it'll pop up
anyway so as you can see here we can
plot stuff like immediately or you can
plot really specific columns so you
could say hey I want to plot average
price and it will plot average price I
will just say right out of the gate this
is not a correct plot we've got other
issues going on in terms of the date and
then also some other things but we'll
get to that in a little bit so I think
for now that's everything if you've got
questions comments concerns whatever
feel free to leave them below otherwise
in the next tutorial we're going to talk
a little bit more about how we can
actually modify this data and work with
this data and if fix this graph and all
that but there's a whole lot of stuff
for us to cover and I just I don't want
to keep making part one super super long
so a quick shout out to the recent
channel members GRA Dinesh Kumar and the
Cassie wacky and Chris Olin thank you
guys very much for your support if you
don't know how you could become a
channel member you can click that
beautiful join button underneath the
video if you want otherwise questions
comments concerns whatever feel free to
below if you want you can come join us
at discord GG / centex
either come hang out or get help with
other stuff anyway
until the next video