what is going on everybody welcome to a data analysis with Python in the pandas
module tutorial series in this series we're going to be trying to cover pretty
much everything you might need to use with the pandas module along with Python
for all kinds of different types of data analysis
so first of all python is the programming language if you don't have
Python already
what you want to do is head to python.org and go to downloads and
download the latest version basically this is for windows because i'm on
windows and they know that and so just download the latest version don't go
with 27 or any version at two but if say there's three point six or whatever or
even Python for or something
you should be able to just download that now moving this aside once you have
Python the way that we are going to develop is I developing the basic idea
LOL II but you can develop in any ID that you want so if you're following
along and you use in a condo with the spider ide or maybe you're using ipython
notebooks or pycharm whatever it doesn't matter
follow along with whatever you want if you're in an interactive development
environment just you probably don't have to do print when we say you know the
data frame head or something like that
that doesn't mean much to you don't worry about it next we're going to need
pandas this is the pandas website you could download it from here and install
it from here but we're not going to do that we're going to use pit and then
finally this is my website python program that net all the tutorials
including this one are posted up here in text form sample code is there
everything you could need it you'll be able to find it here also we cover a
topic that you don't really know about say we cover Python dictionaries you can
always go to search and start typing in diction
in the Rio and there's some dictionaries for you so you could view the basics
tutorial on Python dictionaries
now that's that moving this aside what we're going to do once you have Python
installed the next thing you want to do is install pandas now pandas has a bunch
of requirements and stuff like this and dependencies
but we can get panda's really quickly by doing a simple pip install Candice now
if you run pip install pandas and that doesn't work for you it says many pip is
not a recognized command then what you want to do is give the full path to tip
the way that you would do that it's equal and / and then python your version
number on 3.4 so i would say 34 minutes and scripts and hit and then we would do
install windows
now that shouldn't stop and is along with numpy matplotlib if it doesn't or
at least to check just hit the up arrow or whatever
now instead of pandas Empire and then again I've known by this time
math lovely now as we go along and especially at the very end we'll talk
about maybe sk learn something like that and will need to install that as well
but nowadays you know there's a million ways you can install packages with
python nowadays pip is got to be the easiest way to do it
so that's what we're going to use if you don't if that doesn't work for you i'll
put a link in the description for my pip install tutorial you can always get
panda's another way but not going to waste time on here everybody most people
should be fine with pip install penis
now once you have that we go open up here and then you're ready to put in
your data now the question that you might have is what is pandas why would
we use this pan is module as opposed to just writing Python code because you
could take so like pandas it generally works with data frames you can work with
series to a series is like a list with a column header and an index but you could
take a look at data frame is generally it's a lot like a spreadsheet right if
you print out that data frame to the console you'll see that it looks
identical to a spreadsheet with column headers it's got maybe an index with
maybe you've got dates their line
over something like that and then in between you've got all your data
ok and so it's a lot like maybe an Excel spreadsheet if you're familiar
now why would we use python / excel
well the reason why would use python / excel is excel
despite the name is not very fast at all and it doesn't speed up for you in fact
as you get bigger it's going to slow down and not be the greatest now with
python things are going much faster so i see a lot of people with a Cell starting
to do machine learning with like maybe k-means or something like this or can
use neighbors or whatever and it's very slow
I mean the whole process even on a very relatively small dataset might take you
three minutes or something like that or even a little on a you know medium-sized
dataset most people would call it
large but on really a medium-sized let's say you're dealing with a couple
gigabytes of data
well first of all if you doing dealing with more than like two gigabytes excels
gonna probably stopped responding but also it's going to be really slow its
particular 30 minutes to run even even let's say you're not even doing machine
learning you're just trying to do some basic calculations you probably have a
problem
now you can tie in Python to excel stuff like that and there's other things other
languages that will tie in help you speed it up
but eventually running headless with something like Python or even see or
whatever is in your favor so then why use python / let's say C++ Python is
very friendly
it's a lot of people kind of nagging python saying it's slow especially slow
compared to say c++ that simply technically true but in practice
absolutely false ok python is almost as fast i mean we're talking ninety-nine
point nine percent as fast as see because python is written the sea
ok and a lot of these packages like pandas and panties utilizes numpy that's
a see library it's just a pet Python wrapper around see
ok so because of that it's going to calculate this fast to see because it is
see
okay so-so pandas with python python with numb pine all of these other
packages that people use for data
says it's just as fast and you get the loveliness that Python brings a table
which is you know really nice
it's obviously static types but so you've got you know the syntax is just
easy even for a newcomer to understand it usually you can read it and it makes
sense
so then finally why would we use pandas rather than Python plus a numpy well the
reason why you would use pandas pandas is extremely efficient it's basically
built to make your life easier
so let me go ahead and show you a quick example i'm just going to copy and paste
this code
everything that you're going to see here we'll cover in the series
I'm just going to copy paste the code if you want to copy and paste the code had
to python programming dinette the data analysis or dashboard then the data
analysis tab you can copy and paste the code as well if you want to but really
don't worry about it as long as you're thinking you're gonna go thanks videos
we are going to cover the code anyway i'm going to copy paste
here's some code we're importing paintings as pd here is just standard to
import it as PD is just a shorthand way
matplotlib for data visualization if you want to know more about matplotlib and
data visualization actually have a pretty large tutorial series format plot
lib found here i'll put a link to that in the description and it was as we
cover matplotlib will pry mention that again but we're not really going to
focus too heavily on data visualization this is data analysis and manipulation
now data visualization is a form of data analysis sure but already have a very
extensive series on doing that with matplotlib so we're not going to touch
on that much
so style is just with matplotlib now we gotta start and end a and we're using
the i/o of pandas is another reason why you would use pandas pandas is it works
with all kinds of data types and it can work it can interact between data types
too so you can import from csv output to SQL or import from SQL output to HTML
import from HTML output to csv and so on
you can kind of go between those different data types in the
the most beautiful thing is no matter what the data type you import from is
handling the data frame is identical matter what so it's our way of kind of
normalizing all datasets for data types rather anyway so start and then we use
this
io data is just a way of working with a Web API basically sort of referencing
yahoo finance API those of you who are familiar with my tutorials
we usually end up doing something with stock prices not gonna do stock prices
still doing prices though or at least markets we're going to do the housing
market with real estate will just do some data analysis on real estate
information
this is how we can visualize the data frame will talk more about that in the
next tutorial we can plot various columns I could just closed a column and
we can show it
so go ahead and save and run that it's f5 for anyone who is not familiar runs
that
here's the actual pricing data dates at the bottom and then here is just ahead
of the data this is the head is basically the top five rows so there's
obviously a lot more data points here is just these are the first five
so then you've got date open high low close date is lower than all these other
columns because day is being treated as an index which we'll talk about in the
next tutorial
that's basically it so as you can kind of see now especially visually pandas is
going to put things into what you would probably call spreadsheet like format
same thing with like maybe a table in the database same sort of thing I mean
this is kind of this is how people are accustomed to handling data so its it
works with you that way and you shouldn't be too uncomfortable working
with the data frame it should be relatively a normal feeling to you but
as we go on being able to apply various calculations analysis statistical kind
of stuff on even massive data sets is very fast
and very efficient so it's definitely very useful and there's a lot of
pre-built commands in there for you so you don't have to really worry about
writing them and pandas at the end of the day is highly customizable there's
ways you can map functions either to the entire data frame make a new column
based on a new function or even make rolling applies and stuff like that so
anyways
last up for us to cover if you have questions comments concerns whatever
feel free to leave them in the comments below otherwise as always thanks for
watching thanks for all the support of subscriptions and until next time