hello my name is Tommy and welcome back to my second installment of this pandas
tutorial if you haven't watched the first video yet please look at the video
description and you'll find a link to it this is the table of content I'm going
to go through filtering and sorting equality and on equality logical
operators I'm going to talk about split up apply combine which is what's called
group I and circle as well as pivots going to talk about plotting time series
I'm going to show a little bit about modeling and machine learning just how
to use pandas as a tool to prepare data for these libraries such as scikit-learn
and finally I'm going to give you some some tips and tricks at the end this
tutorial is really just me going through this notebook there is no reason for you
not to do the same so please go to this link github Tomio d awesome pandas and
you can download this notebook and follow it as I talk okay first I'm going
to execute these cells from the previous tutorial just to get the packages and
import the same fake files as we're using filtering and sorting we've
already seen some basic filtering I'm going to show you a little bit more
first I'd like to talk about this syntax where you do the error frame and then
you give it a list of strings pandas is smart enough to realize that this means
that you want to select on columns so if I uncomment all of this quickly
you'll see that I do something like this it's going to filter on columns I can
create a mask like this one now I just say that if the index is divisible by 11
it returns true if not it returns false so the mask looks like this and by mask
I just mean a vector of boolean's and then I can take this data frame
I set the index to the movie title like so and then I can use this mask using
the same notation as above and now it will select on columns so this will give
me that sorry it will select on rows so now it's going to give me the rows which
are divisible by 11 and it's the row number not the index of the rows because
the indexes is the movie title now so that's why I set the index as the movie
title so essentially what it's going to do is give me every 11th row of this
original data frame so DF can select on columns or select on rows depending on
what the input is there is one further example if I create a mask like this
this is a series which consists of Williams again it's every 11th that's
that's true and there's an index as well now it will not select every 11th index
every 11th row it will actually select the indexes indices which are divisible
by 11 so there's a difference between using this mask which is just a vector
of boolean's and using this mask which is a series with an index as well as
values that are boolean and the difference is that this selects every
11th row and this selects the items in the index that are
and divisible by 11 so please be a bit cautious about what you do usually
pondless is really smart and will figure out what you're trying to do but if you
run into issues you should check what you're actually giving the data for ms
as an index so let's look at what this looks like in practice hair so for
instance you can find movies and TV shows from Norway the mask is this thing
I'm just gonna print it out you see that it just gives you the number the index
and true or false if there's a Norwegian movie it should be true and very few
movies are Norwegian of course so delete this and then I use this mask to select
a subset of rows here some more advanced example I can use the and operator I
want hi I am the Beast course I want a country that's not the US and I want
high grossing movies for instance and I want the top 3 so I create this mask and
I use the mask and then I selected the three largest you have to notice two
things I'm using the the end operator here I'm not using the word and this is
element wise and you have to do it like this the second thing that you have to
notice is that I explicitly use parentheses here because the evaluation
or growth of statements like this you have to be explicit about your
parentheses there are really just a few ways of doing logic like this there is
element wise and element wise or and you have the not operator and equality and
inequality okay let's look at group membership and
string filtering I can use this is in method to figure out if the country
column is in if each row is in this this list or not so that's going to give me
true or false and then I can use this so this these are the top five movies from
Japan or Hong Kong top five
you can negate this using the tilde so these are movies and TV shows they are
not from Scandinavia I can use the string accessor different students find
the word Lord the first tie I convert the movie title to lowercase and then I
search for the word Lord and then I can use this mask and in addition ask for
rows where the cross is higher than at the bottom 25% and that gives me this so
I get Lord of the Rings and Lord of war and stuff like that let's talk about
split up apply combine and pivots if you've ever used sequel the split apply
combine pattern is just like get the sequels in group by operator so this
statement to span the statement and the sequel statement is really the same
thing just in two different languages let's look at what it does
I select top three country and then I compute for each country the average
IMDB score and I rename it and then I select that from the data frame and I do
it in in groups by country and this is the equivalent syntax in pandas I want
to specify that I could have passed a string containing country air as well I
don't have to D F dot country I could have passed a string just
of the country and here's a nice visualization of what the split apply
combined pattern s you split your data into chunks depending on some some value
here it's depending on the airport so if the origin is airport one it goes into
this chunk airport two goes here and airport three goes here and then you
have this apply where you apply some function here it's the average value so
each of these chunks of data you apply the average value and then you combine
it into a data frame or into a table in the sequel case which is returned to the
user there's an example I see now that I have actually I ran this and I would
like to reset it sorry I'm just going to quickly restart and clear output and run
it again because I don't want to see the output before I run the cells so I
restart and then I'm just going to run the cells are you shift under to run the
cells going down to where we're talking about split apply combine them okay
sorry about that so let's look at the data we have countries and we have I am
the beasts course if we just filter out these two columns and we can run this to
compute the mean IMDB score per country and actually sorry it computes the mean
IMDB score and the gross and if I want to get IMDB score I can do something
like this
there we go I there's some alternative syntax that I
was talking about too which is this should return the exact same thing yeah
so if I just do mean it will just do mean value over every numerical column
then I can pick out whatever I want if I don't want to compute the mean over
every column and I know that I just want to IMDB score I can do this filtering
first I could do something like this and then now it will just compute the mean
over that one column which is one Americal column which is IMDB score okay
let's use this for something let's find the directors with the most movies so
I'll just run this first this is the output a nice way to look at what's
going on here is just to uncomment everything so I'll do this I take my
data frame and I groove by the director name and that just returns a group by
object and the computation is actually done when I call and unique which is the
number of unique so per director name I want the number of unique over every
other column like this so now it's that for for Adam McKay here it's the number
of unique director names gross values movie titles and so forth and you see
he's actually he only has two unique content ratings but it has six unique
movie titles then I retrieved the movie title column of this data frame then I
compute the top five largest and I convert it to a data frame that's just
so the printing will look a little bit prettier like so so Steven Spielberg it
comes out on top air with twenty five unique movie titles
these are the I'm not sure if this is everything but these are some of that
built-in aggregation functions you can call and groove by objects you can do
group I mean goodbye sound with by account and so forth let's make it a bit
more interesting by introducing several groups and aggregations we can groove by
several columns to do something like this so for every country and content
rating I'm computing the mean IMDB score then I'm just rounding the values and
converting to frame and getting the head that's not not very interesting but I
can do I can use several groups or several columns as my groups I can also
use several aggregation functions now I grouped by director name and I aggregate
using the mean the minimum and the maximum and then I retrieve that I own
DB score so these are perhaps not as interesting some of the other ones
everything looks pretty much the same Adam McKay he made several movies his
minimum score on IMDB is 6.3 is maximum at 7.8 and it's meanness approximately
6.9 you can also define your own function I can compute a spread as the
maximum minus the minimum I can do group I director name and compute the mean
spread a number of unique for instance
and here I just filter out the rows having number of unique greater than 1
this is something I didn't talk about earlier you can pass a function into
this location accessor so it's a function of the data frame and you can
apply logic there and that's really nice if you're you would like to do this
sometimes if you're doing method chaining because now that this data
frame is whatever this object is that's pretty useful
so you can define your own function and the mean number of unique pandas knows
about it doesn't know about the spread if I do this it won't know what to do
with it let's look at the error message series purify object has no attribute
spread so it doesn't know about the spread function I have to actually pass
a function but if it if it's something that's that's known to it you can just
pass the string I could also here pass like numpy mean and I would work I could
pass pandas series mean and I would also work so I think it just calls this Panda
series mean under the hood when you do this I can do several columns and
several aggregation functions so for instance here I group by country and
content rating and I a grenade using this logic so I am DPS chorus aggregated
using aggregated using the mean and the standard deviation
while the gross column is aggregated using mean and Max looks like this you
see some model numbers some missing values here probably because there's the
standard deviation was not defined in this case maybe there's just one value
or something so you can do combinations of columns
you can do several functions and you can do different functions on different
columns when you aggregate them so you have a lot of flexibility of course you
can do all of this in in sequel as well but I think it's nice to work with
pandas because the syntax is more concise and sequel yeah you sometimes
have to end up typing a bit more maybe that's just me maybe I'm a bit
inefficient in the way that I use sequel but I prefer using pandas stacking and
unstacking let's look at a new data set this is population
in the world so here for instance in Afghanistan we have the population a
number of people from the night from 1960 all the way to 2015 notice the
strange format of this data you have one country per row and one column per year
this is not good this is not the way that you want to store data you want to
store it so that each row is an observation and each column is an
attribute of that observation that's not the case here there's this nice paper by
week am called tidy data which talks about this it's not a new idea it's
something that people who work with databases know but he summarizes it
really nicely here and his his mantra is that each variable you observe should be
in one column and that's not true because the population is in many
columns each different observation of that variable should be in a different
row that's also not true because this is not one observation this is many
observations in this row there should be one table for each kind of variable
that's true here this is just one one type of variable in one actually it's
actually doesn't really comply with that either because if you add a new country
then maybe you'd want to have a country's table let's let's talk about
how how your data structure then we might be getting into too many details
anyway this is this is not good for for our setting this is not good for the
kind of analysis that we want to do we can put it in tidy format like this
this is much better now we have one observation as a row which corresponds
to a country a year and then the the actual observed variable which is the
population so this would be that you would think of country on the air as
being the index kind of and then population is your actual observation
here and when you add new information you just add more rows you don't have to
add any more columns like you would have to do it here so when you analyze data
you want it you want it to be tidy like this let's look at how we we used stack
to accomplish this I first here's the data I first set the index to data
source so the country is in the index then I use stack 0 and stack moves that
I think of stack is moving the columns down to the rows so here's what's
stacking the 0th column thus you can have multiple columns you can have not
hierarchical column structure so I stacked the zeroth one and it looks like
this then I rename it to population now it's called data source I didn't
actually change this it just gave it the name down here population then I convert
it to a frame looks like this now we're getting really close but I would like to
just take these out of the index just to show you what you're probably more used
to which is just having the data as in columns instead of having this multi
level index although that's really nice in some cases to have to so I reset the
index like so and I just rename some of the columns and I convert the year to a
day time format so it's not an integer
so stacked moves data from columns down to rows it's very useful in this case I
talked a little bit about this tidy format and how you would want to store
data in this way if you have a database and you have several several tables you
should read about the boyce-codd normal form which is about how to structure
data over several tables in a database and making sure that you have some
tightness in your data it's it's really useful to know about some cases we have
a problem with plotting libraries which might not want data a tidy format they
might want it like this if you want to track the population of Norway and
Sweden we we would pass it pass up something like this to applauding
library and to do that we use the inverse of stacking which is unstacking
so I set the index to this country in year and then I unstuck and unstacking
moves from it was the opposite of stacking I'm stacking moves up from rows
to columns while stacking moves from columns to rows so unstack the first
level that's this level if I unstack the second level the corresponding to the
first index I would get this which is not what I want I want to move the first
one up so I get this and then I filter this is a multi index column slice none
it says whatever is in the first one that's just population and then on the
second level which is this level I want these two countries Norway and Sweden
then I get this and that's exactly what you could pass to a lot of parting
functions
pivoting and melting these are more versatile versions of stack and unstack
you could say here's the data set we can pivot this like so so going from this to
this I tell the pivot method that I would
like the index to be the year I would like the columns to be the countries and
I would like like the values to be the population and it moves it up and it's
it's something that we could have accomplished using unstack as well but
if it is more powerful it can do more than just stacking and unstacking while
some things can be accomplished by both
okay let's drop countries where there are missing values and what I want to do
now is I want to compute a relative change since 1960 so I take this first
row and then I divide the entire data frame this thing divided by the first
row and then it's going to broadcast so it will divide this by the first row it
will just be one see it there and it will divide the second row by the first
row and the third row by the first row and so so forth this is called
broadcasting in numpy and this is the result so now I have a relative change
and if I want to go back I can use melt melt is the inverse of pivot and then I
have tidy data again and I have the population in relative growth since 1960
and I can use this to for instance make a plot like this
this shows the population growth of the five countries with the largest relative
change we see that the united arab arab emirates is on top of this this metric
so to summarize we have two powerful methods and their inverses for
converting data to tidy formats and to formats which are not tidy which which
is sometimes useful if you are passing to a parting function as we saw and if
you're computing things like this relative change it's sometimes easier to
do that when you have structured your data so that it's not tightly
there's unstack which moves information up from rows to columns and they're
stacked which is the inverse operation there's pivot which moves from a tidy
format and and generally kind of move things up as well there's melt which is
the inverse of pivot and you can play around with this it's not always easy to
figure out immediately how to use them especially pivot and melt but they're
very powerful and you can do a lot of stuff which is which is helpful in these
situations merging in pandas is what's called joining in sequel and it's the
exact I might the same idea you have data coming from different sources and
pandas that's different data frames in sequel that's different tables and you
would like to merge or join information together so here's an example what I
want to do now is I want to in my data set I want to make a new column showing
every directors number of movies no sorry not number of movies I want to
show the mean I am the be score for every
director so I compute the means of the directors by group I mean and then I
choose to IMDB code I'm DB score column like this and then I
can merge this data into I merge my director means data frame into my
original data frame like this and it's a left merge this means that in my left
data set I keep everything and if I have information corresponding to the right
data set then then that's that's great but I'm going to keep every index here
so if I have a hundred people here and I only find information in this table or
data set about twenty of them then there will be maybe 80 missing values
corresponding to that but what I still keep everything but if I didn't inner
join I would only get information about those 20 which are on both data sets so
that's the distinction I don't left join now because I want to
keep all the information in my original data frame and I want to merge in this
means if I find them and I need a key to join on to match on my left key
corresponding to data frame that's the director name column so this is director
name and on my right data set which is director means I use the index which is
this so merging these together you see that I have this new column called
director IMDB score and so for instance James Cameron his IMDB score average
IMDB score over all his movies seven point nine so maybe he just made that
one movie well some here have different different information for that single
movie and for the director here is another example I get
all the directors of movies made in the US for these I do a group by director
name and I compute the mean so that's directors directors movies
that are in the US and I compute the mean and I do the same thing for Canada
so now I have some directors who have made movies in the US some have made in
Canada and some are have made both types of movies and here I do another join or
merge as it's called and pandas this is an inner join so now I get information
about the directors that have made movies in both the US and Canada and I
compare the mean IMDB score for those movies if I change this to the left just
to show you I get a lot of missing values these are all directors who have
made movies in the US but not in Canada if I do a right join the opposite
happens here's the director that has made movies in both Canada and the US
and this director Andrew Currie is made only movies in Canada and none in the US
so I only get this piece of data and this one is missing but if I do if I do
after I get both and I will get every director and it's possible that they
have made movies that are either in the US or in Canada or both if I do in ER I
only get two directors that I've done both types of movies you can play around
with this there is a great thread on stackoverflow pandas merging 101 which
you can check out if you want to read more about emerging not merging
let's talk about plotting there are two ways to do plots and pandas who honest
use the built in plot methods there are about 10 slightly more than done that
are built in here's an example we take the country column and we do value
counts we get the top done and do some sorting convert to frame and here's the
crucial part we we call plot on this data frame let me show you the data
frame that's not printed like so if I if I do a semicolon in the strip or
notebooks the output or the statement is not printed but if I remove it it's
printed so I have this data frame which I want to visualize calling plot bar
edge creates this for me if I just call bar it will create this so that's quick
and easy and it produces a plot that's I mean it's reasonably good and it's
really nice if you just if you have a new data set and you'd like to inspect
it and just look at things quickly but if you want more control over what your
plot looks like there is the option of going through math plot lab which I'll
show you but first a little bit more about the plotting capabilities of
pandas there are inputs like these into the plot function which you can call I
can turn on a grid I can change the color change the font size give it a
title change the figure size if I want to and this helps somewhat I don't I
still don't think this bodice for it's not perfect it's not exactly how I would
want it to look in a publication say but it's it's fairly good
here's uh here's a final example doing a line plot if I just call plot and I
don't do plot dot bar H or a dot bar or anything like that it assumes that I
want the line plot so you can play around with this you can get help for
this plot by doing for instance help pandas dataframe plot yeah and you see
you have quite a few options the second option is going through a math plot lab
this gives you more control and you can really make some high quality plots if
you do this and what you do is you create a figure and an axis using
matplotlib I'm not going to talk a lot about math but lip since it's it's not
really the scope of this tutorial but I I create this new series here and then I
call the plot method and it has a keyword argument called axe for access
and I send this access in and it returns the access adding adding these lines so
then I take the returned axis object and then I tweak it by for instance setting
limits setting labels and so forth and then I get something that I think is
better compared to that defaults looks a bit better if I want to save this I can
just call this and then going to my tutorial folder I see this figure now
and it's a beautiful PDF and it's vectorize you can zoom all the way in
and it looks really good
f/11 and then I'll just uncomment this again here's another example of plotting
and I'm doing this kind of to show off my library KDE pi4 kernel density
estimators I want to use Delta values instead of dot numpy because I'm using
pandas 0 23 and to numpy is introduced in the next version which I'm not using
right now what you should probably use to numpy if you have that version of
pandas but I'm just going to change this quickly here's the graph showing I am
the bees course and I am the B scores way weighted by the growth of the movie
and the reason why I wanted to show this is because you can expect that people
who make libraries necessarily take pandas dataframes as inputs so then you
have to convert it to typically a non pra or maybe a Python object and the way
to do it is to use stuff values so here I take data
I am DB score and then I pass these values to me to my KD v method and the
same thing I do here and then the weights are the gross values so if you
are interfacing with other libraries you can't necessarily send in a series or
data frame will get an error must be of shape-up stems because if I call shape
then it gets confused because it expects the input to be an array so you should
be aware of that if you're using pandas to pipe data into other libraries
there is more to say about plotting there's this math plot Lib gallery
there's this other package called Seaborn which produces some beautiful
graphics and there is some nice YouTube video talking about the Python
visualization landscape here you can see format lot lib just the amount of
figures that are accessible and if you click on any of these you get the code
generating the figure so that's really useful or a seaborne it's beautiful you
can create all kinds of cool visualizations it's really high level so
you can look at that if you're interested in visualization time series
let's look at some data from Google Trends showing Python pandas SAS
Enterprise guide and Microsoft Excel there's a problem with this data set and
it's these values because instead of encoding them Asaro they're encoded as
less than 1 so pandas will believe that this column in particular is a string
because of this less than symbol so what I'm doing here and you can inspect this
at your own pace what I'm doing is I'm replacing this with zero and then I'm
doing some other transformations such as changing the name of the columns and
setting the index to the date and then we have this data and you can visualize
it using the plot function and will then draw three of these if I wanted to add a
grid for instance I could either the grid equals true sorry
or I could go axe grid true down here and remove this and it should accomplish
the same thing yeah so I can either work in in the plot method or I can change
the return access object afterwards
let's look at the seasonal component of this I can create a new column called
week of year like so I already talked about assign I create a new column
called week of year I take the data frame as as input to an anonymous
function and I from the index I convert it to a series and then I use this
date/time accessor which lets me do daytime lay like method calls on this
new object and it has one called week which returns the week of the year so
then I do a group I which you know all about I compute the mean I get something
like this so it's aggregated over all these years
and we should see these these dips around Christmas we should see those in
both Python and we should see them in Microsoft Excel you see them quite
clearly there is seasonal component so if I plot this you can see at the end of
the year there's a dip that's the average over all of these tips here can
do the exact same thing as above both grouped by month instead of weeks
let me just DT dot month and stud and you can google this DT the date/time
accessor and you can figure out what all of these are you can do things like day
of week you can do the day of the year the month you can do
I think you can do the quarter which quarter it is in the fiscal year and so
forth so quite a lot of functionality
rolling window functions pretty neat you can access them using this doctor
rolling and then you what's window here window is just the size of the rolling
window if I take it down to like four I just smooth it out a little bit if I
take it up to 28 smooth sitting even more and if I go to 50 it's very smooth
so this is a window of size 50 it's a triangular window and it's centered and
then I compute the mean so it's just this yeah rolling mean over over the
time series I could do something like rolling standard deviation no I can't I
said this okay I don't think that's possible
I'll show you how to do it later using rolling and then aggregate I think it's
like aggregate and P okay I'll show you later okay let's just take this down to
twelve or something yeah here's the standard deviation what did I do
I use aggregate so I do rolling and then aggregate and I do PD for panda series
standard deviation and then you get a rolling mean which is this solid line as
well as a rolling standard deviation which is this dotted line so you can see
there's more variation around Christmas it seems more local variation because
you have this dip every Christmas
you can read sample time series here I show you how to up sample to daily
frequency because this original data is of weekly frequency and have to down
sample to monthly frequency and when you sample when you do resampling you have
to specify some function so here it's it's padding the data just repeating the
same value and here it's taking the mean within each month so you can do these
kinds of computation some time series there's more to say about time series
but to be honest it's not something that I will use to pound us for a whole lot
but if you go to the official documentation you can find out what
other functionality exists now let's talk a little bit about machine learning
I'm going to show you how to create dummy variables or one of K encoding one
hot and coding which is needed when you feed data in to many machine learning
algorithms and also going to show you how to train a model train a really
simple model just using linear regression because it's not really the
purpose of this tutorial to show you machine learning just to show you how
how to use bound us we can use this set option to just display a few more rows
and columns we then reload the data just to make sure we have a fresh coffee drop
missing values drop duplicates and then the data looks like this so what I'm
what I want to do here is I want to dummy encode this content rating because
I can't send this into a machine learning algorithm then I'm going to
show you how to dummy encode genres here which is a bit more tricky because we're
gonna have to do some splitting on this and then we're gonna try to predict the
gross value based on all these other columns so based on everything else
we're going to see if we can predict value and getting a dummy variables is
really simple using this PD get dummies so if I if I give it this content rating
column it produces this you know the five here as well so we can compare this
so this first row has PG so you get a one for PG here and zero for everything
else the second one has R so you get a 1 for R and so forth when we give this to
a machine learning algorithm it's going to choose a coefficient for each of
these because we give it because we're going to feed it to a linear regression
model so it's just going to be a linear combination of columns and then if you
get a 1 here it's going to learn a coefficient for this column and then we
take these dummy variables and we use the merge which we talked about and we
left merge it into our original data set then we drop the original column so it
looks like this now we're getting close to something but before we can give this
to an algorithm we also have to dummy encode these values they're a bit more
tricky because the number of distinct values is a function of the row so here
we have two and here we have four and so forth
so first I am going to figure out how many genres there actually are and to do
that I will go to the genres column I will for the string I will loose the
string accessor and always we'll split it on this pipe or a vertical vertical
line and I will apply the set function and that's going to give me this it's
going to give me a set for every single row then I'm going to do a Union over
all of these sets and that's what I do here so these are the available genres
in this data set then I loop over every genre here I
create a new column with that with the name of that genre and I ask if the
string if the original column that string contains that specific genre then
this new column is set to one if not it's set to zero so then it looks like
this I I got rid of the original column and I added all of these columns instead
now you can have several like this first movie it's both a fantasy and an action
movie so you have interactions between these these columns cool now we're ready
to pass this to machine learning algorithm let's try to predict the
logarithm of the gross value I'm going to take logarithms of the gross and the
budget because they're so skewed there so there are some really high values and
it's very the distribution looks very log normal to me so I'm just going to
take the logarithm then I'm going to drop the original columns and we have
something that's a bit easier to work with
since the outliers will not affect it as much and it looks more normal to
distributed now here sorry can interface scikit-learn you can actually just send
in series and data frames into scikit-learn and it will figure it out
by itself so here I make this super simple model because my my X my inputs
it's just a vector of once so linear regression will then just learn the mean
value so this is just if you take the mean of every log gross
then and you just predict that mean then you would get a score like this so the X
that I feed into the machine learning algorithm
is a non PRA well the why is a pondless series so you can combine this and in
general cyclic learn is a library that interfaces pretty well with with pandas
if I wanted to convert this I will do dot values it shouldn't matter so as
long as it gets something that I can work with it doesn't really matter if
it's not B or if it's panda serious or pandas dataframe this first model got
root mean squared error of 0.92 let's keep that in mind let's try a slightly
more serious model now I dropped the columns which don't matter for this
algorithm the movie title and what I'm trying to predict the logarithm of the
gross eisah pronounced in gross or gross I'm not sure I just realized I'm not
sure how it's pronounced I'm sorry if I've been pronouncing it wrong okay I I
create this linear regression and then I do cross-validation ten times these are
the columns that I now use most of them are just the result of this dummy
encoding and this is my new model I just wrap these scores which is a numpy array
into a series and then I use the box plot function just to visualize it
quickly and now the mean score dropped so this model is better than this one by
quite a bit which is not surprising because this one doesn't really use much
information it just computes the mean well here we have some freedom to look
at the duration the budget and what type of movie it is to make a prediction
finally let's talk about a few miscellaneous tips and tricks I'm going
to talk about performance and then I'm going to talk about exporting
so you want to use the built-in methods instead of doing things your own way
if there is a built-in method for instance if you want to get the top 50
of something you want to use the built-in and largest two alternatives is
to do maximum to get the top value 50 times in a row and sorting and then
getting the top 50 and we'll see now when we created a data frame or serious
with a million entries that this one is going to be slow this one is going to be
slow and this one is going to be fast so
the first one computing the maximum 50 times the takes about a quarter of a
second of course that's so fast it doesn't really matter but we only have a
million data points this is a bit faster sorting and then computing top 50 but
it's still about a fifth of a second while this is very fast
it's 27 milliseconds which is almost exactly 10 times faster than this so the
reason for this is there's a smart algorithm for finding the and largest
which is it's kind of like a mix of these two actually which is optimal and
you should use the built-in methods if you can especially if you have a lot of
data here it doesn't really matter but if you have ten times four hundred times
more data it will matter more you want to do filtering before you do
computations if you can this applies if you're getting data from a database and
it applies if you're doing computations in boundless hair I compute the mean
over every column but I'm really just interested in the IMDB score so that's
an expensive operation compared to this it's like a thousand times faster hair I
I just I first get the column that I want and then I compute the mean here I
compute the mean over everything and then I have to call
so thousand times faster you can use a sorted index and this can speed up
filtering so here I I first sort by the gross value and then I set set the index
what am I doing here I'm setting the index to the director name and then I'm
sorting it and then I'm doing a query on the director name yeah that's what I'm
doing so here this is more slow than this
because here has to go through every director director name and check if it's
equal to oh and here it will occasion let's see what it can see what this
looks like there is your name is sorted in the explication sliceable and does
this look like so we're doing oh I don't remember precisely what I'm doing here
what's the slice
I don't I'm sorry I don't remember I thought I thought what I wanted to do
was to look at like juicing director starting with Oh or something sorry
I'll add a comment explaining this if I figure it out I'm not sure what I was
trying to do my point is that if you have a source that in likes some
filtering operations and stuff like that can be faster okay moving on sorry if
you can use a vectorized function that's better than using Python built-in
built-ins I already did this example with mat versus supply were Mathias is a
Python function on every entry while apply uses a vectorized numpy function
over the underlying vector here so that's faster and of course if you have
more data it's going to be even faster compared to this it's going to matter a
lot more this be a difference of approximately what is it seven eight
here's another one we want to compute mean of differences there is a fast
numpy solution I first take a vectorized difference and then I compute the mean
over that vector and that just takes two milliseconds this pure Python solution
is going to be a lot more slow so the speed difference here is about three
hundred times faster so that that matters pretty quickly actually so you
have to be aware of that there's this great online book called from Python to
numpy which talks about vectorization and you should really have a look at
that if you're interested in how this works and why it works and what kind of
problems you can efficiently vectorize equality checks is also an example of
something that you should vectorize and again it's really about using the
built-ins if I define this function s c-usa
that's going to be slower than using this this vectorized
check here as you can see it's three times faster because this has to do a
lot of function calls and stuff you can export data save it as a CSV file if
you're sending it to a colleague save it as an excel file if you're sending it to
a business person you can generate latex code which you can copy into your
researcher into your documents you can generate HTML this just looks like
printing it out usually thus but here you see the HTML you can copy this and
use it on a website you can also write directly to databases using data frame
to sequel there are some resources which you can look at there is this repository
of panelist exercises and that's really good if you want to do some exercises
now because I haven't really included any exercises in this because this
repository contains a lot of good exercises you can go here and you can
solve these exercises if you feel like you need practice it's probably a really
good idea if you didn't like this video or you want to watch more videos I've
collected a lot of videos in my repository here so you can click on any
of these and you can go on YouTube and look at videos there are a lot of good
videos out there and I'm sure this is not going to be the best of them but I
hope you enjoyed it anyway thank you for watching and I hope you learned
something