Hi there and welcome to Data Analysis with Pandas. 1
So what is pandas? Pandas is a library built on top of the Python programming language. 2
Now if you're unfamiliar with Python it's a language developed by Guido van Rossum in 1991. 3
So it's a little over a quarter century old. 4
And it's really taken off in popularity among the data science community in the last five or so years. 5
And that's due in large part to the growth of pandas. 6
Pandas is just one of tens of thousands of available libraries for Python and all that a library is 7
is a bundle of features and functionalities. 8
They add on new things that you can do with the existing programming language and each library kind 9
of has its own focus or specialty. Pandas focus is data analysis. 10
So really it's a tool that you can use for analyzing filtering manipulating aggregating merging pivoting 11
cleaning data. 12
These verbs don't even begin to capture the full scope of everything that you can do. 13
But any time that you're working with a data set if you know you want to tweak it to get it into the 14
shape that you want or you want to analyze it to get some kind of computations out of it. 15
Panas is the tool for the job. 16
So if you're familiar with spreadsheets software like Excel or numbers on a Mac or a Google sheets Panas 17
is basically the same thing. 18
It's just built on top of Python. 19
So I like to call it Excel for Python. 20
However the advantage of PANDAS is that it's a lot more powerful. 21
It can work on data sets that have tens of millions of rows and it can perform calculations on it within 22
milliseconds which is why I also like to call it excel on steroids. 23
So there are two sets of prerequisites for the scores required and recommended. 24
As far as the required set on the left a basic knowledge of spreadsheet software is required. 25
What that means is that you should have played around with excel at some point before. 26
If you have never played around with a spreadsheet software if you don't know how rows and columns operate 27
or basic things like Salm or average functions this course will be quite challenging for you to take 28
on. 29
And in addition strong basic understanding of data types. 30
By that I mean knowing what a string is knowing what the difference between an integer number versus 31
a floating point number is knowing what a boolean value is. 32
Things like that. 33
And if you don't have any experience with these kinds of things I do recommend starting with Excel. 34
And the reason is because it's going to give you a more visual interface to data sets. 35
Pandas is still visual but it focuses more on the code. 36
So having that visual interface for you to play around with in Excel is greatly beneficial when you're 37
just starting out and you need to see everything as you make those changes. 38
As far as the recommended set of prerequisites. 39
A strong knowledge of spreadsheet software will go a long way. 40
And by that I don't mean that you need to know how to code in Excel or know any complex statistical 41
or mathematical functions. 42
All that means is really things like the lookup functions or counting functions some functions. 43
Pivot Tables are great any time. 44
If you've ever worked with a data set and you've had to mold that or look at it from a different perspective 45
or aggregated and kind of perform some kind of computations on it that's really all that's needed this 46
basic understanding of how you can take a data set and turn it into something a little bit different. 47
Look at it from another angle. 48
And in addition it is recommended to have a basic understanding of Python similar to the Excel requirement 49
or the spreadsheet software requirement. 50
This doesn't mean that you have to be a software engineer or that you even have to have built anything 51
in Python. 52
I just mean the rudimentary programming basics like assigning values to variables defining functions 53
and understanding that they take input and return output. 54
Understanding what a list or an array is. 55
Things that are kind of the basics of any programming language. 56
In fact if you don't know Python but you're familiar with any other programming language you should 57
be fine as well. 58
Just kind of those simple high level you know universal concepts that exist in programming languages 59
and if you have studied python in the past in the past but you need a little bit of a refresher. 60
This module concludes with a python crash course where we'll dive into just the quick review of all 61
of these things we need to know in Python. 62
So if you've covered this before and you just need a quick refresher. 63
We've got that covered in this course but if you're coming into it with absolutely no knowledge of Python 64
at all and you've never programmed I do recommend spending an hour or two with that before beginning 65
this course. 66
So about this course it's split up into several modules and each focuses on a specific feature in Pandurs. 67
Now Panas is a colossal library so there's no way that we can cover every available function and every 68
available tool. 69
So each minute focuses on really the most critical components and they're designed to just kind of get 70
your feet wet and give you a sense of everything that the library can do. 71
There's probably millions of different permutations that you can do to get data into the way that you 72
want it for your project. 73
So my goal is more to give you enough of the tools that you need to get the basics and have the knowledge 74
to pursue the advanced stuff yourself. 75
And in addition this course is bundled with CXXVI data sets for us to use. 76
So in order for you to be able to follow along and see the same thing on your screen as I'm seeing on 77
my screen I would like to use these consistent data sets. 78
I know in other tutorials I watched on panels the instructors do things like generate random data sets 79
and I don't like that approach because what the students sees on their screen is completely different 80
from what the instructor sees. 81
So in this course we'll be using these static CSFB data sets and I'll be telling you which ones to import 82
as we proceed through each lesson. 83
So let's get started. 84
In order to begin working with pandas we need to install Python and pandas Python as the foundation. 85
So you can't run pandas without it. 86
And in addition pandas exist in an ecosystem of libraries. 87
What I mean by that is that if a data analyst is working with pandas it's highly likely that they're 88
working with other Python libraries as well. 89
In fact it's necessary because pandas itself is dependent on other Python libraries to run kind of it 90
has its own foundation of libraries that it uses to build its panels functionalities so we can go and 91
try the brute force method of installing Python and then Pandurs and then maybe the 5 or so libraries 92
that we need in addition but a much easier path for us is installing what's called a distribution. 93
A distribution is just a big bundle of Python pandas and a bunch of different libraries. 94
And the one we're going to install is called the Anaconda distribution by continuum analytics. 95
It's entirely free and it's basically one big installer package installer engage that takes together 96
Python pandas and about 100 libraries for data analysis and statistics and mathematics and scientific 97
learning and that kind of stuff. 98
Now we certainly are not going to need all of those 100 libraries but it's much easier to just install 99
this one thing than it is to install everything we need one by one separately. 100
So we'll be using the Anaconda distribution after we install Anaconda. 101
We'll use the computers terminal or command line to update the libraries that are installed and also 102
install some additional ones. 103
And if you're unfamiliar with terminal or command line or anaconda or Python or Panas don't worry I'm 104
going to take you through the entire installation process step by step. 105
You're going to see everything along with me on my screen. 106
I'm going to start completely from scratch. 107
So don't worry about that. 108
The set up is covered in this course a little bit about our development environment which is called 109
Jupiter notebook. 110
Now if you come from a programming background you may be familiar with working with things like an integrated 111
development environment called an idea or a text editor like sublime or atom. 112
If you are more familiar with general word processing you may be coming from you know notepad or textedit 113
or Microsoft Word. 114
So there are many different tools that we can use to write code throughout these lessons we'll be using 115
Jupiter notebook and it's a little bit different from the typical coding editor Jupiter notebook consists 116
of things called cells. 117
You can actually see one on the screen right here in the picture. 118
It's that little box that's in gray that has all the code. 119
And the great thing about the Jubran notebook is that we can write our code in these cells and then 120
we execute them. 121
And right below the cell the result of our code is output. 122
You can see that little table that appears below that's the output of the code that's in the gray box 123
above. 124
And this approach really encourages a trial and error approach to solving a problem. 125
You can write code and see how it'll make the results look. 126
If you're dissatisfied you can keep altering the code until it looks exactly the way that you want. 127
So it's really popular among the data science community for that reason. 128
It doesn't punish the user. 129
It allows the user to recover from mistakes recover from typos. 130
You know even have this playful playground mentality of you know try one thing even if you don't know 131
it's going to work. 132
And very quickly see the results. 133
And there's also other big advantages that will dive into in a few moments but this is just a quick 134
overview of the development environment we'll be using what is called Jupiter notebook. 135
All right a big heads up. 136
We all love pandas. 137
Then by that I mean the furry animal. 138
But the library actually has nothing to do with the furry animal I think it's a bear. 139
Panas is actually short for panel data structures which is some kind of complex acronym that's used 140
by economists or statisticians or econometricians I think I'm pronouncing that correctly. 141
It has nothing to do with the bear just as a heads up. 142
I don't know why they come up with these funky names but it's kind of funny so pandas. 143
It is. 144
So I'm going to conclude this lesson by actually doing a quick Google search for a few sample Juber 145
notebooks and these first two results that pop up the links to the best Python and Juber notebook's 146
gallery as well as a gallery of interesting Python notebooks I'm get hub are two really great links 147
if you just want to take a look at how a notebook functions or how a notebook might look in real life. 148
When we're talking about real data scientists or you know statisticians that work on these kinds of 149
things. 150
So I'm just going to click on one of these links and show you a sample notebook 151
I'm going to just pick this one that's called the important features and pandas at this point you don't 152
have to understand any of the code. 153
What's really more important here is understanding the structure and design of a typical Jupiter notebook 154
and how it operates by the way the old deprecated name for a Jubran notebook is called and I Python 155
notebooks so in case you see those two just remember that they're interchangeable they're both the exact 156
same thing but here you can see these gray cells that we've talked about we can see here we are using 157
pandas and the great thing about a Juber notebook is that it combines a lot of different things together 158
so we have code but we can also add that things like images or videos or sound effects. 159
We have these headers which allow us to kind of mark our code and talk the person through the process 160
and you know convert it into sections and give them a better sense of how we're proceeding through the 161
code. 162
So we know that as we move through it we can see how the person approached the problem we can see their 163
operations we can see oh at this point they're worried about the time series operations all the code 164
below is going to focus on that in addition to the actual code and we can see all these outputs we have 165
regular Python functions and here we also have things like bar charts and line graphs. 166
Let me scroll down and see if I can find one here. 167
There it is. 168
And these can be directly embedded into the notebook. 169
So when you read it top to bottom it sort of reads more like a story or more like a book than it does 170
a scientific document because everything from the code to the commentary to the visuals is all bundled 171
within the single file. 172
So you can see how the person approached the problem what their methodology was. 173
Each step that they took what they were seeing as they went through it one step at a time. 174
So it really is a very effective environment for teaching and learning. 175
So it's going to be a pretty effective tool for us to use to try out pandas one step at a time. 176
And that concludes this lesson. 177
So that's just a quick overview of everything that we're going to be doing in this course in the next 178
lesson we'll get started with installing the Anaconda distribution. 179
So I look forward to seeing you there.