‫Alright.
‫So now we've got our notebook a little bit laid out and we know what steps we're taking, we know what
‫problem we're trying to solve, we know where our data is coming from.
‫We know our valuation metric and the features.
‫Let's import the data and get started.
‫So in my case, we've imported it to a folder called Data.
‫You might have created a folder the same.
‫So when I write the file path, your file path might be different to mine, but mine is going to be
‫called data.
‫So let's do that and we're going to import the training and validation sets to begin with if you're
‫wondering why we're importing them at same time, this will seem clear in an upcoming video, so you'll
‫just have to trust me for the time being that we're importing both in one set, and if we go Blue Book,
‫we can probably use tab autocomplete.
‫Surely.
‫There we go.
‫So we want to train and validate csv.
‫Wonderful.
‫We hit shift and enter.
‫Now, this might take a while because as we'll see in a second, this is a fairly large data set.
‫We're getting a warning.
‫What do we have here?
‫D type warning columns.
‫1339 4041 have mixed types.
‫Specified type option on import or set low memory equals false.
‫So if we go here, we can get rid of this warning by setting low memory equals false.
‫So what this is essentially saying is when pandas read CSV, when this function is instantiated, depending
‫on how large a dataset is because data frames live in RAM, Pandas tries to minimize the amount of space
‫they take up.
‫So what we're doing is we're just setting low memory to equal false to go, Hey, we know you're trying
‫to minimize space, but don't worry too much.
‫We've got plenty of ram space.
‫So this should work.
‫We should be able to get rid of that warning.
‫Wonderful.
‫So this is just saying.
‫Store it however you want to try and import it.
‫Don't try to minimize the space that you're using.
‫Wonderful.
‫And the first thing we'll probably do is go down info to get a little bit of information of what's happening.
‫So there we go.
‫We can have a look at that.
‫It is the biggest dataset we've worked with yet by a fairly long shot.
‫We have 412,698 rows.
‫That is a lot of rows and 53 columns.
‫Remember.
‫If we have a look at our columns, these are our variables and descriptions.
‫So we should have sales ID.
‫Yes, that's a non null integer machine ID, which is the identifier for a particular machine.
‫Machines may have multiple sales, so that means a machine could be sold multiple times over the course
‫of their time frame.
‫Then the one we want to be paid most attention to because it's a time series aside from the sale price.
‫That's how much our machines have been sold for is the sale date.
‫So this is when the machine has been sold.
‫So we have a look at this sale date.
‫Time of sale.
‫Beautiful.
‫Okay, so what can we do now?
‫Since the sale date is a date, what we might want to do is pass it as a date.
‫But we won't do that yet because I want to demonstrate something.
‫So we're going to set up a plot because this is what we usually do when we first begin.
‫Any kind of project is try to make some visualizations.
‫Looking at this, there's a lot going on here, right?
‫Like you could just look at this.
‫This is hard to infer if you're like me and you're just looking at lists of words and numbers and you
‫just like, Oh, I need to visualize things.
‫I need to look at things.
‫But we can see actually what can we see from this?
‫So we've got 412,698 entries.
‫These are non null.
‫We see we got some missing values, especially for usage band, whatever that is.
‫So if we come up to our data dictionary usage band, so value low, medium, high, calculated, comparing
‫this particular machine sale hours to average usage for the fee base model.
‫So the fee based one is really another feature.
‫There we go.
‫This aggregation of fee model description, far out.
‫We've got a fair few pieces of information here about our bulldozers.
‫Right.
‫So this is actually a pretty good data set.
‫But we can say that for usage band, there's a fair few missing values because there's only 73,670 non
‫null objects.
‫So that means there's 412,698 minus that missing values.
‫And we can figure that out actually before we plot, let's figure that out.
‫So we've got DF is an A, this is how we're going to check.
‫If there's no values.
‫There we go.
‫So Usage Band has 339,028 missing values.
‫Knowing what we know about machine learning models, that's something we're going to have to take care
‫of before we can build a model.
‫But let's push forward.
‫Let's plot because we're trying to predict sale price and we know that our data is a time series.
‫What we might want to look at is something like comparing sale dates with sale price.
‫And because there's so many rows here, if we were to do that, if we were to try a plot, all of these
‫data points in one plot, because you imagine trying to draw 412,000 points on a plot, your computer
‫is going to have a hard time.
‫So let's do about 1000 or so.
‫Why not?
‫Let's go.
‫Sale date.
‫We go here, we'll do the first thousand.
‫How about that?
‫That's a fairly large number.
‫We want to do sale price.
‫And if you want to find out the names of different columns and remember, you can always go for columns.
‫So what we're doing here, we're taking sale date, wherever that is, sale, date there, which is
‫if we come to our data dictionary, this is where we want to be referencing.
‫So the time of sale, so the date that it's been sold as well as sale price.
‫Now is that in our data dictionary?
‫I think that's pretty straightforward actually what sales price is.
‫Oh, there we go.
‫Sale, price.
‫Cost of sale in US dollars.
‫So let's check it out.
‫Let's check when whenever these being sold, we're going to put the sale date on the x axis because
‫remember, scatter is x, y and sale price on the Y axis.
‫So let's check it out.
‫What have we got here?
‫Oh, we've got our equal sign.
‫Well.
‫So you see here we tried down the bottom to put sale date the first thousand examples.
‫So this is why our plot is looking a bit funky and it's just trying to pack in if we have a look at
‫our DataFrame DF sale date.
‫That's what it's trying to plot on the x axis if we have a look at the first thousand.
‫It's trying to plot all of these values along here so you can see how it just looks like one big chunk
‫of a barcode and then just a big chunk of black text there.
‫So what we might do is let's just plot a histogram because we always like checking the distribution,
‫which is the spread of data of as many variables as possible.
‫But in this case, why not do sale price?
‫Because that's our target variable.
‫So sale price.
‫And now, again, if it seems kind of unstructured, what I'm doing here, it's because it really is
‫like there's so many different ways you can explore a data set.
‫One of the best I like is just DFM info check to see if there's any null values.
‫So we know NULL is in a as in missing values.
‫So we know how many missing values are.
‫Check the columns.
‫Plot one of the most important columns from now.
‫I've just chosen sale date because it's a time series problem plot one of the most important columns
‫versus the sale price or versus the target column and then do a distribution of a few columns of your
‫choice.
‫These are kind of the stereotypical things that I like to do when first exploring a data set.
‫But what we can see here, we can see that most of our sale prices are below 20,000 USD.
‫So the biggest column here, or not most, but a large majority.
‫A large portion.
‫And then we've got some below 30,000.
‫Let's just assume that 30,000, 40,000, and then it just keeps going off and there's less and less
‫and less and there's not very many that are costing upwards of 100,000 USD.
‫So that's what we're looking at there.
‫Okay, now.
‫What we might do is because these dates aren't really in a format that we want them in.
‫So see how they're in D type object.
‫When you're dealing with Time series, you want as much information to be encoded in the dates as possible.
‫So that's what we'll do.
‫A little way that we can do.
‫This is called passing dates.
‫So let's have a look at what this looks like passing dates.
‫So when we work with time series data, I want to enrich.
‫The time and date component as much as possible, you might be thinking, Well, how do we do that?
‫Well, let's put it here.
‫We can do that by telling pandas which of our columns.
‫Has dates in it.
‫Using.
‫The past dates parameter.
‫Wonderful.
‫And what this is going to do is turn whichever column we pass to pass dates into a date time object.
‫So let's first see what is.
‫A date time.
‫Object and pandas.
‫I got here Timestamp two date time so you can find out a whole bunch of different things.
‫What date?
‫Time here is date time.
‫Object Timestamp is a panda's equivalent of Python's Date Time and is into exchangeable with it in most
‫cases.
‫Okay.
‫Beautiful.
‫So basically when we run this, let's do it again, import data again, but this time pass dates.
‫Remember, if in doubt, run the code.
‫So PD read CSV and we're going to go data loop book for bulldozers tick and we're going to go train.
‫Try an invalid.
‫See us v wonderful.
‫But this time we're going to pass it.
‫Low memory equals false so we don't have that warning.
‫And then we're going to pass it, pass dates and we'll pass it the column name sale date.
‫Why?
‫Because that's where our dates are.
‫And so take note of what this is.
‫So if we go sail date, it's a D type object.
‫We might just put a little here D type D, type o.
‫So when we do this, let's see what the difference is.
‫Now this is going to override our original data frame so we can go.
‫DFCS Sale Date dot D type.
‫Ah, that's a bit funky.
‫What if we did df saudade?
‫What does it look like?
‫Do you have not sailed out first 2000?
‫Aha.
‫Okay.
‫So if we come up here.
‫We got saoudite.
‫See how they look like this.
‫What PANDAS has done is because we've passed it to pass dates, it's gone over all of the values in
‫the Saoudite column and converted them into this style.
‫So we've got year, month, day and we can see that the data type is date time.
‫64 and if you were to Google.
‫Compare these two because you might be wondering why does it have a different value here compared to
‫here?
‫You'll find that in NumPy these two are equivalent.
‫So you'll just have to take my word for it.
‫You could Google it.
‫We might actually do that just so I can prove to you.
‫So we go here.
‫Control C so what data type is?
‫There we go.
‫That time 64 is a general d-type while maintains is a specific time.
‫There we go.
‫This is Stack Overflow.
‫So this is just a workflow, right?
‫When you don't get something, there we go.
‫MP D-type date time 64 equals empty type mate.
‫That's what we're after.
‫But now that our dates have been passed, let's do the same plot that we did before.
‫So fig x equals plot subplots.
‫Wonderful.
‫And we're just going to do the exact same thing.
‫X don't scatter.
‫We'll finish up this video in a second, actually, because we are actually going on a rush.
‫Right.
‫We're learning.
‫We're exploring.
‫Sale price, and then we'll go here.
‫We'll do a thousand and see what looks different.
‫Wonderful.
‫So see here, I'll down the bottom now instead of when we come back up here, we've just got a whole
‫bunch of black because we're trying to put all these down there because our Saudade column is now in
‫the form of datetime object map plot.
‫Lib is able to intelligently look at the data type and then plot it accordingly.
‫So we got here prices so we can see there's a lot of sales, maybe 2005, something happened, so there
‫wasn't many sales in 2005, 2008, I believe that might have been the financial crisis or something
‫like that.
‫So there wasn't too many sales here.
‫And then again, that's between 2008, 2009.
‫We can see the highest sale was done just in 2007.
‫And then there's a lot of sales around the middle of between 2009 and ten.
‫But again, this doesn't really tell us too much, right?
‫We're just getting an idea of what's going on.
‫So let's have a look one more time.
‫Check it out.
‫Do you have not heard?
‫This is our data firm that we're working with.
‫We've got sales ID, sales price, a whole bunch of different columns here.
‫And we can see that there's a whole bunch of missing values.
‫And if we wanted to find out more about what undercarriage pad width means, remember, we've got our
‫data dictionary.
‫We could have a look here.
‫There we go.
‫Machine configuration, width of crawler trades.
‫Wonderful.
‫So now we've done a little bit of exploration.
‫What we might do is sort our data frame by sale date and then keep going from there, make some more
‫changes.
‫So we'll do that in the next video, but otherwise have a go at exploring the dataframe, importing
‫it and trying to pass the dates with sale date and don't pass the dates and check the difference in
‫data type when you do and don't pass dates.