a few weeks ago I made a video about
data science book's recommendation on
the channel although many of you loved
it I was still unsure if I was already
aware of all the best data science books
out there and this uneasy feeling
motivated me to take a more data-driven
approach to explore a little bit what
kind of data science books are available
out there so I found a data set on cargo
that contains data on nearly 1 000 data
science books on Amazon including the
ratings the price the number of pages
and other characteristics we'll be doing
three levels of analysis what I'd call
Basic intermediate and advanced level so
on level 1 we'll do some simple
exploratory data analysis to answer
questions like do more expensive books
have better reviews is it always true
that longer books are more expensive or
what are the best python related books
and what are the best machine learning
books based on review Stars moving on to
level 2 we start getting a bit more
fancy we want to find out what are the
main types or main categories of data
science books out there based on the
book titles we'll be using k-means for
clustering the book titles and in order
to do that for Text data we use an NLP
technique called tfidf or term frequency
inverse document frequency to convert
the text into numeric features and don't
worry we discussed those Concepts along
the way and finally on to level 3 which
is not for the faint-hearted we script
the Amazon reviews for these data
science books to further enrich the data
and we'll be using an advanced text
summarization model to get a text
summary for each book the main
motivation for this is that if you're
like me you might often spend hours
reading book reviews to see what people
are talking about the books so why not
have a smart machine learning model do
this fast it's actually very simple to
implement this video is also to
demonstrate a powerful collaborative
Notebook 2 deep note who has kindly
sponsored this video without further Ado
let's get started alright let's first go
to Cargo and download this data science
books data set it's rather a small data
set and we'll be exploring it in a bit
so to start off if you're not yet
familiar with deep note is a free online
data science notebook mainly focused on
real-time collaboration it's a new kind
of notebook that looks better and
integrates better with other tools like
git Google Drive and so on you can say
overall deep note is quite similar to
Google collab but deep note is geared
more towards real-time collaboration and
its main aim is to eliminate all the
friction when it comes to sharing
notebooks sharing data and keeping a
version history while working together
in a team I also really appreciate that
it allows you to create beautiful Rich
Text in your notebook adding comments
call outs and to-do lists much like what
you can do in a note-taking app like
notion after you finish your projects
you can also publish a notebook and
customize the final look of your project
in the form of Articles or apps you can
choose to show or hide any sales which
is great because sometimes you might
want to hide the gnarly code cells from
technical audience like your mom and
then you just copy the link and share
around like how you'd share a website I
think it's a great option for passion
projects for quick prototypes and team
projects where you want to collaborate
with others and show your output in the
most accessible and presentable way to
get started with deep note let's go
ahead and click on the get start button
and here you can choose to connect using
your Gmail or GitHub account in my case
I'll sign in using my GitHub I'm using a
free tier which allows you to have up to
five projects at a time and up to three
team members I think the only small
inconvenience with the free tier is that
your Cloud host Machining will go
offline after I think 15 minutes of
inactivity and you have to spin it up
again but I believe it's not a major
thing alright here's how the interface
looks like on the left sidebar you can
find a few sample projects and I'll go
ahead and start a new project you can
choose to import an existing notebook
from your git repo or Google drive there
are also other Integrations that you can
add over here in this tab the interface
looks very similar to a normal jupyter
notebook but everything is run on the
cloud hosted machine so you don't need
to install anything locally and if you
want to run the notebook on your iPad on
your phone it doesn't really matter so
let's make a small to-do list here to
keep track of the analysis that we want
to do now let's import a few necessary
libraries here to start with so numpy
pandas mud plot lip and blood lay
Express and to upload our data set to
deep notes we just need to add a data
set over here
then we can either query this data set
with SQL and save it in a data frame or
you can also copy the path to the file
here and read it as usual with pandas
let me go ahead and do that and if we
print out the DF here we can see a nice
overview of the data including the
distribution of the data in each column
and also the percentages of missing
values in each column which is quite
handy we have in total 946 book titles
and I believe that it contains many cool
data science books here and here we can
see we have the book title the author
the price number of pages average
reviews number of reviews the percentage
of 5 Star 4 star 3 star and so on we
also have information about the
dimensions and the weights and the
language the publisher and also the
complete link to the Amazon page so
quite handy if you want to check it out
now let's do some exploratory data
analysis which is level one to answer a
question like do more expensive books
tend to have better reviews what we can
do is to create a scatter plot that
plots the book price against the average
reviews in this case I'm using plotly
Express to quickly create the scatter
plot with the x-axis being the price the
y-axis being the average reviews and
maybe you can also add the size as the
number of reviews just to see how
popular a book is however on deep note
there's an easier way to do this if we
click on the plus button here to add a
new block if you go down we can see that
we have an option called chart
visualizing data frames if we select
that we have a new option that basically
gives us the option to create different
types of charts and we can also specify
the dimensions here for example I choose
x-axis as the price and the y-axis being
the average reviews then we can
basically have the same scatter plot as
we had before you can see that there's
actually no clear relationship between
the price and average reviews so good
books can also be very cheap or it can
be very expensive similarly if we do the
same exercise but plotting the price
against the number of pages we can see
that there's some positive correlation
here so longer books tend to be a bit
more expensive which makes total sense
because it costs more time and effort to
write those books now let's try to
answer the question of what are the best
python books and what are the best
machine learning books we can do this by
simple rule based method so we select
all the books with the titles containing
Python and now let's select the python
books with most reviews and have highest
average rating we can do this by using
the N largest function for example if we
want to only select the top seven books
and here you can select the columns that
we want to use to sort the data the
reason why we want to sort the data on
both number of reviews and average
reviews is because some books might be
not very popular and they have very few
review but they are very good reviews so
these cases might be more into bias and
in this 7 best python books we have here
python crash course second edition I'm
not familiar with this book we have
automated the boring stuff with python
which is very cool it's a very good book
I'm also very happy here that we have in
the list python for data analysis which
is a book that I recommended so it's a
relief that I actually recommended a
good book and similarly how about the
best machine learning books and we can
do the basically the same thing right
and here we can print out the top seven
machine learning books we have some deep
learning books we have the 100 page
machine learning book which I didn't
really enjoy so much we have pattern
recognition and machine learning which
is a very classical book we have a book
about reinforcement learning so overall
it's quite interesting and informative
but I think it really depends on your
needs what would be the best books
alright moving on to second level we are
doing text clustering with TF IDF and
k-means clustering to answer the
question what are the main main
categories of data science books in this
data set and with this approach we can
automatically classify the books into
different categories without having to
go through the 1000 book titles by
yourself and come up with the
classification rows as you may already
know clustering is an unsupervised
machine learning technique it is a task
of dividing the population or data
points into a number of groups such that
the data points are more similar to
other data points in the same group and
this similar to the data points in other
groups this method is very useful for
understanding the structure of the data
and there are many many clustering
algorithms out there one might work
better than others for particular types
of data but for this project we're going
to use one of the most simple and
popular ones which is k-means k-means is
an iterative algorithm you just randomly
initialize the centroids or the centers
for your clusters and then we simply
assign each data points to the nearest
centroids we calculate the mean of the
location of all data points that belong
to each cluster and then move the
location of the centroids to that
average location this is the first
iteration on this example data sets and
you can see that the Clusters don't make
much sense because it's totally random
and after several iterations the
centroids will naturally be shifted
towards the true clusters in the data
and at some point the position of the
centroids will stop changing in rare
cases this process may not converge but
generally it's a very smart algorithm
and I think it works reasonably well in
most cases now we got this but we have a
small problem here in our case we are
clustering Text data which is a bit more
tricky than the usual numeric data so we
first need to somehow convert the text
into numerical representation that can
be used and understood by machine in a
fancy term this is called text
vectorization people have come up with
multiple methods to do this from basic
ways such as counting the word frequency
in the text and use each unique words as
a feature to more advanced methods like
word embedding that use shallow neural
network on large Text corpus to learn
the underlying context of the words
however features in those methods and
what they mean are harder to interprets
and in recent years we also have large
Transformer models that encodes text
into huge embedding vectors we look at
one of those models in level 3 in this
video for now we use TF IDF which is a
popular frequency based vectorization
method is simple but for sure better
than the simple counts vectorizer let me
take a second to try to explain this
technique so our aim is to convert the
text and in this case the book title
into an array of numbers and the array
would contain the value for each word
that ever appears in one of the book
titles zero means the word doesn't occur
at all in this current title low value
means the word exists in a title but is
not very important for the overall
meaning of the title and the higher
value means the word is more important
for the overall meaning so how does TF
IDF determines those values it does this
by multiplying two factors together term
frequency and inverse document frequency
the idea is quite simple the first
parameter is a local parameter we
calculate the frequency of the word
within the document or in this case
within the book title in the book title
the best python book for python lovers
the word python occurs twice while other
words occur only once so python has term
frequency of 2 and other words have term
frequency of one overall we are assuming
that python is a more important feature
because it occurs more often in this
title and the second parameter in this
formula looks more globally looking at
how rare the word is across all the
documents we measure this by taking the
inverse frequency of the words in all
the documents for example if we have
another book title named the best r book
we can say that the word best is quite
popular it appears in both book titles
so you can assume that words like this
are probably less important so
multiplying these two terms would give a
quite good measure for the importance of
the words and it assigns higher value to
the more descriptive and informative
words and lower value to the unimponent
but maybe more commonly occurring words
in the English language so from Psychic
learn let's import the TF IDF vectorizer
if you haven't installed circuit loan
you may have to do so and then we can
initiate the vectorizer object here
using the stop words being the English
language stop words on Words Like You Me
At The with etc those are the words that
are generally not interesting for Nob
tasks and let's also specify the engram
range here from one to two because in
our book titles we usually see words
like date analysis or machine learning
or data science etc those are the words
that have two words so the n-gram here
basically says how many words we want to
put together at time and then we can
just use the vectorizer to fit and
transform the title into a vector X here
is just basically a huge array that
represents each book title now to
implement K means we also import k-means
from Psychic learn and a small challenge
with k-means is that we have to specify
the number of clusters that we want to
create and we don't know in advance what
is the optimal number of clusters so we
have to find it out ourselves so the
strategy is that we'll just assume a
minimum number of clusters let's say two
clusters and the maximum number of
clusters let's say 10 and for each value
in those potential number of clusters
will perform k-means cross string and
then we'll calculate the sum of squared
distance or inertial as how it's called
in the cycle learn Library it is the sum
of squared distance of the data points
to their closest cluster Center and I'll
just append those sum of squared
distance into an array oops I forgot to
import K means and then after we've
performed all the clusterings for each
possible number of clusters we plot out
the sum of squared distance you can see
that the more clusters we have the lower
the sum of squared distance is it makes
sense right because in extreme case if
we make one cluster for each individual
data point then the sum squared distance
would be technically zero with the elbow
method we want to find the optimal
number of clusters that have the lowest
sum of squared distance we can see that
around cluster 6 the sum Square distance
starts declining more slowly so we can
expect that 6 is the optimal number of
clusters however k-means is not
deterministic so if you run the code
again you might see a slightly different
graph for now we just choose 6 as the
optimal number of clusters and then I'll
pass that into the k-mins model to get
the prediction labels it's quite simple
we just get the labels out of the model
and then zip the labels with the title
in the data and here you can see that
which cluster the books belong to now
you might be thinking okay what are
those clusters actually mean that's a
good question one way to do this is just
to print out the top terms per cluster
however it's a little bit less
interesting I prefer to use a more
visual way to do this we can create a
word cloud for each book title cluster
it's very simple for each cluster we
obtain the text from all the book titles
within that cluster and then we'll
create a word cloud out of it here I
just randomly put some arguments here
you can customize it if you want and
then we just plot those word clouds
foreign
together to easily compare them or use
subplot to make two rows and three
columns because we have in total six
clusters
and Tada you can see that indeed for
each cluster we have different kinds of
topics cluster 2 is all about
programming and python cluster 5 is all
about deep learning python neural
network by tots and tensorflow so it's
quite interesting to look at those
clusters and we can print out here what
exactly are the book titles within those
clusters in deep note you can also
create an input field for example I can
create a drop down for choosing the
cluster number so this can be
interesting for creating a dashboard on
an app later
and with this k-means model you can use
it to predict or classify any book
titles to see which cluster it belongs
to for example here if I just type
tensorflow deep learning it will
classify it as cluster one which is all
about deep learning now on to the next
level which is the most challenging
level we'll be scrapping Amazon book
reviews and then try to create a summary
of the reviews for each of the book so
first of all to scrape the Amazon book
reviews we need the URLs of the review
page for each of the book the good thing
is that in the data we already have the
complete link to the product page so we
can use this product page to adapt it to
the all review page so if you compare
the two URLs we can see that the review
page has the product reviews the product
number and then we have the like a query
string here so I just go ahead and write
a small function here to translate the
product in your oil into the review URL
basically just try to split the product
URL and take the necessary parts and
then paste all the parts together now
with this function we can apply it on
the complete link column to create the
review URLs column and we just exclude
all the empty review URLs because there
might be some problem with that some
different structure Etc
for the Amazon review scraper I actually
found the code from Jeff James GitHub
since it often takes some time to
experiment and build a working script so
I just adapted from his code so kudos to
Jeff James for creating this I won't go
into the code to explain everything
what's happening here but basically
we're scrapping the first page of the
review page and put everything together
in a data frame after we are done we got
4 500 reviews in total and with this
data set you can use it as a toy data
set for for example sentiment analysis
but for this project we'll try to create
a summary for the reviews for each book
text summarization is the process of
shortening a set of data computationally
to create a subset or a summary that
represents the most important or
relevant information within the original
content thank you Wikipedia now let's
merge all the reviews together for each
book we can just do a simple Group by
here by the book title and contact
patinate the reviews together there are
two types of text summarization
extractive and abstractive extractive
text summarization just like its name
basically extracts the most significant
sentences of the text and create the
summary it doesn't change the sentences
themselves in the original Text data on
the other hand obstructive text
summarization is a little bit more
advanced because it can interpret the
context and paraphrase the text in a new
way so the summary is generated by the
model not just extracted from the
original Text data in the scope this
video we'll be doing extractive text
summarization and I think for the review
data this should be sufficient hopefully
we can extract the most important points
people talk about and ignore all the
useless rambling we use Birds model bird
is a Transformer based language model
trained by Google that came out in 2018.
to implement the extractive
summarization using Birds we can use a
library called both extractive
summarizer which implements a paper by
Derek Miller that aims to develop a
model to summarize lecture content it
looks quite interesting but I haven't
read too much into the details that if
you want to go down this rabbit hole and
understand how this model works you can
check out the link to the original paper
here now we're going to load the bird
summarizer model it can take a few
seconds
I'll first create the summary for one
random book in our data set in the
arguments here we can specify either the
number of sentences we want our summary
to be or use a ratio or the minimum
length or maximum length for the summary
and now let's print out both the summary
and the original text
the text is now only in one line I'll
just quickly wrap the text so that we
can see the full text
I don't know about you but I think the
summary looks pretty decent although
sometimes the flow of the sentences is a
bit weird but I think it does a pretty
decent job at removing the noise
information from the most important one
if you want to experiment with
abstractive summarization recently open
AI has made gpt3 model available to the
public which is cool and I bet it will
perform this task very well but the API
is a little bit pricey so it's up to you
if you want to play with it alright
after all the Blood and Tears hopefully
we are happy with analysis we can
finally publish this notebook to show
off our work there are two options for
publishing on deep note one is
publishing as an article and the other
one is as an app if we choose article
layout the article will look quite
similar to a normal notebook we can
choose to remove certain cells or hide
the code or the output from the cells
the other layout is the app layout this
is the case where you want to mostly
show the visualizations in the form of
dashboards or app and interestingly on
deep notes you can also freely arrange
the elements of the dashboard moving
them around to create your own favorite
app layout this makes it very easy to
customize our apps let's say here we
want to create a dashboard to show some
exploratory data analysis on our data
science books data set we can keep only
the necessary cell and then using this
link you can share it with anyone I hope
you enjoyed this project video and got
to know a lot more about data science
books if you decide to give deep note a
try let me know in the comment section
what you think about it I've put the
deep note link to this project and the
GitHub repo with the full code if you
found this video helpful please do
subscribe and you might want to check
out this video over here and other
project videos on my channel to get more
inspiration to build a great data
portfolio thank you for watching bye
foreign