hello world it's Suraj and this is a
guide to Kor ml that is Apple's new
machine learning framework for all of
its iOS devices we're gonna go to the
development workflow of what it's like
to work with core ml we're gonna talk
about its API we're gonna talk about its
features we're gonna talk about its pros
we're gonna talk about its cons and then
we're gonna build a spam classifier for
text messages on iOS on an iPhone device
to really implement what we've just
learned and so that's what we're gonna
do today I'm very excited for this
because Hormel has a lot of hype right
but there's it's not very well
understood in terms of what's happening
under the hood so let's talk about that
right so Apple released this framework
this year at WWDC with a lot of hype has
always got to be hype when Apple does
anything right and so actually let's
start off with the demo right let's
start off with a demo so I'll type in a
text message and then I'll classify it
as ham or spam ham means not spam right
that's it that's what we're gonna do
just that simple thing okay that's gonna
be our demo so that's what we're gonna
do
it's gonna classify it as ham or spam
and we're gonna build that using the
messages framework which which allows us
to build a custom app that uses iMessage
or we could build an extension to the
existing iMessage app which is what
we'll do okay so that's what we're gonna
do today and okay before we get into the
features let's just talk about some
history here this is not the first
machine learning framework that Apple
has released in fact last year it
released two of them - not much
not as much hype as core ml had so get
ready for some acronyms the first
acronym is MPs CNN which stands for
metal performance shader convolutional
neural network and the second one is BN
NS which stands for basic neural network
subroutines
actually it's BNs so there's always more
acronyms there but both of these that
let developers build convolutional
neural networks both of them MPs CNN was
for the GPU and then B n NS was for the
CPU that was the difference there
and both of them let you run inference
on your device so remember there's a
different there's a difference between
training and inference none of these
frameworks are gonna let you train your
model on your iOS device but they will
let you run inference that means running
a pre trained model on your device
without needing an internet connection
so both of them were released last year
they released a set of data types and
function types and different types of
layers and for all intents and purposes
that let you build cnn's but that was it
it was also very hard to create a dot ml
to create a model file a lot of
developers were having trouble with that
so based on the troubles that developers
were having and just the fact that
machine learning is happening everywhere
and Apple is seems to be just now
catching on if they built core ml so
what core ml is also I want to say one
more thing so the reason they built two
different frameworks one for the CPU and
one for the GPU is because the CPU is
sometimes faster than the GPU for
inference not for training for training
the GPU is always faster so you kind of
have to guess and check with using both
but what Apple did was they said well we
don't have to we don't want to have you
have to worry about all these things so
we'll build a framework around both of
them so that runs on the CPU and the GPU
so that's what core ml is Hormel is
built on top of the previously to
release libraries and acts as a layer of
abstraction on top of them and when I
say abstraction I mean abstraction
seriously it is almost too easy to use
like you'll see what I'm saying it's
very easy to use and then on top of core
ml they built three other api's one is
for one is for one it's called vision
and it's for all of your image analysis
tasks object recognition classification
and then there's foundations which is
for natural language processing that's
for sentiment analysis predicting the
next word in a sequence sequence all
sorts of text related machine learning
and then there's gameplay kit and it
allows your app to evaluate decision
trees so those are the three domains
that build on top of core ml and so the
reason they built all three of these on
top of quorum L is for modularity right
so let's say you have an app and what
you want to do is classify
where somebody is so part of your app is
tagging everything you see so let's say
you're on a beach it would tag all the
things that that the app sees it would
tag the water it would tag sand it would
tag sunshine maybe some beach volleyball
players and then once you have all those
tags using the vision library then you
can use the NLP library foundations to
then take those words and classify what
scene it is and it would say based on
these words this must be a beach and
then you'd output a beach so that's just
one example of how these libraries can
interact together and so for the
development workflow what Apple has done
is it's essentially built a PDF for
machine learning models right the PDF is
a standard format right and so that's
what they try to do with machine
learning models it's in the dot ml model
format and so you can use the dot ml
model on any iOS device Mac Mac Apple TV
iOS or iPhones all of it and iPad all of
it and so it's a very simple process
it's a two-step process the first step
is to load a pre-trained NL model the
second step is to make predictions with
it right the training isn't happening on
the device so it's that simple so but
sometimes and so when I say using a pre
trained model Apple has a list of these
pre trained ml models that I have a link
to in this Jupiter notebook and they're
they're all these very popular models
mobile net squeeze nets googling Nets
ResNet these are all convolutional
networks by the way inceptions huge vgg
all of their pre trained models are
convolutional networks and so you can
just use those but let's say that you
don't want to use their there are
pre-trained models let's say you want to
build your own then it becomes a
three-step process the first step then
becomes to convert your pre trained
model in whatever machine learning
library you're using whether it's cafe
or scikit-learn into a dot ml model
using a tool that they've created for
this called core ml tool it's a it's a
Python library we'll talk about that and
then you do the next two steps load the
model and make the predictions and so
here is an amazing list of all sorts of
machine learning models that you can use
for core ml
right there's a bunch of pre-trained
core ml models and then they have models
for different libraries that you could
then convert to Cornell now I had do
have a very disappointing things to say
they don't have support for tensorflow
I know what are you
there is no tensorflow it's okay
whatever whatever whatever you can build
your own conversion script for your
tension flow models to run encore ml in
fact there must be something on github
I'm just making a prediction right now
prediction anyway so it's either a
two-step or a three-step process and so
it's the ml model is like the PDF of
machine learning all you have to do is
once you've trained it you drag and drop
it into Xcode so you just drag and drop
the dot ml model and then you drag and
drop your data set or whatever you want
to test your model on and then you can
call it very very simply I'll show you
the code right here here's a code the
sample code but basically when you
import the model into Xcode it's gonna
look like this you've got it's it's
gonna parse the data inside of it it's a
very it's very neatly packed for Xcode
to read so it'll be able to say oh here
are your inputs here are your expected
inputs and their data types and it's
description here's the output it'll tell
you the learned parameters and the
weights that means the weights and the
biases as well so it's a very clean
format for Xcode and so once you've
imported that model you can call it just
like this you can call it by saying let
model equal ResNet 50 let's say for
example that's the one we're using and
then you'd say here's my image let pixel
buffer up type CV pixel buffer equal my
image wherever it's located and then
I'll just make a prediction using model
dot prediction feed it the image as the
parameter and then print it out right
that's it that's your prediction it's
that simple
so let's talk about the pros and cons
right so here's an image of all of the
machine learning libraries that are
compatible with core ml that means that
that means that the core ml tools Python
package can convert a model trained with
these frameworks into the dot ml model
format that you can then run that you
can then use to run inference on your
device caffeic arrows XG boosts live has
SVM scikit-learn Tory right so the pros
are it's optimized for on device
performance which minimize minim
memory footprint and power consumption
that's the one thing you know about
apples library as opposed to any third
party framework you know that it is
gonna be optimized AF for its devices
because they make the hardware and the
software we also know that Apple really
cares about user privacy that means use
the privacy of users data so you that
means you don't have to send the users
data to the server it stays local and
encrypted and it means that you can run
and because inference is happening on
device
you don't need an internet connection so
they could be in like a ditch or a
prison or why am I think of these weird
places but some are without internet
connection and then it will be able to
perform inference there right you don't
need an internet connection and it
decides itself remember it's a it's a
layer of abstraction on top of the
previous two machine learning libraries
it decides itself whether to run the CPU
or the GPU or both so it's it's it's
self optimizes what it's running on and
because it can run on the CPU you can
run it on the simulator which you can't
if it were to just run on the GPU
because the simulator doesn't support
the GPU yet and it supports many model
types that's the last Pro it supports
support vector machines they're all
listed here
it supports three types of neural
networks a convolutional network a
recurrent network for sequences and then
a feed-forward network right and then
you've got tree ensembles as tree
ensembles like random forests and
boosted trees linear regression and
logistic regression okay so that's it
for the pros and we have to talk about
the cons right it's not perfect the
first is that there is no there is only
native support for supervised machine
learning models that means that models
that require labels there is no support
for unsupervised models or for
reinforcement learning so that's that
that can be a big a pain and no training
on device right so you can only perform
inference on a device also it only
supports certain layer types so you can
just create a new layer type yourself
and add it to core ml it's it's
impossible to extend core ml's of native
layer types and it only supports a
specific set of training tools not
tensorflow however you can write your
own custom conversion script for
tensorflow
models you can't look at the output
produced by the intermediate layers
only the output layers it only supports
regression and classification that means
no none of the unsupervised techniques
like clustering or ranking or
dimensionality reduction and my biggest
gripe is this kind of ties into no
training on device but no federated
learning right federated learning is a
technique Google actually published a
blog post on this very recently but
basically you can utilize all the phones
that you deploy your app to you can
train it on their data locally instead
of having it train on a server and so
you can combine all of the that training
into one big model that is then deployed
to everybody so every you're you're
learning from all the devices that
you're that you're deployed to and so
killed you are core ml is super simple
we saw a very simple code simple already
but it's limited in its functionality so
if you want full control you're gonna
have to DIY with either the two native
libraries that it's built on top of or
you can just use another third-party
framework or ml is not the only way to
do machine learning on iOS devices there
are other ways surprisingly right and
I've got a list of them right here there
are a lot of third-party frameworks that
work with iOS that leave you a bunch of
different machine learning tasks so if
core ml doesn't fit your needs
specifically or you don't want to extend
it then go ahead and use these alright
so what are steps in this tutorial art
are steps are going to be first in
Python to look at what it means to
import the data set train a model and
then convert that trained model into a
dot ml model then we'll go into Xcode
and then in Swift we'll drag and drop
our data sets are trained ml model into
Xcode then we'll write our basic
prediction code and then we'll run the
app alright let's get started with this
so for our Python code so I'll write out
the swift code but we'll just glance
over the Python code so for our
dependencies for the Python code we have
to write we have to one is for numpy for
matrix math and then the second is
scikit-learn in all of its sub modules
right we're gonna run we're gonna train
three models and then we're gonna pick
the best one and I'll talk about what
those three models are but you can get a
little hints from here as well as two
techniques to vectorize the each of
these models so in total they'll be six
different
pipelines that will build so for each of
the models we'll try out to different
vectorization techniques right so two
two and two so which makes six right so
that's it for scikit-learn for numpy and
so core ml tools the the Python package
only supports Python too so if you if
you have Python three you can use these
commands to initialize your Python
environment and then you can import core
ml tools right just like that so I'll go
back up here compile but uh okay and so
now we can import our data so let's take
a look at our data set really quickly
here what does our dataset look like we
have an SMS spam collection data set and
it's a bunch of human labelled SMS
messages either ham or spam right spam
or not spam alright so that's the data
set we're going to use and so what we
can do is we can open that data data set
it's a txt file well go through it we'll
split all the lines and then we'll
convert it into a training and a testing
set using scikit-learn strain test split
function we can print out the training
set to just see you know what it looks
like it looks kind of messy but these
are some text messages and it's just got
a label ham or spam that's it right
binary classification right so that's it
for importing our data and now we can
look at our models right so we're gonna
use three models and each of these
models I've created a video just
explaining the entire model so
definitely check out each of these
videos in my math of intelligence
playlist I've got one for the
multinomial naive bayes
support vector machine and for the
random forests which are the three
models that we're gonna use but to just
go over them at a high level multinomial
naive bayes is a specialized version of
the naive bayes classifier these are all
classifiers where it's just computing
conditional probabilities for all the
words in a set of documents right it's
just iteratively computing conditional
probabilities but definitely check out
my naive Bayes video on that and then I
have we're gonna use a support vector
machine so it's a little refresher here
writes the support vector machine is a
type of machine learning model that can
classify two different classes and what
it does is it builds a hyperplane by
using the by using these support vectors
that are the points that are the closest
to each other between the two class
and then maximizes the margin between
them and draws a hyperplane right in
between them and so when we add a class
depending on what side of the line it
goes on we can classify it and for a
random forest a random forest is a set
of decision trees you know yes no it
just it asks a series of questions and
then it the result is what it what
something is it'll classify it and what
we can do is we can say let's create a
bunch of decision trees have them all
classify some data have them vote and
then pick them pick the winner right
what is the most likely class based on a
set of decision trees random forest
right school that's why it's called a
forest because this set of trees and
then in terms of vectorization
strategies we're gonna use either the
count vectorizer which is just basically
a bag of words it just counts the number
of times a word appears or we could use
a more advanced way of doing this called
tf-idf and what this is is it's a
technique to score each vector right for
a term I in a document J so for each of
the words what we can do is we can say
the score it's tf-idf score is the
number of occurrences of I in J times
the log of the total number of documents
or SMS messages in our case over the
number of documents containing hi and so
we'll compute both of these the count
vectorizer
and the tf-idf for all three of these
models giving us six different pipelines
to use right so here's here's how it's
building the pipeline right we have six
different pipelines for each of the
models we'll try out both tf-idf and
count vectorizer
once we have all those pipelines we'll
put them into a list and then we'll
perform classification using all of them
and so we'll find this is running right
now or running live will find that the
support vector machine using tf-idf wins
right this has the highest accuracy 98
98 98 so that's the one that we're going
to convert into a ml model file so what
we'll do is we'll say ok we'll get all
the ordering of the words and from this
dot txt file and then we'll vectorize it
and then we'll convert that model what
we'll say here's the model the linear
SVC fit it to the vectorized data and
then use core ml tools to then
convert for specifically for
scikit-learn it's got converters
scikit-learn dot convert the model based
on the two rows the message and the
label and that will create a core ml
model in memory and then we can save it
using the save function as a dot ml
model file so when we when we compile
this it's gonna save it and then we can
then drag and drop it into Xcode okay so
now that's the first part we did it
we've trained a model in Python on our
machine now we're going to deploy it to
Xcode so let's check out Xcode now so in
Xcode what I what I have here is it's is
the a skeleton file it's a skeleton file
all it has is it's got a storyboard with
you know label and then a button right
and the the button and the label are
wired back to the view controller right
this is in swift we've got one view
controller with all the logic which is
no logic as of now and then we have the
storyboard right so what's happening
here is that for the view controller we
have an IB action that's gonna fire
every time the button is pushed so
inside of here we'll put our prediction
logic and then we have this other
function for it to perform tf-idf that
we're gonna fill out okay so let's let's
do that but first let's drag and drop
our created ml model to our project just
like that we'll drag and drop it and
also our spam collection our data set
because we can test it out on that on
the training data as well okay so then
let's go back here and let's start
writing this out let's see what this
looks like okay so so first of all let's
retrieve the text that the user has
typed in where we want to retrieve that
message so then we can classify it as
spam or not spam and so what we'll do is
we'll say if the text is copied let's go
ahead and vectorize that text by saying
let vector equal tf-idf and now we
haven't actually written out the
functionality for this function yet but
we can just say well if we feed the text
into tf-idf it'll create a vector for us
and then in a do statement
make the prediction right we'll say let
the prediction equal try the message
classifier right that's our message
classifier make the prediction the
message classifier is named right there
so we can just call it just like that
and then we'll feed it as the parameter
the vector that we computed using tf-idf
and that's going to store the prediction
we can print we can print out the
prediction for our own logging purposes
and then we can - the what we can send
back that data to the interface or the
storyboard by saying set the label to
the prediction and then so we'll also
have a catch statement so if it can't
make a prediction then we'll set the
label to no prediction just like that
and that's it for our logic right here
and then we'll go to our TF D F function
and so now let's write out this DF DF
code so what's happening here is we can
say first we want to import the words
ordering and the words so we'll go ahead
and import both of those just like this
and so it's it's finding where both of
these text files are and the next step
for us is going to say is going to be
let's create a do statement and the next
step for us is going to is going to be
let me just write this out like this
just like that and so the next step for
us let me just paste this in right here
boom like that or delete this okay just
like that and so we're gonna say let's
retrieve the ordering so so inside of
our do statement we'll retrieve the
ordering data we'll retrieve the SMS
data we'll remove the trailing newline
from both of them as well right removing
the trailing newline and so once we've
retrieved both of those text files we're
going to vectorize our words just by
using this we're gonna vectorize our
words we're gonna have a collection of
each of those words that we're going to
split by the separator the empty
space and then we will vectorize each of
those words and so now for the
vectorization part here's where the real
meat of it it goes right so we're gonna
say for each word let's count the number
of times the word shows up in an SMS all
right we're going to count the number of
times it shows up in an SMS and then
we're going to multiply it by the log of
the total SMS messages divided by the
SMS messages that contain that word and
so that's going to give us two tf-idf
score which we can then store in the
vectorized array and then we'll return
that array of all the tf-idf scores of
each of the words in the SMS documents
and that's it for that and then we can
go ahead and compile this and it works
just like that so overall I think this
is a very fun library it's gonna it's a
great way to introduce regular
developers who've never done any kind of
data science or machine learning to
machine learning because it's so simple
to use if you've never coded for iOS
before I would highly recommend it it's
a lot of fun and if core ml doesn't give
you what you want you can always use the
two frameworks that it's built on or you
could just use a third-party library all
right that's it please subscribe for
more programming videos and for now I've
got to be thankful for automatic
reference counting so thanks for
watching