Hello and welcome to this art tutorial.
So today we're starting a.
New branch of machine learning which is reinforcement learning.
And that.
Is taking us closer to the field of artificial intelligence, because robots.
And artificial intelligence that comes with.
It are partly built with.
Reinforcement learning.
So to.
Prevent for a minute disappointment
in the next tutorials, we're not going to build any.
Robots, but we will solve a very.
Interesting problem which is called the multi-armed bandit problem.
And we are going to.
Solve this problem with the two most popular algorithms for.
This.
These are the upper confidence bounds and the Thompson sampling algorithms.
So we're going to start today with.
Upper confidence bounds.
We're going to implement this algorithm in R.
And in this first. Tutorial.
We are going to import the. Data set and.
Explain what the problem is.
That is we will explain what the multi-armed bandit problem is about.
So let's start with the basics.
Let's set the right folder as. Working. Directory.
So let's go to our Machine learning A-Z folder.
Then part six Reinforcement Learning and section 32 upper Confidence bounds UCB.
All right.
And now in this folder make.
Sure that you have the ads CTR.
Optimization. Covers for click through rate.
So we are going to try to optimize the click through rate of.
Different users on an ad that we put on a social network.
And therefore that's the name of our.
Data set CSV file.
So if you have this data.
Set, you're now ready to click on this more button here.
And then set as working directory.
Good.
And now we are going to import. The data set.
So as usual we are going to call our variable for the data set.
Data set.
And then.
Equals read dot csv.
Here it is.
And then in parentheses we just need to. Add.
In quotes the name. Of the data set.
So it's right here.
And it is ad underscore. CTR.
Underscore optimization.
Here we go.
And let's not.
Forget to dot CSV. In the end.
And now we're ready to import. This data set.
So let's do it.
Let's select. This line and.
Execute data set.
Will import it.
So now.
Let's have a look by.
Clicking on data Set.
Right here.
Here we go okay.
So remember in part.
Three classification we dealt. With a problem that.
Consisted of.
Classifying and targeting
the users on the social network for some core company marketing campaigns.
Remember we had this business client of the social network.
That put ads. On the social network.
And then we made these classification models.
To target the users on the social network most likely
to buy this brand new luxury SUV that the.
Car company launched at a very. Low price.
And basically, to.
Prepare this marketing campaign, this car company prepared an ad.
That they would. Put on the social network.
And what happened is that the Department of Marketing
prepared some different versions of this same ad, you know, putting the car
in different sceneries, like, for example, one ad had the car on a beautiful road,
and on another version of the ad, the car is on the mountain
and maybe on another version. It's on a beautiful bridge.
Well, the.
Department of Marketing prepared different versions of this same.
Ad that they would put.
On the social network.
But the problem is that they prepared ten great.
Versions of the same ad.
The ten versions of.
This ad look great, so they're.
Actually not.
Very sure of which ad to put on the social network.
They want to put the ad that will get.
The maximum clicks, you know, so that most.
Users buy the SUV.
And so they need to put the ad that will lead to the best conversion rate.
So what.
This car company did.
Is that they hired us as a data.
Scientist, and they said. Okay.
I have ten versions. Of the ad.
We have a limited budget to place the ads.
On the social network. Because.
Putting these ads on the social network costs some money.
And so this car company would like.
A data.
Scientist to find the.
Best strategy, to quickly find out which.
Version of.
This ad is the best for the users.
That is, which version of the ad will lead us to the.
Highest conversion rate.
That's the CTR, that's the click through rate.
We want to find the.
Ad that will. Get the most clicks.
And so now.
Speaking of this, this is leading us to the key difference between what we.
Were about to do now and what we've been doing earlier.
Because earlier we had a data set with.
Some data containing independent variables and one dependent variable,
and then we did some clustering where we had independent variables only.
And now things are different.
We start with no data.
I know we have some data set in front of us.
But this is just a data. Set for simulation.
Because what happens in real life and we're going to pretend
we're in real life, we're going to pretend.
That we. Don't have any.
Data yet.
Well, what. Happens in real. Life is that.
We are going to start experimenting with these
ads by placing them on a social network, the different versions of the ads,
and according to the.
Results we observe, we will change our strategy to place these ads on the.
Social network.
So here. Are the different steps.
Of the process.
We have.
Ten versions of the same ad, ten versions of.
This ad trying to sell this cheap luxury SUV.
And each time a. User of.
The social network will log in to his account.
We will place.
One version of these ten ads, and that will.
Be around each.
Time a user connects to its account.
We will show him one version of the ad.
For example, ad three.
Version three of the.
Ad, and we will observe its response.
If the user clicks on the ad, we get a reward equals to one,
and if the user doesn't click on the ad, we get a reward equals to zero.
And we're going to do this for 10,000 users on the social network.
We're going to show the ad to 10,000 users.
We're going to.
Observe if the user clicks yes or no on the ad, if the user clicks on the ad,
that will give. Us one reward.
And if the user. Doesn't click. On the ad.
That will give us zero reward.
However, we're not going to show the.
Different versions of the. Ad to each user. At random.
There's going to be.
A specific strategy to do this.
And the key.
Thing to understand about reinforcement learning is that.
The strategy will.
Depend. At each. Round on.
The previous results we observed at the previous rounds.
So for.
Example, when we are at around ten, well, what happens behind the scene
is that the algorithm will.
Look at the different results observed.
During the first ten rounds.
And according to these results.
It will decide which version of the ad it will show to the user.
That's why reinforcement learning
is also called online learning or interactive learning.
Because the strategy is dynamic.
It depends on the observations from the beginning of the experiment
up to the present time.
And so now. What is this data set?
This is just some simulation of what is going to happen when we.
Show the ads to the users.
In other words, this is what God knows because we have no idea
on which ad each user is going to click on.
And that's what. This data set is telling us.
It's telling us for each round, that is for.
Each user.
Connecting to its account on which versions.
Of the ad the user is going to click on.
So let's give an example.
Let's explain what happens for the five first users.
So let's take. The first round.
And according to the simulation or according to God.
This first user of.
The social network is going to click on the ad.
If we show him the first version, the fifth version and the ninth version,
and if we show him the second version, the third version, the fourth
version, six, seven, eight or 10th version.
Well, this user is not going to click on the ad.
So this is what God knows.
But as far as we. Are concerned, we have no.
Idea on which ads this user will click on.
So what about the second user?
So that's the second round.
At the second round we show another version of the ad and according to God's.
Truth, the second user will only click on the ad.
If we show him the ninth version, the third user will never click on the ad.
Whatever version we.
Display, the fourth user will only.
Click on the second version and the.
Eighth. Version, and the fifth user will never click on the.
Ad, whatever. Version we show to him.
All right. So that's the idea of the problem.
And so we're going to build two algorithms the.
UCB. Algorithm and the Thompson sampling algorithm.
And these algorithms will decide. At each.
Run here.
Which version of the ad to show to the user.
And depending on the.
Reward the ads will get that as reward.
Equals one.
If the user clicks on the. Ad, or reward equals zero.
If the user. Doesn't click on the ad.
It will decide which ad to show to the.
User the next round according to the.
Previous observations.
And so we're. Going to have 10,000 rounds.
If we go down here.
We can see that we are showing the ad to 10,000 users.
And so of. Course the goal of the.
Algorithm is to maximize.
The total reward that is.
The sum of all the different rewards at each round
obtained by the different selections of.
The ads.
Okay, so let's do it.
Let's start with.
Upper confidence bound. The UCB algorithm.
But before we start implementing
this algorithm, I would like to show you something.
I would like to show you what. Would happen if.
We randomly select the versions of the ad at each round.
You know. No algorithm, no strategy.
Each time a user connects to its account, we are displaying one version
of these ten. Ads totally. At random.
So I actually prepared this algorithm.
We're not going to implement it.
Together because this algorithm.
Is actually not very relevant.
It's just to give you the motivation of what
we will implement in the next tutorials.
But this algorithm is actually provided in the folder.
You see it's this. Random selection file.
And actually I prepared it here.
That's the algorithm.
So as you can see I call this algorithm random selection.
Here I'm importing the data set.
As we just did.
So I don't need to execute that again.
And in this section I am implementing the random selection algorithm that.
Just. Consists. Of selecting.
At random one version of the ad. At each round. That is.
Each time a user connects on its social network accounts.
So I'm going to execute this section right now.
So here it is well implemented.
And we can see the different results.
Of this algorithm.
So the most. Important.
Result is the total reward.
That is this variable is the sum.
Of the different rewards.
Up to the last round.
That is up. To the 10,000 user.
And so what is this total reward.
This total reward is.
1242.
So what happened is that the.
Random selection algorithm randomly selected.
Each ad at each round.
We can actually see.
The random selections in this.
Ad selected list.
So we can clearly see what happened at ground zero.
For the first user.
The random selection algorithm selected the version number for then,
then the second round, the version number for,
then the third round the version number three, then the fourth run
version number one, and the fifth round version number four.
That's the random selections.
And so at each round, based on the.
God's true results, the selection of the ad generated a reward.
So at the first round, for the first.
User connecting on its account,
the random selection algorithm selected the ad number four.
And we see a zero here, which means that this first user.
Doesn't click on this ad.
So we get a zero reward at the first round.
Then what about the second selection for.
Well, we see here the.
Second round that there is.
Also a zero.
For ad number four, which means that the second user doesn't click on this. Ad.
And therefore we also get a zero reward.
And the.
Total reward that we observe here is actually.
The sum of.
All the rewards that it gets, whether it's zero.
Or one.
At the end of the 10,000 rounds.
All right.
So the.
Interesting thing to remember here. Is that.
When we randomly.
Select the ads, we get a.
Reward of 1200 and. 42.
Well, you know, there is this random factor.
And so of.
Course if we select.
That again we'll get another. Reward.
But it will be very close to this value here.
I'm going to do that again.
And as you can see.
We get. 1232.
And I can even do that again I get 1246 again, 1236.
So we always get a total reward close to.
1002 hundred.
So let's keep this.
Result in our minds, because then we will compare.
It to the total reward that we get.
Thanks to our more advanced algorithm, which is the upper confidence bound
and then the Thompson sampling algorithm.
So 1200.
Let's see how.
UCB and Thompson sampling. Beat that.
And now just. The last thing to show. You. As for every.
Algorithm we implement in this. Course, we get.
The exciting step in the end which is.
To visualize the results.
Well in this part, reinforcement learning, the.
Visualization of the results. Will consist.
Of visualizing a histogram where we see.
The different selections of the different versions of the ad.
So I'm going to show you what happened for our random selection algorithm.
Let's do it.
Press Command and Control plus enter to execute.
And here we. Go.
Of course. Since our algorithm randomly.
Selected.
The different versions of the.
Ads at each round, well, of course we get.
A nearly uniform distribution of the different versions of the ads.
The ten different versions of the.
Ads were selected more or less the.
Same number of times.
All right.
So that was just to give you a little extra motivation.
And now time to go pro.
Let's go back to our UCB.
Algorithm and start implementing it.
So remember 1200 total reward for the random selection algorithm.
Let's see how UCB beats that.
So we'll find out in the next tutorial.
And until then enjoy machine learning.