Hello my friends, and welcome to this new practical.
Activity, the first one of part six.
Reinforcement Learning.
Where we. Will be implementing the.
UCB. Algorithm.
Upper confidence bounds.
It is one of the most exciting branches of machine learning
because you know it is the one. The closest to.
Artificial intelligence in the sense that, you know, we are making some programs.
That.
Play some actions just like. A robot.
So this is very exciting.
This is one of my favorite branches, if not the number one.
And so I'm so excited right now to.
Teach you about the.
Fundamentals of reinforcement learning.
Especially. I'm so excited to implement two.
Of the best.
Reinforcement learning models.
With you, which are UCB.
And Thompson sampling.
So first in this section we will implement.
UCB Upper. Confidence Bound.
And once again we will apply it on a business.
Case study.
Which you know will be the next.
Part of the story we.
Had in part three. Classification.
You remember that. SUV.
You know, that brand new. Luxury SUV that this.
Car company was trying to optimize the targeting thanks to classification?
Well, this time we were going to optimize the online.
Advertising.
Meaning that we are going to find the.
Best ad.
You know, among different ad designs.
The best ad that.
Will convert the maximum customers to click on the
ad and, you know, potentially.
By the product, by the car. Okay.
So I will. Explain this story a bit more later.
But before.
Let's just make sure everyone here is on. This.
Same. Page.
I gave you the links to this whole folder right before this.
Tutorial in the article, so make sure to. Click it.
And now we should all be on the same page.
So let's do this.
Let's end to. Part six.
Reinforcement learning. And we're going to start.
As we said with.
Upper confidence bound. UCB.
And so this time you not only.
See the two folders. Python and R.
But you also see the full slide of the UCB.
Algorithm. And you will also.
See the full. Slide of the Thompson Something algorithm in the.
Other folder.
So let's have a look at it. Make sure to download it.
And if you want you can print it and post it on your wall.
There you go.
You have the three steps of the UCB.
Algorithm which. We.
Are going to implement together. You know.
I will actually give you a lot of.
Exercises in this implementation.
I will tell you before. We implement each of the. Steps.
To implement it. Yourself.
So at first you. Will have to implement step one.
Then once step one is implemented we will implement step two.
You will do it first before we do it together.
And then. Step three. So you see it will be a very.
Learned by doing. Process.
All right so that's the slide. Make sure to download it.
And now let's go into Python first
to implement the UCB algorithm.
All right.
So as usual you have two files here you have two data set add CTR.
Optimization CTR means click. Through. Rate.
And that's what we are going to optimize thanks to upper confidence bound first.
And then Thompson Sampling.
And then we have the implementation. Of course.
Upper confidence bound in the Ipynb format, which you can open with either.
Google Collaboratory or Jupyter. Notebook.
All right.
So as usual let's start by.
Explaining what.
This data set is about.
So as I said, we are doing.
The next. Part of the story of this car dealership.
Trying to.
Sell. That new. SUV.
We've already done the targeting. You know.
We've already.
Optimized tutoring thanks to classification in part three.
And now we're going to optimize.
The click through rate.
Of some ads we're going to make. For this car.
Okay.
So what happened exactly is that the.
Advertising team.
Prepared.
Ten different ads.
You know, with ten different designs.
For example, on one ad. We will see the SUV in.
A beautiful mountain.
On the other ad, we will see the SUV in a futuristic city.
And another. Ad, we'll see the SUV.
In a charming city, you know, like a charming city in the south of France or.
Italy. On another. Ad, we will see the.
Car on the moon, you know why not on another.
Ad? We'll see the car.
On a beautiful countryside cornfield.
You know, something like that.
So basically, all the ads.
Have different designs.
And advertising team is wondering.
Well, which ad. Will convert the most, you know.
Which ad will attract.
The most.
People to click the. Ad and then potentially buy.
The SUV.
So we have these ten different ads. And what we're going to do.
And that is. The process. Of online learning.
We're going to show these ads to.
Different users online.
You know, once they connect.
To a website or to.
A search engine, you know, it can be the ads.
That appear at the top of a page.
When you type.
Of research on Google, we're going to show one of these.
Ads each. Time the user connects.
To the web page.
And then we're going to record.
The result whether this user clicked.
Yes or no on the ad.
Okay, so just to recap, there is a first user.
That connects to.
Let's say a web page or.
Algorithm, which. Will be here.
First, UCB will.
Select an ad to. Show to this user.
And then the user will. Decide to click.
Yes or no on. The ad.
If the user clicks on the ad, we will record it as one.
And if the user doesn't click on the.
Ad, we will record it as zero. Okay.
And then a new user connects to the web page and same.
The algorithm. Selects.
An ad to show to this new user.
And if this. New user clicks the ad.
Then it's a one.
And if not, it's a zero.
Okay.
And we're going to do. This for lots of users actually.
10,000 users.
And that's what. This data set is about.
However, now. There is something you must.
Absolutely. Understand.
And that. Is very. Very important.
Make sure to understand it and make sure to rewind if this is not understood okay.
So I'm going to explain this.
Please listen carefully.
So you know, in. Reality.
What happens is that. Users.
Connect one by one to the. Webpage.
And for each of them we.
Successively show. Them the ad. Right.
So everything happens in real time.
You know, it's a dynamic process.
It's not a static process with a static data set
which was recorded over a certain period of time.
It's a real time process.
And therefore the.
Only way to simulate this would be either that.
I, you know, make ten real ads right now, you know, general.
Ads of a car.
Then I open a Google AdWords account, and then I show.
The ads for real to.
Some users, you know, real persons connecting to the website.
Of course, I'm not going to do.
This because first of all, this is costly.
And then, you know, this.
Would deceive the users.
Well, you know, I would have to really sell a car somehow.
So of course, this is.
Not an option and therefore I have to make a.
Simulation. Okay. I have to make a simulation.
And this simulation is. Exactly.
Given by. This data set.
Because in this.
Data set what happens is that each row.
Corresponds to. The. Different.
Users. Connecting to the. Web page.
And to whom we're going to show the ads.
And then. Each. Column of. This data set.
Corresponds to the different ad okay.
From ad one to AD ten.
And this. Data set.
Is a simulation in the sense.
That each. Time a user connects.
To the web page, well, this data set tells us.
Even if we wouldn't know.
In reality, this data set tells us on
which ad the user of the row would click on.
You know?
So for example, this first user, you know, this corresponds.
To the first user to.
Whom we going to show. The ad.
And what the sales mean.
Is that this user would click.
On add one. If we show this user ad one.
Then it wouldn't click at two.
If we show at two because there is a zero.
Here, then the user. Wouldn't click at three.
If we show at. Three, it wouldn't click at. Four.
If we show ad four, but then it would click at five
if we showed at five and etc..
So in other words.
We know thanks to this simulation, you know this.
Data set doing a. Simulation, we know.
That this user would only click at.
One at five and at nine.
If we showed these. Ads.
And then if.
We showed all of the other ads at two, at three, etc., up to.
At eight and AD ten.
Well, this user wouldn't click the ad.
Now, I know we wouldn't know.
That in reality, but that's. Why I'm saying.
That this data set is a simulation.
And this. Is the only way we can actually.
Run the. UCB.
Algorithm or the Thompson algorithm.
If not doing it for real, you know, with a real advertising campaign.
Okay, so I hope it's clear.
Please rewind if it's not clear. Because.
I think I've said all. The keywords.
You know, this. Data set. Is a simulation.
Therefore, for all the users.
You know, corresponding to these. Rows.
We know on.
Which ad the users will click right.
This user, for example.
Would click only on ad number two. Or.
Ad number.
Eight, right, but wouldn't click on all the other ads.
And so that's the only. Way we.
Can indeed simulate. The.
Thompson sampling or. UCB algorithm.
All right. So I hope it's clear. Then we have let's see.
Let's scroll down to the bottom.
We have in total 10,000. Users as we said.
And so we're going to run. The UCB.
Algorithm first and then the Thompson something algorithm.
To figure out the ad that has the.
Highest conversion rate.
Right. The ad on.
Which the users clicked. The most.
So I know that.
We could do.
It, for example,
with a naive strategy, you know, a naive algorithm, like a simple one.
Where we collect.
Some simple statistics.
To see which.
Ad is most frequently clicked on.
But remember, as Carol explains in the intuition lectures,
each time we impress an ad, you know, on the website.
Or the Google.
Search engine, well, this incurs a. Cost, right?
It has a. Cost to impress.
Ad. Therefore, we need to figure out as fast as.
Possible, you know, in the minimum number of rounds, because you know.
The users here.
Are represented as rounds because we.
Show that add to.
The users one by one, as in one round after.
The other.
So we.
Need to figure out and a minimum number of rounds
which ad converts to the most meaning.
Which is the best ad to which the.
Users are most. Attracted to.
And that's why we need a stronger.
Algorithm than a simple. Statistics. Algorithm.
And that stronger algorithm will be first. UCB.
And then Thompson sampling. And we will even see.
Which of the two. Is the most. Powerful.
All right.
So I. Think that's enough explained for this. Data set.
Now we're going to start the implementation I can't wait.
This is a very. Exciting and actually.
Widely used.
Algorithm in online advertising or digital marketing.
So let's do this.
Let's click this.
Implementation and then let's open it.
With Google Colaboratory.
Or Jupyter Notebook as. You want.
All right.
So now it is loading it.
It is.
Loading the. Notebook. Laying out the notebook.
And now here you go.
Welcome to the UCB implementation.
All right.
So as usual we're going to create a copy because this.
Is in read only mode.
So in order to re-implement this from scratch we're going to click.
File here.
And then save. A copy and drive.
This will create a copy. Inside.
Which we will be able to.
Re-Implement the whole algorithm from.
Scratch.
All right.
So it is opening you notice. That I.
Have my data preprocessing template opened.
That's because we will use it very quickly.
You know just to actually import the libraries and the data set.
And so now.
Before we start let's delete all the code cells.
Here, but not the. Text cells. And
very soon we should.
Be able to start okay.
So that's a. Simple implementation you know simple. Structure.
But this cell will actually be long.
And that's where. You will practice by doing first.
Some steps of the implementation before. We do it together.
And so let's have a look.
Welcome to. The UCB. Implementation.
We will start.
First by importing the. Libraries.
And then we will import. The data set.
Then we will implement the full UCB algorithm
just following the steps on the slide you know with the three steps.
And finally we. Will visualize the. Results.
And by that I mean that we.
Will plot a histogram where we will clearly see the ad that was.
The. Most selected.
You know.
And that of course.
Added.
Was identified as the strongest ad, you know, the most.
Attractive ad for the users.
And one thing I forgot. To say.
And which is really, really important.
This data set actually suppose
that each ad has a fixed conversion rate.
So ad number one.
Has a. Certain conversion rate.
AD number two has another conversion rate.
And then same for all. The other. Ads. And that's.
Of course because it is a required.
Assumption.
Of both the UCB.
And Thompson sampling algorithm.
Basically reinforcement learning algorithms for online learning.
And that's, you know, anyway.
The case in reality, for. Example.
With the.
Slot machines in the casino. Well, they all have a.
Fixed conversion rate unless they change it over time.
But that's another. Question. There you go.
Usually an ad that you show online.
Has a. Fixed conversion rate because. It will.
Convert over time, the same rate of people.
So we will assume this.
And besides it's close to reality.
But there you go.
That's an important assumption of online learning.
All right. So now. We're ready.
We're ready to begin.
This implementation I hope. You're excited.
I hope. You understood.
This data.
Set and the fact that we're doing a simulation because. We.
Actually don't have too much of. A choice.
And so if everything is all good.
Well, my friends, let's begin this implementation in the next tutorial.
And until then, enjoy machine learning.