Another big case study from my book talks about
building a recommendation engine using LLMs.
If you recall an example that I used many lessons ago
about eBay using BERT's recommendations to build,
or embeddings to use recommendations, then this is going to sound very familiar,
because that's going to be our goal today,
is to use embedding engines to perform the act of recommendation for LLMs.
Some kind of content recommendation system.
Now, to understand the types of recommendations will be quite handy here. There is,
generally speaking, two types of recommendation engines,
or at least two types of recommendations.
There are content-based recommendations, where, as the name suggests,
the recommendation is being made because there is some content similarity
between something you have liked and something we will recommend.
For example, if I buy a lot of cat food on Some e-commerce website.
If the e-commerce website then says,
here's a ton more cat food because they are all similar in content,
that would be a content based recommendation.
A collaborative-based approach to recommendation would move
away from looking at the actual content of the item,
but instead look for similarities around user behavior. For example,
if you've ever seen the phrase, people who bought this also bought that,
that's a type of collaborative recommendation.
They are relying on historical data to say, well,
people who have bought the thing you were about to buy in the past, They also,
in short supply, or in short time, bought this.
You might see this as things like, oh, you just
bought a microphone, a podcasting microphone?
People who bought that also buy a, I forget what they're called,
a filter that goes in front of the microphone that would block your plosives.
But people buy those things together.
Yes, there is obviously similarity in the content,
but the reason behind the recommendation is less about that content-based
and more about the fact that people who bought A also bought B.
We are gonna be relying on a collaborative filtering
mindset to generate pairs of similar content.
And in this case, the content that I'm gonna be using
is a very massive dataset of anime content.
I specifically found this dataset because it not
only has Tens of thousands of anime titles.
It also has, literally, millions of human reviews and ratings of those animes.
Not reviews, ratings of those animes.
Rating them from a score from 1 to 10 for how much they like it.
So I'll be building a collaborative system
by defining animes as being similar if the
types of people or the literal people who liked both of them are the same group.
Meaning I'm not gonna be looking at the actual content
to decide if two pieces of content are similar.
That being said, we're gonna be building more of a hybrid approach by using
the content of the item to try to build a collaborative recommendation.
So I'll be hoping to train an LLM to produce embeddings based on the content,
which can be seen in this dataset here.
The content will include things like the name, the genre, the synopsis of the title.
We'll be combining all of those things into a single text description of that anime.
So in this case, I have an example. Cowboy Bebop is a TV show. There's a synopsis.
Its source is its original, premiered in 1998. Its genres are whatever, whatever.
For each of these animes, we will have a description like this.
This is our content-based part.
This is the part that's going to get shoved into an embedder, an LLM-based embedder.
I'm going to be then fine-tuning those embedders
Based on similarities between their audiences.
So here's what the system is going to look like at the end of the day.
We're gonna use a score called net promoter score or NPS to measure likes.
If you're not familiar with NPS,
it's a very common metric used in customer support where you
are giving a grade to a piece of content from one to 10,
and then you bucket one to 10 into three categories.
1 to 6 are called detractors, 7s and 8s are called passives,
and 9s and 10s are called promoters of the title. It's just a bucketing system.
9 and 10 is good, 7 and 8 is neutral, 1 through 6 is bad. That's it.
That's all Net Promoter Score is.
So the way our system will work is, given a user,
we will first look at all of the animes they have promoted, i.e.
given a score of 9 or 10.
For each of those anchor titles, which I'll call anchor titles, for each of those,
I will embed the content,
shove it into an embedding model to grab similar embeddings,
and use those as the basis for these might be your next recommendations.
We then grab the top scores, again, using this cosine similarity,
our embedding models, and then given an N, like give me three NMAs,
I will propose those three NMAs be shown to that user.
We will then check against a test set, as always, check against a test set to see,
did that user actually grade that anime at any given point?
If yes, we check, well, did they detract, passive, or promote that title?
And then given that score,
we will eventually be able to grade the embedding model
on how well it delivered recommendations for that user.
Now, to be very fair, this is arguably an extremely simple recommendation system.
Maybe the graphic doesn't really show that,
but simply using a recommendation embedding engine
to do the recommendations themselves is pretty,
not old school per se, but it's pretty naive, if I'm being honest.
However, it is often used as at least part of a broader recommendation engine.
The easy thing to do here is to say, well, look,
if they've already liked 10 things,
use an embedding model to grab 100 other candidates,
and then maybe send those through a more rigorous process and filter those out.
But a recommendation engine using an embedding engine
is an easy way to just grab potential recommendations.
Here, I'm just simply using those as our recommendations with no further checks.
So given that, I could now have to define what are similar animes. Because for,
as you're about to see, some of our experiments,
I intend to fine tune an embedding engine.
If you recall from several lessons ago,
when we talked about evaluating embedding Tasks, understanding embedding tasks.
One of the things we talked about was, well,
most embedding systems are built towards semantics, meaning two things,
if they are semantically similar, are embedded to be close to each other.
But remember, we are not going for a purely content-based recommendation system.
If so, we would simply use OpenAI's embeddings off the shelf
because it's going to put similar titles next to each other.
It's like saying, Oh, you like Pokemon, the TV show?
Have I got several dozen Pokemon movies for you.
You might get sick of always seeing semantically similar recommendations
just because you liked one of those contents in the past.
Sound familiar?
So we need a different way of defining similarity.
So in this case, I'm going to use a metric you've never heard of, well,
maybe you have, but I haven't mentioned it yet, called the Jacquard score,
or the Jacquard index, sometimes referred to.
The idea of a Jacquard score is given two sets of things, doesn't matter,
given two sets of things,
how many unique items are there among the two of the sets,
and the numerator is how many things in common do those two sets have.
So in this case, I will use the sets of promoters.
Given two animes, in this case, anime 473 and anime 94284,
on the second screenshot there, given those two animes, In that top row, first,
find the number of people who are, well, actually,
let me use the example I actually used here,
the Anime473 and 36732 on the second row there.
I first find the set of all Anime 1 promoters, people who gave it 9 or a 10.
Then the list, or the set rather, sets are unique in items,
of all the people who promoted Anime 2.
And then I see how many in common are there.
Of all the unique people who have promoted at least one of these,
how many in common are there?
So the Jacquard score as a set similarity metric now becomes a collaborative metric.
Because now I can literally say a higher Jacquard index means
people who like anime one also like anime two and vice versa.
Because this is gonna be a symmetric task. If I like anime one, I want anime two.
If I like anime two, I want anime one.
So my embedding should go both ways.
So my Jacquard score is showing if you like either of these, In this case, 0.945.
Of all the people who liked all of these, roughly speaking,
there's a roughly 94% chance that you also like the other one.
That's not exactly what your card score means, but roughly speaking,
a score close to one, which is the maximum, by the way, zero being the minimum,
a score close to one effectively means that if you liked one,
you were also likely to like the other.
Again, people who like A also like B.
The Jacquard score is literally quantifying that.
So...
Given that, I will use my new promoted Jacquard score,
which is the Jacquard score of promoters of pieces of content,
is now a score from zero to one that I can use.
That score from zero to one is also going to be
what I use to fine tune my embedding models.
Given two pieces of content,
use their Jacquard score as their intended cosine score.
If two pieces of content have a Jaccard score of 1, perfect,
I hope their embedding similarities are also close to 1. Remember,
cosine similarity is bound from negative 1 to 1.
So I'm basically only using data that could be from
0 to 1 to say, look, at worst, this is neutral.
Maybe give it to them, maybe don't, but at best, it's going to be 1.
You can change this around. You can offer negative scores if you want,
and it's more of a recommendation philosophy at that point.
In my case, I would rather only give anything between zero and one,
meaning maybe you won't like this, but who am I to say what you won't like?
At worst, I'll show it to you, you'll try it and hate it,
and now I have more data for my next round of fine-tuning.
So my promoted Jacquard score Now it becomes my target for my embedding models.
And what I'll do is I'll take open source embedding models, some fine-tuned,
some not, fine-tuned on my Net Promoter Jacquard score,
and compare them to off-the-shelf OpenAI models,
which at the time of recording, there are three.
Text embedding 3 small, text embedding 3 large, and text embedding Ada 002.
All of with which you can see their MTEB evaluation score of 61 to 64,
which I don't really care about because I'm not going for anything semantic here.
I'm trying to understand if I can recommend NMAs
based on my collaborative historical data.
Now, as one more just kind of to hit the point home, for these open source models,
what I'll be doing Is I'll be taking their two pieces of anime,
tokenizing their content-based descriptions by their genres, their synopses,
all that, running them through a BERT model,
and then training them to match the promoted Jacquard score
as their cosine similarity in an attempt to see if I can
use the content of the anime To recommend whether or not
this person is going to like it given their past likes.
This is why this is a hybrid.
I am using the content of the anime as my input to the system.
However, I am attempting to map the content to a collaborative output.
It's not always possible, but sometimes when you do it,
you get pretty darn good results.
So here I have six models.
Six models, which I have named Anime Encoder Bigger, which is an NPNet,
it's an off-the-shelf embedding model, where I change the context window,
because I'm allowed to do that for open source models,
and fine-tuned to the Jacquard scores.
Then I have...
Oh, and by the way, these are in order using the graph on the top right.
And the top right graph is showing the NPS scores
on the test set of my given recommendations,
normalized to one. So instead of nines and tens, we would look at 90% and 100%.
So the score is normalized to be at most one.
So in this case, our best model on the test set was a fine-tuned model.
The next one, which albeit was not that much worse on just NPS score alone,
was the oldest OpenAI embedding model.
The third one was the newest OpenAI embedding model,
and then it was an off-the-shelf model,
then it was a very small off-the-shelf model that was fine-tuned,
and then at the very, very bottom was that very
small off-the-shelf open-source embedder.
Now, if you look at the graph on the top right,
we can see that our scores are ranging from roughly 83% at worst to 91% at best,
that percent being how often we were actually
given a promoted animate in the test set.
So our open source model is barely eking out
a win across a closed source embedding model.
So at first glance, oh, and on the bottom right,
we can see that broken up for which index the recommendation was in,
meaning if it was in the top three scores versus the four, five, six,
or seven score versus anything after.
So if you're scrolling on content recommendation, Site of choice,
your first three recommendations will kind of
make or break how you feel about that system.
So it might be a good idea to check, are your first three recommendations good?
Are your next three recommendations good? What about your 20th recommendation?
Where's the drop off for where you just don't know what they want
anymore and you need them to actually try something and rate it.
So that's what the bottom right is doing.
Showing that our open source model is winning on the first three,
but actually barely loses on the next couple of indices
and then gets better at the tail on the bottom right.
So, so far, you might be thinking, okay, cool.
We fine tuned an embedding model and it is beating OpenAI at this task, awesome.
That being said, how much time did you spend slash waste Fine tuning this thing.
That was a lot of effort just to barely beat a closed source embedding model.
Well, with recommendation engines, we don't just care if they like the content.
Of course we care about that, but there's also something else we care about.
And that's called diversity of recommendations.
Maybe you've heard the colloquial saying that if you're on social media,
you're often in a little bubble where you're being
offered content that is not very diverse,
even though you might like it, right?
Well, in a content recommendation system, one could make the argument,
and I will make the argument, that I don't just care if you like the content,
I also care if I'm showing you a more diverse set of options.
Because if I can solve both problems,
meaning what I'm offering to you Not only am I sure you will like it,
having a high precision, I'm also sure that of all the things you will like,
I'm gonna be showing it to you at some point.
That's your recall.
Which is why precision and recall often come up in judging and betting models.
If I'm showing you a content based on my embedding
similarity, will you like it? Precision.
And of the things I think you will like, or of the things you will like,
am I showing you those? That's recall.
In recommendations, recall is often considered as diversity. Am I showing you...
A wide swath of content. And in this case, our best model,
meaning the one getting the best NPS scores, is also giving,
if you look at the scale here,
at least a thousand more pieces of content than the next in better below it.
So not only is our fine-tuned model getting better results, getting better scores,
it is also offering a more diverse array of content.
It basically wins both on precision and recall.
And if I can win on both of those things,
I'm in a really good spot in trying to build
my own recommendation engine using LLMs.