You let's take a look at our code the rest of our notebook here to
assess some open source alternatives as our generator in a retrieval augmented generation system.
So for our first open source candidate, we can use something called llama3. So we talked
about Llama in a previous lesson as an Open Weights LLM for from Meta and Llama
3, as of today at least, is Meta's most recent family of the LLAMA models.
I'm going to use their smaller version, the 8 billion parameter version, to see if we
can simply drop in a different LLM into our RAG system and have everything still work.
This is going to be a good test because it's going to basically tell us that how good our prompt is. Because
as I mentioned in the last lesson, a good prompt doesn't just make your LLM more performant, it should do that.
It's also going to be your safety net should you want to change
to different LLMs. Because as we're starting to understand you and I, at
least LLMs are in a lot of ways all the same.
Some are bigger, some are smaller, some can generate different languages, some can generate only in English.
They're trained on different types of data, they have different techniques under the hood
to make them faster or more performant. But at the end of the day,
virtually every single LLM that we call generative text AI is an autoregressive LLM
that has been trained on the autoregressive language modeling task, which implies that a
well structured prompt should be able to translate between models.
So let's see if that works.
We're going to go ahead and grab the tokenizer here of our Llama 3 billion.
Specifically, we're going to use the instruct version of Llama 3, which from a previous lesson.
This means that this LLM has been chat aligned, meaning it is not
simply trying to regurgitate text that it saw on the Internet. It is
actually specifically trained to answer a question or hold a conversation.
So to make this work, I went ahead and deployed a Llama 3 model on a hugging face
endpoint. Now, if I'm being honest, at a future lesson, I will be walking through how to do
this for the sake of our time here today, in this lesson, all you need to know is
that behind this endpoint, which is a hugging face endpoint, huggingface cloud lives llama3 8 billion.
So, meaning if I were to pass in a prompt into
this function, I will be getting back a response from llama3.
Now, technically I could have used the tokenizer to Format the prompt into the string. But I wanted to write it
out just to make it painfully clear that any LLM, GPT and Llama 3 included all have these special tokens.
Beginning of text, start header ID all of these special tokens that I'm highlighting end of text ID, start header id
all of these are defined by meta and in a later lesson, lesson 10 I believe we're going to build our
own chat LLM and define our own special tokens. So this is going to be a pattern that we will also
follow. But just to show you, this is the pattern for llama 3 if you were to look under the hood,
this is what's actually getting passed into llama 3 when you ask it a question on some chat interface.
Anyways, we will call our API with this prompt and we will specifically
set any inference parameters here, grab our output and simply return it in
this function. So if I typed in one one question? Verbosely I will
see here is our prompt and Our response from llama 3 is 2.
If I pass suppress equals 2 I'm simply just going
to be given our string here of two Easy.
So then to make Llama three work I simply just need to create a new chat LLM. So
this is why I structured it this way, so we don't have to rewrite the whole rag system.
In theory I should just have to rewrite the chat LLM to not be gpt4
but to be llama3 instead. So similarly I'll be able to pass in a temperature.
This is a different inference parameter that OpenAI does not have access to.
Do sample do sample true is default.
It means you're picking tokens from that probability distribution.
If you set do sample to false, it will always pick the top token. So it's basically
like setting temperature to zero in theory and setting top k to one. We're just grabbing the
top token and then the number of max new tokens I want to send our generate which
is going to match our generate function, our method from our GPT4 at the top of this
notebook, just going to pass it through our llama 3 method or function and return the response.
So if I instantiate a llama chat LLM and ask it who is the US President again?
As of today, me writing this I get as of my knowledge cut off. The current president is Joe Biden, blah blah blah blah.
So in theory now it should be and spoiler alert you're looking at should be
as simple as taking the ragbot and instead of chatgpt I e GPT 4 I
am simply going to pass in my llama LLM with the same stop pattern.
I'll set verbosity off just to show you the result. And if I write I lost
my Medicare card, this same semantic search is still happening. The same threshold is still happening.
All that pinecone stuff is completely separate from the actual generation.
But we can see here, this is the response. If you've lost your Medicare card, you can request
a replacement online or call this number to order a new one to be sent in the mail.
A decent response. Is it better than the one that GPT4 gave? I don't know. That's up to you.
Frankly, that's subjective. Now this one doesn't give the URL, so that's one thing to
notice. But I technically never told the AI to return the URL if it found
one. So really, to be fair, I can't really knock at any point.
It's not doing anything wrong or anything I told it to do is not happening.
But if we think about it, this the AI response is still just going to
look pretty similar. In fact, let me go ahead and print this out just so
we can see what our context looks like. Print this I lost my Medicare card.
If your context the same context that I'm getting from our pine cone here, our
score, our thought has sufficient information. And then this is the LLAMA response that was
shown up here. So our prompt seems to be pretty good in the sense that
the same exact prompt between GPT4 and Llama 3 still seems to work.
Again, is one better than the other?
That's the part of evaluating generative text, which can be tricky.
Now as I kind of add a left field here just to kind of show you again and kind of
get you pumped for the world of open source models, I'm going to show you one more open source model.
What I like about Llama is that it's a drop and replace for GPT4.
It is effectively the same type of system, just smaller and open weights.
Some open source models are specifically fine
tuned to be retrieval augmented generation models.
And this is where it gets really interesting in the world of open source.
So in a previous lesson we talked about Cohere as a provider, as
a competitor to OpenAI and a provider of closed source autoregressive models.
Command R I'll show you the blog post here. Command R is a
model specifically made by Cohere. We're on Cohere's website, but specifically made for
rag and tool use, which we'll Talk about in our next lesson.
As companies like Cohere start to learn about what we the consumers are doing with
their AI, it is in their best interest to actually create models that do those
tasks. So to that end, let me show you what Command R can do.
Now the reason why this is a little bit of out of left field is I have to move away from my
Ragbot implementation because the architecture of command R. Well let me rephrase that, not the architecture of Command R. The response from
Command R is not going to be a drop and replace like llama 3 is. Let me show you what I mean.
If I were to grab the tokenizer for command R, which again this
is an open source model that we can find on hugging face.
If I pass in a conversation role user content, what is the biggest penguin in the world? This is
the example I have on their hugging face. We'll get to our example in the next code block.
The way this model works is they cohere have a predefined rag
prompt. So like we wrote a prompt, they wrote a prompt.
So instead of using our prompt, which we have to use their
prompt because their model was specifically fine tuned on their prompt.
So to use their prompt we need to pass in a list of documents. So here their documents
have to be title and text. Again, this is an example, a toy example from their website.
And then we can use their tokenizer to translate the documents
and conversation to into their specific prompt, which here it is.
I wrote none of this.
This is all the prompt that Cohere researchers wrote for their model.
So all of this about instructions override the task description, style guide.
I didn't write any of this. This is all Cohere.
And to make their models work at their most performant, I have to
follow the prompts that they wrote because that's what their model is expecting.
Now technically we can change this. We don't have to use this verbatim.
But just for the sake of doing what they did, I am going to use the prompt that they wrote.
So we have their system preamble, user preamble, style guide, all the stuff that
Cohere wrote. And then finally we can start to see document zero, document one.
Then comes their back to their prompt.
Firstly decide which of the retrieved documents are relevant.
Sounds like chain of thought to me.
Followed by writing relevant documents. I see some prefix notation followed by a comma separated
list of document numbers. More chain of thought. If none are relevant, instead write none.
So we are allowing the AI to literally write out, is this sufficient or not like we did.
Secondly, decide which of the retrieved documents contain facts that should be cited. This is the interesting part.
Their model and their prompt specifically tells the model,
don't just use the information, but cite the information.
And then thirdly, write answer prefix notation followed by a response to the user's last input in a high quality
natural English, which I'm sure you could change. I don't know how well it would do. But again, their prompt.
So my point is, I wrote none of this, but we
can already see how it kind of looks like our prompt.
The only difference is they didn't include any few shot examples.
You can argue that by writing this format here, it's kind of like a few shot example.
But really the only difference here is they don't have any few shot examples, which is okay.
So if we were to actually do this, we're going to apply
the generation template. So now we have our actual prompt here.
And then I'm going to actually pass it through the model. Now, heads up what you're looking at here.
Locally, I had to run this with a gpu, which my little laptop here does not have a good
enough GPU or really any GPU to be able to run this model. So the code you're looking at
was technically run in colab, but I just ported it over locally so we could see it here.
But I grabbed the model just like we grabbed models in the
past. I have our tokenizer, our auto model for causal language modeling.
So our causal language model, that is our autoregressive language model.
I will generate meaning, pass in the tokens,
the generated tokens which are coming from our.
Right here, our token here, and then we'll decode it. I don't need to
reinstantiate the tokenizer here, but whatever, I run it through our model with parameters,
temperature for determinism, do sample, whatever I want. We've seen these before.
I will decode it from our tokenizer and then I will look at it and say, well, here's everything that they wrote and starting.
Well, let me see where it ended here, just to make sure.
Chatbot token.
Here we go.
Here, this is the.
What comes after. This is the generated part.
Relevant documents 0 and 1, apparently. So both documents we inputted.
It's going to cite document zero. Here is the answer. And then this little thing
here, this little tag here is how it's citing the information. So where it says
CO0 that 0 refers to which document it's citing, which is pretty cool.
Now, to make this more relevant, to us.
Let's take our Social Security Medicare information from the FAQ from the government.
Let's turn it into the same structure that the Cohere AI is expected.
So to make it kind of fit a little bit better, I will given
a query I lost my card. I will grab the documents from Pinecone.
I will pass those documents which we've already seen. Only the first one is relevant.
I will pass them into the same exact format. So here's my conversation, here's my
documents that I have formatted to be in the same style so title and text.
I will pass them through our tokenizer, pass them through our model and then let's read what happened. So here's
our at the top is going to be our preamble. We didn't write that our documents at the bottom here.
Let's find it Here it is.
Relevant documents zero. Correct. Only the first one Cited Documents zero. If
you've lost your Medicare card, you can request a citing replacement online.
You can also citing access an official copy from your online Medicare account or citing
again call Medicare 1-800-Medicare to have a cited replacement mailed to you. So this is
pretty neat. I have to say I sound pretty pumped up, because I am.
It's not just about writing a prompt, shoving it into a really good autoregressive AI.
Some companies out there like Cohere, are taking steps to actually push that field a
little bit forward by saying, well, let's make a model specifically for this application.
And at the end of the day if I, Sanan Ozdemir, wanted to make something like Cohere's
Command R all in theory, quote unquote simply would have to do is grab a bunch of
training data which I could find on Hugging face, put it into a format like Cohere has
with this prompt, and then spend a lot of money fine tuning an autoregressive LLM which as
we'll see in future lessons is going to be, can be at least quite cost prohibitive. But
luckily Command R is not only available open source like I'm using here, but it's also available
through an API through the Coheres API. So it is really worth looking at these other competitors
to OpenAI to see what they're doing to help out with real practical use cases.
So I'll leave it there for now.
Happy retrieval Augmented generating.