All right.
Welcome back.
Now we know how to use the license model as well.
But for this lesson we won't use a model that requires a license.
Instead, we'll use a smaller and easier to work with llama variant called Tiny Llama.
Ideally, you would run this on a GPU, but if you don't have one at hand, it's okay.
It's not mandatory for this lesson.
The code will still work.
It will just be a bit slower.
All right, so let's load our model and send it to our device.
It's going to take a moment to load.
But in the meantime we can load our tokenizer.
Okay.
Let's use the same messages we tested with previously.
Next we'll prepare them for our model.
So using the tokenizer will apply the chat template to them to make the format just right, and then
transform them into tokens.
We need to make sure we add the generation prompt parameter at the end, so that will tell the model
we want it to talk next.
Okay, let's move these and the model to the same device and look at what we have.
Unsurprisingly, they are still only token IDs.
It's nothing different from the first part of the course.
Now we are simply structuring them in a smarter way.
Great.
Let's pass them through the model and get some generations.
Now.
Next, to make sense of them, let's decode them back into readable text
All right.
Looks like the model completely hallucinated an answer, but at least it's a coherent one.
Okay, let's try to understand more about how this all works.
We'll remove the last two messages, keeping only the initial query.
Let's see how the model responds only to that.
Okay, it is still hallucinating, but the format works.
We took only the first query and the model came up with its own contextually relevant answer.
That's great.
Let's look at some of the most important parameters that we have in order to control this generation
Firstly, we want to control how much the model is able to generate, or the maximum number of new tokens
it can add in this generation.
It's very important to understand that these tokens, the newly generated ones, count on top of the
input or the prompt tokens that come from the previous messages in the conversation.
So if we look at its size, we can see we already have 75 tokens used up.
Let's do a generation where we only allow the model to give us only ten extra tokens, up to 85 total
tokens.
Okay, we can see that its output gets truncated.
We basically stop him after the first ten generated tokens.
But this is an important thing to have control over.
Remember that the sequence length of a large language model gets divided between both the prompt and
the generated tokens.
Bislama version has a maximum context size of 2048 tokens.
So our total input plus the generated output needs to fit inside that window.
We can see that even capped at 128 tokens, the model is still getting its output truncated, and we
have a total size of just over 200 tokens.
Okay, but this is still inside the range.
Let's say we had a longer context to begin with.
What would happen then?
Let's artificially make this message longer and see what happens.
Okay.
So right from the tokenizer we get a warning that the input is already longer than the maximum size
of the model.
Not only want the model be able to see and process all the input, he would not even have a place to
give an output in.
So basically strange things would happen.
But let's test it either way.
Okay.
Yeah, well, it didn't crash, but nothing useful came out of it.
All right.
Let's see what would happen if we were to be right on the edge of the size limit, not over it, but
close enough so that the prompt fits.
But the answer might not.
Okay, so in this case, the output will simply be truncated.
The model will generate as much as it can within the size constraints.
Okay.
That's a nice and interesting behavior.
Now, if we run this generation a few times, we can see that each time the results are the same.
So by default, our model is deterministic.
The same inputs will result in the same outputs.
But maybe we don't want this all of the time.
And we want our model to be able to give more results for the same query.
This behavior is controlled by the sample argument.
When this is on we basically allow a form of randomness inside our generation logic, and we can see
that by running it now it gives us a different result.
In fact, running it a few times gives us a lot of varied results each run.
Now, in order to control this variation, we can change the temperature parameter This controls how
random the responses can be.
A higher temperature will allow the model to use a wider range of possibilities when generating a response,
thus resulting in more creative outputs.
We can see that with a temperature set at one, the responses are very different and varied.
Let's lower it down now to 0.1 and see the differences.
With this configuration, the model seems to be basically deterministic for these input messages.
In practice, you will probably either disable sampling or set the temperature somewhere in between
in order to have the best of the two worlds.
All right.
The last parameter we will look at is top p which also controls the randomness of the output.
Working together with the temperature parameter.
While the temperature adjusts the overall shape of the probability distribution for the next word prediction
in order to allow for more possibilities.
Top P tells us which subset of that distribution to use, essentially enabling a cutoff.
A top P of 0.1, for instance, will only allow us to use the first tokens that together make up 10%
of the total probability volume, meaning the model can use only the words that it is really sure about.
Okay.
And we can see that even with sampling enabled and with the highest possible temperature, we still
get deterministic responses from our model with such a low top p probability.
We usually want our top p pretty high in order to allow the model to explore more possible answers and
not restrict him too much on the words he is allowed to use.
But we can't have it too high.
Lowering it up a bit will allow us to avoid rare but very weird responses from the model.
All right.
And with this, we have looked over all the basic generation parameters that we will need to know for
now.
If you feel like you need more advanced inference and deployment logic and see how to use these large
language models effectively in the real world, check out my course on that link in the resources of
this lesson.
But for now, our next step is learning how to train and fine tune these models for our use cases.
We look at this and more in the next section of this course.
I want to thank you for coming this far already.
I'm happy to have you here.
We will resume our work in the next lesson, and the next section of this course.