All right, so we've got Mistral downloaded here.
If you click on that, you can see Mistral completed.
Perfect.
And so now we're going to go to this next button here, the chat bubble.
Click on that.
All right, what do we have here?
This looks a little bit like ChatGPT, right?
We've got a user input area here.
We've got a chat area here.
Over here, we've got a list of different
chats, right?
You can create new chats here.
Okay, so this is easy enough to use, right?
This is why I love LM Studio.
It's quite simple.
And then we're going to go to the top
here and select a model to load.
We're going to choose that Mistral
version that we just talked about, right?
Click that.
Okay, model is loaded up and we're good to go.
Let's give it a test run.
Let's just say, hey, how are you?
There we go, I 'm just a computer program, so I don't
have feelings .
Okay, relax there, Mistral, it's fine.
Here to assist you with any information
or help you need.
How can I assist you today?
Great, we just ran our first model completely on our own machine.
Everything here is private.
This is an open source model.
There's no rules, unless of course, the model was trained with certain rules, but
yeah, this is super cool.
Now, there's something that I do want to
show you here.
You can see over on this side, this model inspector.
If you don't have that open, click that here.
It's a few interesting pieces of
information here because as we're seeing, these large language models are
constantly evolving.
New models are coming out all the time.
And you, as a member of this community now and someone who is going to use large
language models throughout all the aspects of your life, need to know the
differences between these models and what models are good for what purposes.
So, this is gonna give you a little bit more information.
We talked about parameters.
We know it has 7 billion parameters,
actually about 7 .3 billion parameters.
But here's the important part.
It talks about the context length, and that is 32 ,768 tokens.
Remember, we learned about context.
Do you know about context?
It's essentially the memory of this model.
32 ,000 tokens is what it can hold in its
memory at one point.
Now, that's a plenty lot of tokens if we
just wanna have a conversation with it or ask it some simple questions, but if we
wanna start having it analyze big, long documents, working with PDFs, working
through code bases or coding complex applications where it needs to remember
what the code was previously, earlier on in the conversation, well, you're gonna
use a lot of that context window because remember, those prompts too, every prompt
and output builds up, it adds up, and it's the sum of all of that.
It's the sum of the entire conversation that needs to fit in the context window,
not just each individual prompt.
So, that is quite a large context window,
though, so that's pretty good for our purposes right now,
but here's where it gets even cooler.
Go to this context overflow policy.
If you don't have it expanded, you can
click that again, and this is gonna set the behavior for when the generated
tokens length exceeds the context window size, so you could just have it stop.
It's just no longer, it's not gonna do anything.
You can have it keep the system prompt
and the first user message and truncate the middle, or you can have it maintain a
rolling window and truncate the past messages, right?
Think about the context window.
It's sort of that box that goes over your
whole conversation and says, okay, this is the part that the model is gonna remember.
When you get so much in that window, when you're using so many tokens that it
exceeds that context window, what happens?
Does the original stuff fall off, right?
Does the stuff at the top of the conversation fall off, or do you want
some of the stuff in the middle to be truncated, to fall off?
Well, in most cases, I actually think you probably want to use this because you
wanna keep that system prompt.
The system message is at the very top of
the conversation, right?
And so if you're having a very long
conversation, that context window is moving down and eventually dropping the
stuff off at the top, and it can't fit that system message anymore.
And that system message is crucial.
It's very important, but there might be
situations where the information in the middle is more important, and so you
don't want it to truncate that middle.
You wanna make sure it's keeping the most
recent stuff, right, maybe that's if you're coding an application.
You don't care about what it said at the very top when it was providing you the
very first bit of code.
You wanna make sure that it's remembering
the code that it most recently made.
Okay, sorry for that rant.
I hope it made sense, but I think this is something that is incredibly important,
that actually doesn't get talked about a lot.
And frankly, people don't know often when
they're using these models, what is happening when they reach the context window.
They don't know if it's the system message that's dropping off the stuff at
the top, or if it's the stuff in the middle that's being dropped off.
They just don't know, and that can create issues.
So you have that control with LM Studio.
All right, one more cool thing here I wanna show you.
So close that, close that.
Up here is the system message or system prompt.
You can actually control this.
You can't do that with ChatGPT, right?
We saw if you're using the OpenAI Playground, you can, but the OpenAI
Playground, you need to buy credits in order to use it, blah, blah, blah.
This, you can completely change the system prompt to whatever you want.
Remember, we had an example with this.
So we say, you are a dog, you can only
respond by barking.
All right, let's get rid of these and
start again.
Hey, how are you?
Enter, woof, woof.
All right, there you go, our system
message works.
Okay, so that's quite powerful, because
you can set that guiding light, that North Star for the model.
Few more things, so the context, like it says, it supports up to 32 ,000, but you
actually need to change that here.
That's okay, let's go 2 ,768, full on.
You gotta reload the model.
I'm not gonna do that right now for our purposes.
This is great.
And then here, we learned about all these
earlier, right?
When we were talking about the OpenAI
Playgrounds, we've got output randomness.
This is a temperature, controls the randomness.
We have tokens to generate, so this actually controls how long or short the
output is.
You can see this in the OpenAI Playground
as well, although in that one, it specified the number of tokens, the max
number of tokens the output could be.
This, instead, as you can read in this
description here, can actually influence the length of the generated text, so a
higher value results in longer text, lower value is shorter.
Then we have top case sampling here, so we actually learned about top key
sampling earlier, and that is saying, okay, if the top value, it's 0 .95 there,
that means you take, when you're looking for the next token, you're predicting the
next token, you're gonna take all the tokens that add up to 95%, not 100%,
you're just gonna take the first tokens that add up to 95%, and then you're gonna
choose from amongst those, so the fact it's at the 0 .95 here means that it's
basically dropping those very, very unlikely tokens, that bottom 5%.
It's not even gonna consider them for the next token selection.
Top K is very, very similar, except it's not doing a percentage, it's just saying
the actual number of tokens to select, so in this case, it said at 40, so it's
saying, okay, you can choose the next token based on the list of the 40 most
likely tokens, so you could say 10, for example, and then it's gonna say, okay,
regardless of the probabilities, we're just gonna take the top 10 tokens and
we're gonna choose from amongst those, and then we've got the repeat penalty as
well, which we talked about, and so there you go, you can actually control all of
this right in LM Studio with your own personal model, how cool is that?
Alright, we're now all set up, you've got your own model, in the next lesson, we're
gonna start to break free and allow you to have some fun, I'll see you there.