- [Instructor] Running an LLM from Open Source
is one of our first tasks.
There are many ways to take on this task,
but we'll assume that you have limited
computational resources,
and that you'll need to rely on optimizations
that help get things up and running on a CPU.
Though everything we cover
will also function well for GPU use as well.
We'll also assume that you don't intend
to train these models,
and that you only intend to perform inference.
Training is a very resource-intensive process
where you use data to update
the weights in your model.
And inference
is a less resource-intensive process
where you call your model,
and have it generate tokens
to complete a sentence
or respond to a question,
which is what over 99% of people
using LLMs are doing.
Like when you visit your favorite website
that has a chat bot,
we'll be focusing on the use
of an implementation of transformers code
that was put together
by the Open Source community
that's called Ollama.CPP.
Their goal was to enable those
who wanna run these models
off their laptops to do so,
and we'll be doing
all of our work within a Linux machine
by spinning up Ubuntu on GitHub code spaces.
and would utilize Ollama,
which helps automate the process
of installing different models,
and running Ollama.CPP
without having to get into
the nitty gritty of the code.
The final thing that I'll mention
is that we'll also be using quantization
of the original models
in order to help our inference run faster.
That just means that instead of
using the original weights in our model,
we'll use a truncated version of those weights.
So for example, instead of storing
a weight value
of this very long number here,
we would store a weight value
of this much shorter number,
cutting the amount of RAM required
to store the weights in half.
This makes running everything much more efficient
with a very small loss in performance.
Finally, to wrap everything up,
we'll be looking into a few model parameters,
which can be tuned,
and we'll discuss how they impact inference.