Welcome back.
Now that we have covered all the fundamentals, we are finally ready to start our first, most basic
form of training large language models.
For this, we're going to need access to a GPU enabled machine.
For this lesson, I've prepared an RTX 4090 with 24GB of video memory.
Let's start with a quick setup to ensure we have the required dependencies that we'll need for this
project.
We'll be using Python 3.1 for this lesson, and we'll be working on ubuntu.
If you can, I'd recommend following along on a Linux machine as well.
Now, to make things easier for us, we are going to install Miniconda and use it to manage our Python
environments.
Let's just search for Miniconda or follow the link in the resources of this lesson.
Here we'll just need to select the installer for our platform.
In this case we are using Linux on an x86 platform.
Okay great.
Let's just copy this link address and then we can download it on our remote machine.
Here we'll just need to wget the link and the installer will start downloading.
If you are not working remotely and you're doing this on your own local machine, then you don't need
to specifically do it like this.
All right, now let's just run the script with bash and install Miniconda on our machine.
Okay.
Awesome.
Let's start a new shell.
And here we can see that the installation was successful.
We have the base environment already on.
If we check this comes pre-installed with Python 3.12.
This version of Python is rather new, and not all the libraries that we are going to use are supporting
it very well.
As of right now, there are quite a few viable ways to go about training a large language model.
Different solutions and libraries that you can use.
For this course we are going to use a freely accessible open source library that is both very powerful
and flexible, while also keeping things simple and easy to use.
This library is called axolotl and we can find it on GitHub.
As this description says.
Axolotl is a tool that makes it easy to train llms.
It supports a wide variety of models and technologies, and all we need in order to run it is Python
3.1.
So what we're going to do is create another environment for our training.
But first we're going to follow the quickstart instructions and clone the axolotl repository on our
local machine.
Okay great.
Then we're going to create a separate conda environment.
I'm going to call it train and specify that we want it to use Python 3.1.
Okay let's activate it.
Now with our environment on, let's install axolotl.
We can see that it also requires a certain version of torch to be pre-installed.
So let's also install torch and make sure we have everything ready for axolotl.
We'll go with the latest version of torch here.
This installation is going to take a while, so let's just wait it out until it's finished.
Okay, torch was installed.
Now let's finally install axolotl.
This might also take quite a while, so let's just wait it out until it's all finished.
All right.
And with this, our environment is done.
We are ready to start our first train.
Okay, we'll pause here for now and resume everything in our next lesson.
Thank you.
And I'll be seeing you there.