Hello guys.
So we are going to continue the discussion with respect to Long Chain.
Uh, in this video we are going to create a very simple application, uh, LM application or generative
AI application using Lang chain.
And uh, we will also be understanding what all tools we will be specifically requiring.
So what kind of task or what kind of application we are just going to generate over here is that let's
say I have a specific website and that website has some content.
Let's consider that it has some kind of text format, some information over there.
Now what we are going to do is that we are going to extract that entire information.
Uh, again, for that, we definitely require some data ingestion technique.
Uh, we will be seeing in long chain.
There are multiple data ingestion techniques such as web based loader.
You have uh pie PDF directory reader, you know, and you can also read PDF directly.
There are lot of libraries that are available in long chain, which we will be looking at it one by
one.
But uh, in this video, what we will do is that I will be considering a website, and from that particular
website we will try to take up the content or will try to scrap the entire content after scrapping that
specific content.
What we are going to do is that we are going to divide that entire content, you know, or entire text
into chunks, and then we will try to convert that into vectors by using some vector embedding techniques.
And after doing that we will be using LM along with prompt engineering uh, to specifically get uh output
from that particular page.
So there are lot many things that we are specifically going to discuss.
Uh, so let me just go ahead and share my screen.
Uh, so here you can see, uh, I have my this Lang Smith page.
So what I'm actually going to do, I'll go down over here and there are some documents.
Okay.
So this documents, uh, we will try to read one of this documents and let's see whether we will be
able to scrap this entire thing.
So let's say this is my tutorial over here.
Uh, let's pick up a page where I will be able to see some information.
So here let's consider this is my page.
So I'm going to probably take this entire page.
And I'm going to read the content from this particular page.
And with the help of LLM and Prompt Engineering, if I ask any question to my uh, Jenny AI app, it
should be able to give the answers.
Okay, so that is what I am planning to do it.
So let's go ahead.
And here you can see in getting started we have learned about all these things.
Now I'll go ahead and create my next file.
And we'll be doing the coding over here.
So let's go ahead and select the kernel.
Uh as usual as I said that we are going to generate a simple, simple JNI app using Lang chain okay.
And here again we are going to use OpenAI initially.
And then we'll also go ahead with uh some open source libraries okay.
That also will be seen okay.
So let's go ahead and execute this.
Uh here I'm going to just go ahead and create some cells.
Now the first thing uh, which we have already done in getting started is and this is the most important
thing, specifically loading all the keys.
Right.
So I will just go ahead and load all the specific keys.
So here you can see from dot e and V load underscore dot.
Then I need my open API key along with that lang chain API key lang chain project.
So this will be specifically common.
Now as I have already told that, uh, what I am actually going to do is that if I want to retrieve
the information from that particular website, uh, I have to first of all, read the entire content
from the website.
Right?
That is what I really have to do.
So what I am actually going to do over here is that, uh, we will first of all, go ahead and import
one important library, which is called as beautiful soup.
Okay.
Beautiful soup four okay.
Now why this library will be used because this library will help us to scrap the entire website data.
Okay.
And as you all know right now, we are going to use this particular website data.
So we'll be taking this URL in order to scrap this entire text and retrieve it at our site.
We will be specifically using this Beautifulsoup for library.
Okay.
Now, uh, after doing that, uh, first of all, let's go ahead and install it.
So I've already done I've written it in the requirement dot txt.
I will go and open my terminal.
And here I'm just going to write pip install minus our requirement dot txt.
So once I do the installation here, you'll be able to see that my installation will take place uh in
the same V and V environment okay.
Now once this is done I will close this.
Uh, let's quickly use one data ingestion technique.
Now, first of all, we'll focus on data ingestion.
Data ingestion basically means from the website, right from the website.
We need to we need to scrap the data.
Right.
Scrap or scrape the data right?
I'll not say scrap it, scrape the data.
Okay.
So that is what we really want to do.
Okay.
Now from, uh, Lang Chin usually provides you an amazing module, uh, which is specifically available
in Lang Chin underscore community.
So I will go ahead and import from Lang Chin.
Uh, Lang Chin underscore community.
Okay.
So here, uh, what we are basically going to do is that uh again for this we require lang chain underscore
community library.
So let me just go ahead and import it over here again.
So here you have this Lang chain underscore community.
Let me open my terminal and let me quickly install this.
So because I will be requiring it okay.
Lang chain underscore community has lot of document loaders right.
Document loaders basically means you will be able to load the entire data from different different sources.
That is the reason I am installing this particular library, which is called as lang chain underscore
community.
We'll be discussing more about it as we go ahead.
Okay, so now the installation has been taking place.
Uh, what I'm actually going to do, I'm going to go back to my code.
So here now let's see if I go ahead and see this lang chain underscore community okay.
And here we are going to basically use document underscore loaders.
Okay.
And here I'm going to import web base loader.
So web base loader is nothing.
But you give the website link and automatically it will be able to load it.
I'll get an error I guess.
So here you can see okay this is a okay.
Let's see whether I'll get an error.
So user agent environment not set.
Consider setting it to identify your request okay.
So right now it is some kind of error.
User agent environment variable not set user agent.
So I have selected the virtual environment over here.
Let's see what is the issue over here okay.
So this has got executed now.
Now what I am actually going to do over here is that I'll go ahead and use a loader and I'll say, hey,
use this web based loader, okay.
Web based loader.
And inside this I will give my link.
Right.
So link will be nothing.
But let's go ahead and see the link which it is.
So I will go to quickly.
I will be probably giving this entire website link because this is where I really need to scrap my entire
data, right.
So I will just go ahead and give my link over here.
Okay.
So this will be my Https document docs dot smith dot lang chain dot tutorials.
Uh slash manage spend.
Okay, now after I load this particular data you can probably go ahead and just print this loader.
So here you'll be able to see hey it is a document loader web based loader.
Right now web based loader is something it is a kind of library.
Whatever link you specifically get or give right a website link, it will be able to scrap the entire,
um, scrape the entire content from that particular website.
Now, uh, the prerequisite is that you really need to have this beautiful soup library because internally
web based loader uses Beautifulsoup for.
Okay, now, once I get this entire, uh, documents.
Right.
What I really need to do is that I will load this document.
So here, if I go ahead and write loader dot load.
Right.
So in short, I'm going to get all my documents.
If I go ahead and execute this.
And let's go ahead and print my docs.
So you'll be able to see all the documents inside this, all the page content I'll be able to see,
see page content, uh, hosting tutorial every the entire scraped, uh, scraped data from the website,
uh, which we have actually exported.
We have actually got it over here.
Okay.
Very much important now very much important thing.
One more step after this, what we do.
And usually in every Rag application we usually performs this thing right.
First of all, we read our entire data source from a specific data source itself.
We'll read our entire data from a specific data source.
Then we will load this and convert into our documents once we get that particular document.
Now this is a huge document.
Right now in this particular case, if it is a huge document okay, we need to perform, you know,
we need to probably divide this entire document into chunks.
You know, we cannot directly give this entire document to our LLM model.
The reason is very simple.
With respect to every LLM model there is some context size.
There is a limitation with respect to context size.
And as we go ahead and see different different LLM models that are going to come up, the context size
will keep on increasing.
But it is always a good idea that we divide this entire documents into chunks of text, right?
Or whatever text is basically present inside this document, will try to divide that into chunks.
Okay.
So, uh, that is the next step that we really need to do.
And after we divide all our data into chunks, see I'm just going to write it down over here.
Initially we load the our data, then we uh, get all the docs okay.
By just executing loader dot load.
Then after that we divide our text into chunks.
Okay.
Into chunks.
Chunks basically means smaller sized data.
Right.
And it is very simple.
Why do we do that?
Because every LM model has a restriction with respect to context size.
Right?
Then after we divide a chunk text into chunks, then the next step, what we are specifically going
to do is that convert this into vectors by using some kind of vector embedding.
Okay.
Now what is vector embedding?
Vector embedding basically means these are some, uh, techniques wherein we'll be able to convert all
this text into vectors.
Okay.
And again we have to use some different different vector embeddings.
There are a lot of different different vector embeddings.
Uh, some of the vector embeddings I will be using is open AI.
And we'll also see some, uh, open source vector embedding techniques.
And after you probably get the vector embeddings, uh, all the vectors, we will try to store it in
a vector store DV okay, so this are my steps that we are going to follow in this entire tutorial.
Okay, so I hope you got an idea with respect to this.
Uh, here we are going to load the data, divide our text into chunks text vectors vector embedding
and vector uh vector store db.
Okay.
Finally, uh let's go ahead and do this.
So first of all, what I'm actually going to do is that, uh, I will go ahead and use from lang chain
underscore.
There's something called as text splitters.
Since we really need to split our entire documents into chunks of text.
Right.
So I'm going to use this text splitter and I'm going to import something called as recursive character
text splitter.
Okay.
Now this recursive character text splitter is a function which will be helping us to entire split our
documents.
Right?
So let's go ahead and initialize this recursive character text splitter here.
There are some parameters.
First is your let's see.
So okay I'll go ahead and initialize this recursive character text splitter.
So let me just go ahead and define this.
This will basically be my text splitter over here which is nothing but our recursive character text
splitter.
Uh, which we are specifically going to use.
Okay.
Now why we are using this, is that because with the help of recursive character text splitter, uh,
we will be simply, you know, we will easily be able to convert our entire documents, you know, into,
uh, chunks of documents, in short.
Right.
So we will be able to do that, right?
Uh, so chunks of text, I can basically say now, inside this, you also have an option to specify
your chunk size.
Let's say I go ahead and specify my chunk size is 1000.
And I can also overlap like the my text while we are doing this uh, splitting.
Right.
It can also overlap.
And the number of characters that can overlap is 200.
Okay.
So this is what I'm going to specifically use with respect to recursive character text splitter okay.
Now coming to the next step after we do the text splitting Right.
Uh, what we really need to do is that we will use this text splitter, and then I will split all my
documents.
So I'll go ahead and split my, all my documents.
And obviously this docs that I'm actually going to give will be available inside this right as a parameter.
So once I do this I will be able to get all my chunk of documents right over here.
Perfect.
Uh, till here you have actually done each and everything, right?
So if I go ahead and execute it, and if I go ahead and show you.
So here will be my documents.
And this is how it has been divided.
Right.
Based on the chunk size and the overlap.
Initially I just had one document.
Now I have multiple documents.
And we really need to do this because every LLM has some limitation with respect to context size.
Now once you have done this, the next step will be that converting all this text into vectors.
Okay.
The reason we convert that into vectors.
Because, uh, whenever we work with Q&A, chatbot or document Q&A, chat bot or uh, let's consider
rag application.
You know, there are similar a very simple algorithm is basically used which is called as cosine similarity.
And based on the cosine similarity usually gets applied in the specific vectors itself.
Right.
So that is the reason what we really need to do is that, uh, in order to convert this into vectors,
first of all, I will go ahead and use some embedding techniques.
Now in this case, since I'm using open AI, I will go ahead and write hey, uh, train underscore open
AI, I'm going to import something called as open AI embeddings.
Okay.
So this is the embedding technique.
And this is a very efficient embedding technique wherein it will take the text and it will convert that
into a vector.
So here I will go ahead and initialize.
This embedding is equal to OpenAI embeddings.
Okay.
So once we have done this.
So in short this is my embedding technique which will convert our text into vectors.
So I have executed this okay.
Now the next thing that I'm actually going to do is that I will be also installing one library because
with the help of embeddings we will be converting the text into vectors.
Right.
But where we will be storing that particular vectors, we have to store these vectors in some kind of
vector database.
Right.
And in this example I'm going to take a simple example of this vector database for this vector database
is uh, you know, it has been created by Facebook and uh, you can use it very much efficiently.
And there also similarity search is applied in the back end.
So first of all I will go ahead and import my files dash CPU okay.
So since I it's okay.
You can also have the GPU version.
But right now I think uh CPU will be more than sufficient.
So I'm going to go ahead and install this and let me go ahead and open my terminal and quickly install
all the requirement dot txt okay.
So now you'll be able to see that my CPU has been installed.
Okay.
Now, once I do this, the next step will be that, uh, I will just quickly go ahead and import from
long chain underscore community dot vector stores the different different vector stores like fires,
chroma DB and many more.
We will be discussing about them.
Uh, later on when we do projects we'll be getting to know different different vector stores.
There is also like a one vector store database which is called as object box will also be seeing some
example from there.
Now I'm going to import files okay.
After importing files, uh, you'll be able to see now with the help of this embedding that I have,
right, OpenAI embedding, I will be converting all my documents into document text into vectors.
And then all those text, all those vectors will be stored in this particular, uh, vector database.
That is nothing but files.
Now, here, uh, you can actually see that I have converted all my things into documents.
Now, let me just go ahead and create my vector store.
DB okay.
Now this vector store DB here I'm going to specifically use for this.
And then I will be using a function which is called as from underscore documents.
And here I am actually going to give my documents comma embeddings okay.
So documents comma embedding basically means all the documents which I am actually going to do.
And all these documents are which embedding I really want to apply.
That is nothing but OpenAI embedding.
So I have given it over here.
Right.
And finally you will be able to see that I will be having this vector stored database.
Okay.
So once I go ahead and execute it, you will be able to see that, uh, it will get successfully executed.
And now I will be having my entire vector stored DB okay.
So here is my entire vector store DB right.
And now from this particular vector store db, I will be also.
You will also be able to so save this in your local environment or in any hard disk wherever you want.
Right.
And now from this particular vector store DB I'm going to query and based on the query I can ask anything.
It will be able to provide me the response.
Okay.
And over here I have discussed about this files.
Uh I've store I've discussed about vector store DB and many more things.
Right.
So um, yes, uh, right now in this video I will stop it till here of creating this vector store.
DB.
Now in the upcoming videos I will be talking about more amazing things.
Uh, wherein uh, we will be discussing how we can retrieve queries from this particular vector store
DB and get all the content based on the similarity search.
Right.
So yes, uh, this was it from my side.
I hope you like this particular video.
I will see you all in the next video.
Thank you.
Take.