In your journey of machine learning, data science or AI?
You might have gone through multiple research papers, research papers on classification, on regression,
time series analysis, on blah blah, blah.
Multiple topics.
You know what the most important and crucial thing going through a research paper is?
Patience.
Many people lose patience while going through the research papers in depth because research papers usually
have equations, complex wordings, which is kind of difficult for people to understand.
Not everyone can easily understand from the research papers.
Can we leverage the generative AI concept or the large language model concept to create a portfolio
project, which can actually make my task easier?
I am thinking like a chatbot, which can take my research paper as an input, and I can ask something
in plain English or easy terms that what this research paper is all about.
What are the main components of this research paper, and the bot should be able to respond me with
a layman's term.
Is it possible it is 100% possible.
And in this video we are going to talk about this amazing portfolio project based out of large language
models and rags concept.
And we are going to take it ahead and build a chatbot using a custom data set, which is a research
paper.
And our final chatbot should be able to ask, should be able to answer my questions in a very, very
easy and effective way.
Hi.
Welcome to this practical session on rags.
So in this particular video we are going to focus on a particular use case of using a custom data.
And then based on the custom data we will be calling OpenAI APIs.
So before getting into the custom data, I would also like to show you this part.
What is this part?
This part is very simple where you are passing your entire input, your prompt, and then based on your
prompt, you are calling a OpenAI ChatGPT model could be GPT 3.5 or GPT four.
And then you are getting a response exactly the same fashion how you interact with ChatGPT or Poe or
You.com, right?
Let's say you're asking, what is EDA?
Now this basically goes as an input, and we are calling the GPT 3.5 turbo model.
And the turbo model is responding as this way.
Right.
So let's try to get into that.
Now.
Before getting into that, there are a few things to note.
We will be installing few libraries.
First of all we will be installing OpenAI.
Then we will be installing Lang, and then we will be installing Lang Chain Community.
And then we will be installing five CPU.
Now I will explain each of them.
So do not worry about it.
And on top of that, let me use a code for this.
On top of that.
Um, I will also be sending you the requirements.txt so that you can pip install the requirements.txt
directly so that all the models will be installed.
Uh, installing libraries.
So let's start one by one.
We'll first get started with pip install open AI.
In my case it says requirement already satisfied because I have already installed it.
In your case it will be installing it from scratch right.
So you can run these four pip installs one by one.
I'm not running it because in my case it's already installed.
The next thing is importing libraries.
Okay.
So first of all we will be importing open AI.
Then from lantern document loaders we will import py pdf loader each and every method.
Each and every library, each and every function will be explained, so do not worry about it.
From lang chain dot embeddings dot OpenAI import OpenAI embeddings okay.
And then from lang chain dot text splitter import recursive um character text splitter.
Okay.
We'll be importing all of them.
So everything is successful because all our modules are already installed.
Now the next thing is, as we will be calling OpenAI model, remember this thing.
In this case, we are going to call the open AI model, right.
Open AI model is nothing but your ChatGPT, right?
OpenAI has already provided APIs.
You can directly call them and talk to the LLM.
But in order to access this API as a user, when you are accessing open AI's API.
You need a key because it's not directly accessible, you need a key.
Now how to create a OpenAI key?
Well, unfortunately I cannot show you because I'll have to again log in to my VPN and then open.
Go to OpenAI website and create a key.
Um, because in Hong Kong, open AI is not accessible.
So I will leave that up to you because there are tons of videos on YouTube on various other platforms
where you can leverage those videos to create a OpenAI public key, right?
Nowadays, OpenAI has made it very strict.
Initial $5, you will get it for free.
But beyond that, if you want to do some more tests, you will have to recharge your OpenAI bucket,
maybe with $5 or $10 and then try it out.
Do not.
While you are testing all these things because you are in your initial phase of your journey.
My recommendation will be stick to 3.5 turbo, because if you do GPT four, the cost will be high because
GPT four is at least 100 times more costlier than GPT 3.5, GPT 3.5 is around $0.01 for 1 million input
tokens.
However, GPT four is around ₹37.5, or around $0.4.
And again, GPT four oh.
I'm not sure it would be even more expensive than that.
So in my case, I'm using my, um, OpenAI keys.
So I'm setting up, uh, a parameter.
Now I'm going to use my key.
Uh, but by the time you will receive this video, this key will not be available to you, because this
is a private key that I'm using, and I don't want to make it publicly available because unnecessarily,
my key will be used by many people.
And I'll have to pay that.
So create your own key.
Okay, so I'm creating my own key going forward.
I'm just hiding that code because I don't want to show the key to a lot of people.
Next thing is I'm directly calling the from long chain dot Llms import open AI.
That's it.
Okay, done.
And then I will be calling LM is equals to open I.
And here I will be passing.
Open.
Open I.
API key is equals to.
I will be passing my original variable.
Open I key.
That's it.
Now what's happening is you have initiated your open AI large language model.
Now you have to just ask a question the way you ask to ChatGPT.
What is editor.
Now let's say we will be asking this thing.
What is EDA?
Uh, explain EDA in just two lines.
This is the question I'm asking.
Simple.
And I'm getting an answer.
Now, here we will doing.
We'll be doing the same thing.
I'll be calling LM dot invoke, and I will ask this question.
Let's see what the answer is and we'll try to compare the answer.
Okay.
So that's great.
Let me try out with another API key.
So I have created another API key.
I will unfortunately not be able to show that open API key, because that's an active API key that I'm
using for all my proof of concepts.
Let me run this and here you will get an output.
Um, exploratory data analysis is the process of blah blah blah.
Let's try to compare this output.
Okay.
Um, let me just break down it into few lines.
Let's say like this.
And data visualization like this.
I'll go to my Po and I'll try to get this answer as well.
And we'll try to compare the outputs.
See it will not be a replica, but eventually it will be a very similar answer in both the cases.
So in the in the first case in my Python code, it is responding.
edX is the process of examining and visualizing data to gain insights and understand the underlying
patterns and relationships.
It involves statistical methods and data visualization.
And in this case, it is telling you it is the process of examining and visualizing data.
You can see it's almost correct till here.
To gain insights, but here to uncover patterns and detect anomalies and gain insights.
Okay, so it is more detailed in data understanding and decision making before advanced.
So see the whole idea is almost the answers are same but the wordings are different.
It is exactly like the same thing.
Let me try to explain you in this way.
Um, let me go down and explain.
So first of all, your I'm calling through Po, right?
Po is nothing but your open AI APIs right now here when the user is asking.
I'm basically getting an answer, right?
Obviously, when a user is asking, we are asking through a particular session, right?
When you are asking the same question in your Python notebook, eventually you are also using open AI
APIs, but you are using a different session.
What's happening is here also.
The question is same here.
Also the question is same, right?
Here the embeddings and here the embeddings will be same.
The retrieval mechanism is also same, but the chat completion is different because here imagine this
is person a person A is answering like this and person B is answering like this.
Because chat completion is completely random, the the context is same.
It is also explaining you idea.
This is also explaining the idea, but the way of writing the answers is different, but the matching
mechanism will be almost same, right?
So I hope you are able to get my point right.
So this is the basic thing.
Now we are done with the basic things.
Now what we will be doing is we will be onboarding the PDF document.
So we will be converting the document into some sub documents.
And then we will be doing the embeddings.
And all the embeddings will be calling to the model.
Similarly, when the user is asking a question again you are doing embedding again.
So basically here If you, uh, try to do this way, what's going to happen is embeddings and embeddings.
You get the most relevant passages, and then you call the model for chat completion, and then you
get the output.
So we will be following this architecture right now.
Right.
So let me write it down here.
So here using custom data set.
Let me just build it out.
Then let me write down the code.
And then I will explain you the code.
Okay.
So here what we are going to do is we will be using a um we will be using a rag paper.
I'm going to use this particular rag paper as an input.
And this is loaded.
So I'm copying the path.
So PDF reader is equals to pipe pdf loader.
And here I will pass the path.
That's it.
Okay.
Let me write it down.
Give me some time I'll write it down.
And then I will explain you the code.
So documents equals to PDF reader dot load.
Um.
My bad.
And then after the documents, uh, we need a text splitter.
Text splitter is equals to recursive recursive character text splitter.
And here I will be defining my chunk size.
Chunk size.
By default there is no chunk size.
Take splitter.
Anyways, let's say I'll pass some chunk size, let's say 1000 chunks and then chunk overlap 200.
Um.
That's it.
And then chunks equals to text splitter.
Um, dots.
Split documents.
Documents.
Okay.
Let me run this code.
Uh, it's not working.
Py PDF package is not found.
Um, py PDF package is not found.
Py pdf loader is using py pdf loader.
That's so weird.
My PDF is already loaded from Long Chain.
Uh, so it should work.
Package not found.
Please install pdf.
That's weird, because in my case, it should have been already installed anyways.
If it's not installed, maybe we'll write it down in the requirements.txt.
So I'll install this py pdf.
I have already installed Langton.
Internally, Langton is using Pip EDF, so it should logically also import also install this.
But it's not.
Anyways, I'll jump back to the code.
I'll run this.
It should work fine.
And yes, it is working fine.
Now let's try to understand this piece of code.
What's happening?
It's very simple to understand, right?
We have taken a PDF document.
Right.
And then we are doing the loading here.
What exactly is recursive character text splitter.
So if you go back to this architecture this is the part okay.
So recursive character text splitter maybe also I'll write it down here.
Recursive characteristic splitter is a text splitter that splits.
The text into chunks, um, trying to keep paragraphs together and avoid.
Losing context over pages.
That is what recursive character text splitter is doing.
Okay.
Basically, the resulting chunks of text are stored in variable chunks.
After your text splitter is done, you are storing all your resulting chunks in chunks.
That's it.
Now moving forward from lang chain dot vector stores import face.
What is f I s f a I s is nothing but a.
It's it's basically a meta.
Meta has provided this very good library.
It is a library used for efficient similarity search and clustering of dense vectors.
So let us import that from lang chain dot embeddings.
Import.
We have already imported open AI embeddings.
No, we have not.
Okay.
And then after that.
So let's say create embeddings.
Embeddings is equals to and directly call this open embeddings API key is equals to.
I will pass my open ai API key then.
And then db is equals to f I ss from documents, documents equals to chunks.
Um and embedding equals to embeddings.
Yeah.
Uh, could not import tick token.
Anyways, um, we will also be importing, uh, installing tick token.
I'll write it down here.
Okay.
Tick token should be installed.
And then let me run this piece of code.
And then I will explain you what's happening.
So till now everything is clear, right?
We have taken the PDF document into chunks.
Right now chunks needs to be created embeddings.
So here I'm basically calling the open AI embeddings using this open AI embeddings.
Um and it is obviously present inside the lantern library right.
It uses the open AI API keys because using that you are creating the embeddings, right?
So open AI embedding class basically is responsible for generating the embeddings, um, for the provided
textual data using the OpenAI GPT models.
After creating the embeddings, you can see we are creating the vector store.
Here I am passing my documents.
I am passing my embeddings.
Okay.
And then I'm creating the vector store.
Now what is faces?
I already talked about faces is basically used to create and manage vector stores based on the faces.
Face simply means.
Facebook AI similarity search.
Yeah, it's a powerful, powerful library for similarity search and clustering of dense Tense vector.
See?
Right now we are using faces going forward after this particular session.
We will also be creating some pinecone vector store.
And I will show you how the vectors are getting stored on pinecone right now.
I cannot show you, but when we use pinecone, we will be able to show you the vectors.
So you will be able to have a better feeling.
Oh, this is how it is working right now.
Let's try to move ahead from long chain chains.
We'll be creating some chain conversational chain conversational retrieval chain.
And then from lang chain um prompts import prompt template.
So condense let me write it down.
And then I will explain you in one go.
Contents.
Import prompt equals to prompt template from template um, given the following conversation and a follow
up question, rephrase the follow up question to be a standalone question.
Uh.
Chat history.
And then I will write it down as chat history.
Follow up.
Input question.
Stand alone questions.
Okay.
Um.
Should be good.
And then I will create Q a um is equals to conversational Signal chain.
Um.
Conversational chain from LM.
Conversational retrieval chain.
The LM right.
LM is lm LM equals to lm.
Um retriever is equals to db dot as retriever.
Condense question prompt is equals to uh condense question prompt.
Um and then return source documents equals to true.
Okay.
Uh okay.
Successful.
Now let's try to understand this piece of code.
Okay, so what we are doing here, we have imported two things, right?
We have used conversational retrieval chain and we have used prompt template.
Now first of all what is this conversational retrieval chain when we are calling conversational retrieval
chain from LM.
Basically this method initializes the chain with the specified parameters right.
The LM parameters is nothing but your open AI lm right.
The retrieval parameter is nothing, but it is the retrieval being used for retrieving the relevant
information from the document.
What is DB?
DB is nothing but your embeddings, right?
If you go here.
Your embeddings.
So let's say you this is your DB.
Let's say this is your face DB right.
This is nothing but your DB as retriever.
And then this is the template that we have passed.
What is this?
This specifies.
This basically specifies the prompt template to be used for condensing the follow up questions into
standalone questions.
Because you will be asking not just one question, right?
Let's say what is two plus two?
What if we add five to it?
Now you you can understand this thing.
This is the first question that I have asked right you when you are asking the next question, the previous
question also needs to be taken care of, right?
So accordingly, you have to give your prompt or else your prompt could be also this way let's say your
prompt could be let's say you want to create just a Q and a chatbot.
You don't want to create a chatbot to keep asking questions from the previous context.
Each and every question will be completely different, but you want your chatbot to behave like someone,
so you can also do that way.
Behave as a AI expert and try to answer questions from this particular research paper, or behave like
a research scientist who has 25 years of experience into research into deep learning technologies and
try to answer the question.
And then when you are asking a question, your bot is created as if he is a it's a 25 year old research
scientist.
Similarly, going forward, if you are doing a insurance related project or a medical related project,
you can also give a prompt behave as a general practitioner and answer.
So that is basically that you are passing in the prompt template.
Then what is written source documents.
True.
This basically indicates whether to return the return the source documents along with the response verbose
false is nothing, but it is a control that it is the parameter that controls the verbosity of the chain.
Okay, now this is done.
What's next?
Your queue is ready.
Our next task is to just ask a question.
Okay.
Let's say here I will write it down as ask a query.
Okay.
So let me define my chat history.
Let's say chat history is an empty empty list.
Query is equals to.
Um okay.
Let's say uh from this research paper, what is this research paper all about?
What is this research paper all about?
Explain in just 3 to 4 lines.
Um, and then I'm calling.
The result is equals to I'm.
I'll be calling this conversational chain.
And then here I will pass question is my query and chat history.
Do we have a chat history?
I'm not passing any chat history.
So chat history is chat history.
If you have a chat history you can always pass that.
Um that is my result.
And then I'll be just printing my result.
Answer okay, let's see what the answer is.
Great.
So our rag paper is being trained right now.
If you just ask this question directly to po, Po has no idea about which research paper are we talking
about.
But here we have actually trained the research paper.
right?
So if you go back to the architecture, I'll probably take you to the bigger architecture.
The complex one here.
Right.
Wow.
Um, okay.
Somehow it's still visible.
So you have the documents, and then you did all this.
Then you're creating embeddings, and then from your vector store, you are getting the relevant passage,
and you are calling the OpenAI model and getting the final output.
This is what is happening.
So what is this output?
This research paper is about a new method for multi paragraph reading comprehension using machine learning
techniques.
It proposes a model called Bert that is pre-trained on a large corpus of text, and can be fine tuned
for a variety of natural language processing.
This paper also discusses its applications in conversational agents and question answering.
Let me ask a few more few more questions.
Uh, something related to, uh, let's say.
Who are the authors?
Who are the authors of this paper?
Okay, let's try this out.
So somehow it is getting the references.
It is not doing a great job.
Um, the author should be.
These Patrick Lewis.
Okay.
It's not doing a great job, but but but it's okay.
Let's say we'll ask something.
What is the rag sequence model?
The rag sequence model uses the same retrieved document.
You can see word by word.
It has been copied.
Uses the same retrieve document to generate complete sequence to generate the complete sequence.
It treats the retrieved document as a single latent variable.
Blah blah blah.
And then you can see it's it's doing a great job.
Um.
What are the different experiments performed?
What are the different experiments performed in this paper?
Using on Bart for partial decoding.
Fact.
Okay.
Um, okay.
Fact verification.
Yes.
It's correct.
Uh, partial decoding.
Uh, partial decoding is also correct.
Partial decoding is mentioned here in the jeopardy question generation.
Him.
Training, setup, details.
Retrieval Ablations.
Yes, to some extent it is doing a good job.
Okay, last question.
What are rags and why do we use it?
Rag stands for retrieval Augmented generation.
It is a type of model that uses a combination of text retrieval and generation.
Now let's say if I ask some random question who is Sachin Tendulkar?
So this particular I mean, I was about to say the same thing because it does not have information about
Tendulkar right now in this case, if only if your requirement is if the question is not found in the
document, try to refer to the OpenAI model.
That is also possible, but we have not done that in this piece of code.
In this piece of code, our main motive was only to train a model on top of a custom data set.
So that's why it is not working.
Similarly.
Last question.
Um.
What is rags and.
Tell me more about use cases of rags.
In a detailed manner.
So rags are a type of natural language processing model blah blah blah.
Uh, some use cases of rags include open domain question answering where it has shown to outperform
state of the art models.
And you can see it's doing a great job.
Right.
That is it.
What are the different use cases?
You can simply take this piece of code and build Hundreds and hundreds of use cases.
One classic use case will be if you have your own documents.
Let's say you have your own machine learning ebooks or EDA, ebooks or statistical ebooks, or any kind
of ebooks you want to train.
Your ebooks for your own custom data.
Chatbot to answer something from that e-book.
You can use it.
And if you want to use bulk PDFs, multiple PDFs.
Instead of using pie PDF loader, you can use a directory loader, which we will be covering in the
next video.
So next video we shall be talking about how to create similar kind of application for uh, multiple
PDFs, custom PDF chatbot.
And going forward we will also be leveraging some different, um, large language models for this particular
task.
And going forward, we have bunch of use cases, uh, to talk about.