- [Instructor] In this notebook, we're going to do a bit
of a deep dive into LangSmith.
Now, this notebook was co-authored
with my friends at the AI Makerspace.
I highly recommend you check out their YouTube channel
for hands-down the best educational content
on all things LLM and all things AI engineering.
Be sure to connect with Chris Alexiuk
and Greg Loughnane on LinkedIn.
I've learned so much from Chris
and Greg about AI engineering
and working with language models.
So I'm going to really quickly breeze
through everything in the first half of this notebook
because you've seen this all before.
We're instantiating an LLM, setting up Nest sync AO
so that we could do asynchronous stuff
in a Jupyter Notebook.
We're instantiating a document loader.
This document loader is actually just a site map loader.
So it's going to go through here,
the site map for LangChain
and just essentially it's loading in all the docs.
So we can take a look at this once it's done.
All right, and we can see here that we have a list
of documents, right,
which is 169 elements, right?
Each one of these document objects
is essentially the entire text
for a particular page.
So this page is generating usable text
with AI, so on and so forth.
And you can check out the blog itself
if you'd like by go in here, right?
So essentially, this is what we've done.
We've loaded the entirety of the LangChain blog
into a list where each page in the LangChain blog
is a document object now.
We're going to split the LangChain blog up.
We'll use a chunk_size of 256,
chunk_overlap of 16.
And so we've got 2014 elements of documents.
So just to verify, right, you can take a look
at one of these split_documents, right?
Let's look at the first element in this list,
the 42nd element of this list.
And you can just see it's just a chunk of text.
We've got the associated metadata with it as well.
It's just text.
We are going to instantiate a embeddings model.
We're going to go ahead and set up a vector store now.
I use capture, so I don't see all the printout
from all the stuff installing,
just keeps the notebook clean.
We're then going to instantiate the vectorstore
and set up a vectorstore retriever, right?
So we've seen this before, right?
If you're forgetting what this process is,
you can go back to the retrieval section of the course
or you can look at the video
on retrieval augmented generation
because that's essentially what we're doing here, right?
So we're going to let this work and do its thing.
So right now what's happening
is all those chunks of documents
are being passed to our embeddings model.
Those embeddings is just a fixed length vector, right?
It's just vectorizing
each one of those chunks,
putting it into this vector database, right?
And now we have a retriever that is set up.
So what's going to happen is
that a user is going to input a query.
That query will be embedded
using the exact same embedding model
that we used to embed all the vectors in this vectorstore.
It's going to fetch similar documents
and inject that right into this prompt_template right here.
So of course, we are familiar with prompt_templates.
We know what this is all about.
We also know all about how to create
a chain using LangChain expression language.
And here we are just going to invoke
and get a response and we get a response.
Great.
All right, so now,
this is where we're going to see some LangSmith action here.
So it's going to be cool to see this side by side
with the LangSmith screen here.
So let's go to the project here,
just setting environment variables so that LangSmith knows
that we're now tracking the LangChain API key.
You should already have that.
I'll go ahead and enter mine.
So let's look what's going to happen here.
So let's go ahead and invoke this generation
and see what happens here on our Projects page.
Now, you can see here that we have a new project
that just was opened up, right?
So this is the current date and time, right?
And this project has now kicked off, right?
And we could take a look in depth
what is happening under the hood.
So we have a RunnableSequence.
What's happening in the RunnableSequence.
We've got, you know, this question coming in.
These are all the documents that are being retrieved.
We get all this source information,
location, all this great metadata,
and we get the output from the LLM as well.
Let's go ahead and dig a bit deeper here now.
So now what we're going to do is we're just going to create
a testing dataset.
So let's go ahead and create a testing dataset.
I'm going to go ahead and navigate to the dataset page here
and you'll see what will happen, right?
This will take a second to load into its thing here.
So we're getting this warning
because apparently I had passed in the wrong name
for an embeddings model,
but it's okay.
This will just work just fine for us.
Let's go ahead and make sure
that our dataset has generated,
which it has, that's the current date and time.
And now we have a dataset here.
You can click in on an element of the dataset.
You can see here, here's a question,
here's the page content
that is retrieved for that.
We can also do some evaluation as well.
In this case, we're going to do some evaluation on, you know,
harmfulness and some additional evals as well.
So let's go ahead and just run this
and we'll see it come up on our Projects here.
You'll see here that we also get a nice link
that we can click on.
So let's go ahead and just click on that link, right?
Here's the question.
We get all this wonderful metadata.
You can see all these runs are still pending.
This is all, again, you can go to Dataset and Testing.
This is going to be saved here, Projects as well.
This will be saved here as well.
So you can see how many tokens we've used,
cost 15 cents to do all this stuff.
And you can see all these particular runs
that are happening, right?
So here's the harmfulness criteria
that is being checked against, right?
And we get the response here
from the language model, right?
And so we get all this wonderful tracing, so much detail.
You can see all the detail that we get from LangSmith.
So we get really full visibility
into our application development.
Here, of course, if we want to do
some more re-ranking, we could do that.
So you'll need your Cohere API to do that.
So this is just kind of added extra stuff here
and I will see our new chain.
So let's go back to Projects here.
We have a new evaluation criteria.
The code itself isn't that important.
I just want to show you how you can see
all this stuff in LangSmith.
So we'll go ahead and let this run, right?
We get a direct link to our project that we can check out.
Of course, it's pulled up right here as well.
We can take a look here
and you can see all of these evaluations that are running.
Get full visibility into all this wonderful information.
You can add your feedback here if you'd like to.
There's metadata associated with it,
or we get information in terms of latency and cost.
And you know, you can click into this
and each one of these, you know, we can see the scoring.
You even click into this a little bit further if you'd like.
And you could see all the different bits
of contents that's retrieved.
You can click on Open Run and get even more under the hood,
see all the retrieved context
and the response from OpenAI here.
You could see how with LangSmith, you get full visibility,
transparency into the development of your application.
So that's it for this video.
This next video is actually not about LangSmith,
but it's about how to manage the length of your prompts.