In this video, we dive deeper into the engine behind our chunking workflow.
You will see how we handle storage, how the main file orchestrates the entire process,
and how everything finally comes together in the output.
Let's get started.
In this part of the system, we focus on how chunks are stored, retrieved, and organized
inside a local SQLite database.
We begin by defining the database name, which in this case is chunks.db.
SQLite is lightweight, serverless, and easy to integrate, making it ideal for small-to-medium
scale document processing pipelines.
The first function is init underscore db.
This function is responsible for initializing the database and ensuring the required table
exists.
It creates a connection to the SQLite file, obtains a cursor, and executes a table creation
statement.
The table is named chunks and contains fields such as the document ID, chunk index, title,
summary, keywords, and the actual text content.
If the table already exists, SQLite simply ignores the creation request.
After execution, the connection is committed and safely closed.
This function is typically run once during application startup.
Next, we have the save underscore chunks function.
This is where we take all generated chunks for a given document and write them into the
database.
We again open a connection to the SQLite file and iterate through the list of chunk dictionaries.
For each chunk, we insert a new row into the chunks table.
We store the document ID, the sequential chunk index, and any optional metadata such
as title, summary, and keywords.
If keywords exist, we convert the keyword list into a single comma-separated string
for database storage.
The full chunk text is also stored in a dedicated field.
After inserting all the chunks, we commit the transaction and close the connection.
This gives us a durable, queryable storage format for semantic and agentic chunking outputs.
The third function, load underscore chunks, is used to retrieve stored chunks from the
database.
It accepts a document ID and queries the chunks table for all rows matching that ID.
The results are ordered by chunk index so that we reconstruct the text in the correct
sequence.
Once the query results are fetched, we iterate over each row and rebuild a structured Python
dictionary for each chunk.
The keywords column, which is stored as a comma-separated string, is split back into
a list.
The function then returns a list of chunk objects that can be directly consumed by retrieval
systems, analysis pipelines, or downstream AI agents.
Overall, with this in place, your chunking system moves from being an in-memory utility
to becoming a fully persistent, production-ready workflow.
Let's quickly walk through the main file, which acts as the central controller for our
entire chunking workflow.
It begins by importing the chunking engine and the storage utilities.
Then it prompts the user to paste a multi-line document, which becomes the input for every
chunking mode.
Once the document is collected, the script initializes the SQLite database to ensure
the chunks table is ready.
Next, it loops through all five chunking modes – fixed, recursive, document-based, semantic,
and agentic.
For each mode, it generates chunks, reports how many were created, and saves them into
the database using the mode name as the document ID.
After processing all methods, the script retrieves every stored chunk set from the database and
prints them out.
For each chunk, it displays the index, any available metadata, and a short preview of
the text.
This gives us a full end-to-end run, taking input, chunking it, storing it, and verifying
the final output.
Now it's time for execution.
Let's open the terminal and run the program using the command.
Once the script starts, go ahead and paste the content you want to process.
This will trigger the full chunking pipeline and store the results in our database.
As you can see, the system has now created chunks for all five modes.
Each set of chunks has been processed, indexed, and successfully saved into our database,
stored inside the chunks.db file.
And here on the terminal, you can clearly see an example chunk with its title, summary,
keywords, and a preview of the extracted text, confirming that our pipeline is working end-to-end.
Thanks for joining in.
I'll see you in the next one.
Until then, keep learning and exploring.