Welcome back.
Hope you managed to complete the assignment.
If not, please follow along with me.
In the earlier lessons, we set up the environment for the raw and the ingestion layers, but now we
want to process the data for our presentation layer.
So in this lesson, let's create the environment for the presentation layer.
You might be wondering why we are creating another container.
Data projects usually have different layers to keep the data for a different number of reasons.
Most importantly, it helps the consumers to understand the nature of the data they are dealing with.
Also, it helps to provide the right access to the right consumers.
For example, you do not want an API to have access to the raw data.
Instead it should have access only to the presentation layer.
So by splitting them into different layers, you can manage your security and access controls as well.
Different architectures named the layers differently and there can be any number of layers depending
on the project need.
Most projects will have at least three layers.
As we've seen, Databricks architecture recommends calling them bronze, silver and gold.
Bronze would have the raw data and silver would have the data with the structure applied and data cleaned,
etcetera.
Gold would have the data as required for reporting and presentation.
We've called ours as a raw, processed and presentation, and they're synonymous with bronze, silver
and gold.
You can follow these conventions or if you want to change this, please feel free to do so.
Let's start by creating the presentation layer.
This requires us to create the presentation container in our data lake as well as create a mount in
our databricks workspace so it has access to the container.
Let's start by creating the container first.
Here we are in Microsoft Azure Storage Explorer, and this is my subscription, which is the subscription,
and this is the storage account that we created for our project, which is Formula One.
And under that we've got the two containers that we created raw and processed, and we want to create
the third one.
And this is going to be called presentation.
Okay, that's now created.
So let's switch to our Databricks workspace and create the mount point for that.
We have already created our setup script, so let's, let's navigate to that.
So that is under this Formula One folder under setup, we got the mount storage and here we created
the Mount for Raw as well as processed.
So now we need the third one, which is going to be the presentation.
So let's execute the notebook.
We don't want to execute the entire notebook because we've already created the raw and the processed
containers and the mount points.
So let's do what's necessary.
So we need this statement just to set up the secret details and the configs here.
And then we also want the function Mount ADLs.
So we just, we want to miss these two and we want to run the presentation.
So Mount ADLs presentation that will mount the new container for us.
That's now mounted.
Let's list the Mount Point just to make sure that we can see the mount.
Yeah, we got an empty string.
Output or empty list?
Output.
That's because we have got no files in this folder, no files or folders in this container.
So this has worked.
So the one more thing we need to do here is under configuration.
If you remember, we are keeping the folder paths, so let's keep the folder path present for our presentation
as well so that we can use it in our notebook.
Okay, so that's done.
So that's everything.
Now, the next thing we want to do is to create the notebook to transform our data.