Hi everyone, and welcome to unit nine of the openSAP course on SAP Data Intelligence for Enterprise AI.
In this unit, we will focus on how to run your scripts developed in Python and R languages, on SAP Data Intelligence,
since they're the two most preferred languages for data scientists.
In the first demo, we will focus on the ways we can run Python on SAP Data Intelligence.
As you can see here, this is the SAP Data Intelligence launchpad,
where you can access your ML Scenario Manager. There, you can open one of the scenarios
that you have already developed. For instance, here, I have developed
Predict Return Propensity for my customers, and that's the latest version that I have created,
version four, so I click on it, and it takes me to my Data Intelligence ML scenario.
And one of the ways you can run Python is through the Jupyter Notebook.
So, when I click on a Jupyter Notebook that I have already created, or create a new one,
it takes me to my Jupyter Lab, where I can install my libraries by doing pip install.
And also, after I finish installing, I can import them, for instance, import pandas,
import NumPy, scikit-learn, in this case, and so on and so forth.
I can also run multiple notebooks as part of my Jupyter Lab.
I can import a notebook that I have already created, or I can simply create a new notebook
using the ML Scenario Manager. So in this case, for instance, I ran my prediction,
I used logistic regression, these are my, for instance, classification report,
my precision recall, F1 score, and things like that. And then once I'm happy with this Jupyter Notebook that I have created,
I can export this as a Data Hub format, which actually adds the API for me that is needed,
in order to run the code in the Python operator, on top of the SAP Data Intelligence pipeline modeler.
And like we mentioned also, we have to remove, you know, for instance,
the part where we have to execute, we have to install the libraries,
and then we use a Dockerfile for them, and we will talk about that.
I can also export this notebook as an executable script, which is a .py script
that then I can make the changes, and add the APIs to it, which run on top of the pipeline.
I can also create a pipeline, for instance, say, Python Producer, or Python Consumer,
which adds the proper Python operators, and also the other operators that can consume that result
as part of the ML Scenario Manager. So this is another way of running Python
on top of Data Intelligence. So here, for instance,
I used the template for Python Producer, and as you can see, I already have
the Python 3 operator there, and I can basically add the code there.
And I can trigger the execution of it from the ML Scenario Manager.
Also, you can create a blank graph, and create your whole pipeline from scratch,
not using any of the templates. For instance, here, I added a read file,
I want to read some file, then I want to consume that file
as part of my Python 3 operator, and then I want to show the result on the terminal.
So, I can easily go to the Operator tab, I can search for the proper operators that I need,
for instance, read file Python 3 in the terminal. Then I figure that I need to add
some ports to my Python 3 operator. For instance, I need an input port,
to get the data from the read file operator, and I need an output port, in order to
pass the result to my terminal. And I realize that the type of data that I'm reading from the file
is different from the type of data that I want to consume as part of the Python operator,
so the system recommends a ToString Converter for me, which I add here.
Then I can open my read file, and I can add the proper configurations,
for instance, where I'm going to read the file from, in this case, I have saved my file locally,
so I say the path to read the file is /vrep/CleanedReturnDataset2.csv.
And I can say if I want it, for instance, there are some configurations that you can do,
and it's available to you out of the box, for instance, the different types of service you can access,
for instance, ATO, GCS, and so on and so forth, and I want to read this file as a batch,
and only read in change. Otherwise, don't read the file.
And so, I can open the Python 3 operator, I can copy my Python code,
and I can paste it as a script, from my Python 3 operator. And after I'm happy with that,
I need to basically go and group my Python 3 operator, and actually add the libraries that I need as a tag,
so that during the execution, basically, Kubernetes knows where those libraries are.
So, as you can see here, so I have, for instance, pandas,
for instance, scikit-learn, these are the libraries that I'm using.
So, I have already created a Dockerfile called PredictReturnLibraries,
but let's say if we want to create it from scratch, I just say, create a new Dockerfile,
and then I go and copy and paste this information, for instance, pip install, pandas,
NumPy, SciPy, scikit-learn, PyDotPlus, and I take that from the com.sap.python36 image,
which is part of the SAP Data Intelligence. And then I want to add different tags to it,
and as part of that, I want to give it a name, PredictReturnLibraries,
and I save that Dockerfile, and then I build that. And after I'm happy with the build,
then I can go to my Python 3 operator, like I mentioned, and group that, and then add the proper tag to it.
So I add a new tag, and I don't need to tag all the libraries,
I can just assign the name that I have used to create that Dockerfile,
as a PredictReturnLibrary. And now, I'll be ready to run this pipeline.
So then I save it, and then I run it, and then that way, after it runs,
I can see the result as part of the terminal. So this is also another way you can run Python
on your SAP Data Intelligence. So you can basically use the Jupyter Notebook,
or you can use some of our templates that we have already developed,
or you can come to the pipeline modeler, and develop everything from scratch.
And once the pipeline is running, you can see the result, for instance,
the features, and then the result of the training
and, you know, F1 score, accuracy recall, and things like that.
Perfect. In the second demo, we will focus on the ways
we can run R on SAP Data Intelligence. The goal here is to forecast the number of car registrations
in the next 12 months in European countries, using R. There are three different regions we want to look at:
Europe North and South, as well as DACH, which stands for the German-speaking countries,
Germany, Austria, and Switzerland. The bar charts on the left
show the forecast for 12 months. In the dashboard, you can see the upper bound
and the lower bound, as well as the actuals. The upper and lower bounds
are the values that we have predicted, and as you can see, the actuals are within the range.
Okay, so this is the ML scenario... this is SAP Data Intelligence,
where you want to access your data, and in order to access your data
you can use the Metadata Explorer, and you can browse your connections.
And in this case, we want to look at the VEHICLEREGISTRATIONS,
which is saved as part of your HANA. So, you can look for a vehicle,
and that will bring up your dataset. Then you can say that you want to look at the fact sheet
related to that dataset, and that will bring up the information about that dataset.
You can do a data preview, let's say, on the 500 rows of the dataset.
And as you can see, for instance, you can see the values there,
and you can also get more information about it. Then you can go to VEHICLEREGISTRATIONS_FORECAST.
You can click on that dataset, and look at the fact sheet of that dataset.
That is also saved in your HANA system. And as you can see, for instance,
you have different column types here. You can see the Data Preview,
and the Data Preview shows you, that the measure for the lower and upper values
are null here for most of the rows in your dataset. And that is what we want to predict,
and the goal is that the actuals that we have are actually within this range.
Okay. So let's go back
to our ML Scenario Manager, and look at the scenario that we have already developed.
Remember, we want to develop this story using our R code. So I have already built the pipeline that I need, to run R here.
If I click on it, it takes me to the pipeline that I have already created.
There, you can see that there is an R operator, that allows you to run your R code.
So here, the pipeline is a little bit bigger, where I pass some of my SQLs to my HANA Client,
so the data generator operator here is only to pass those SQLs to the HANA system that I have.
For instance, SELECT DISTINCT COUNTRIES FROM ML VEHICLEREGISTRATIONS, and things like that,
and then I want to bring all those datasets together, and then run my R code on top of that.
So if I open the R script, I see my R code there,
where I have all of the functionalities that I need to run,
in order to run that forecasting on my time series data,
that can basically have any value. So then, I can,
after I ran my, basically, forecasting,
then I can write the result into HANA. So this is a table in HANA that I have already created,
and once I run this, I can actually see the result in my HANA system.
So like we mentioned, it takes a little bit of time to collect all the resources
that Kubernetes needs, in order to run my pipeline, and then, after the run is completed,
I can see the result in my HANA system. So if I go back to SAP Data Warehouse Cloud,
then I can see that the HANA table is now populated with the lower and higher values predicted,
or lower bound, or upper bound, that is predicted for the registrations.
Perfect. This scenario can be used for production planning,
to more accurately predict peaks in demand. Hope you enjoyed this unit.
In this unit, you learned how to develop your Python code on SAP Data Intelligence.
You also saw an example of demand forecasting in R. In the next unit, we will talk more about
the intelligent services with SAP Data Intelligence. All the best for your self-assessment quiz.