In this unit, we will learn more about working with the pipeline modeler.
In the previous units, we showed you how to use the ML Scenario Manager to create a pipeline.
As we mentioned, pipelines can be used to develop your business processes,
or to productize your data scenario and data science experiments.
When you click on Create Pipeline in the ML Scenario Manager, a window will pop up that allows you to create your pipeline,
using some of the pre-built templates. One of the templates is Python producer,
which can be used for model training using Python. As you can see, using the Python producer templates
not only creates the proper pipeline to run the job, but also submits the KPIs to the ML Scenario Manager,
and also saves the model somewhere in the SAP Data Intelligence's Semantic Data Lake,
which will be accessible to the Python consumer. It also adds the proper API,
as you can see in the picture below, to run the Python code on pipeline.
You can also use an R template, that will create the above pipeline for you,
and will allow you to run your R code on the pipeline modeler, for operationalization.
Also, creating an instance of the TensorFlow template imports the necessary TensorFlow libraries,
and provides you with easy configurations, where you can easily decide to run your model
on a CPU or a GPU. We talked about the Leonardo functional services in unit two,
and we will talk more about them in unit 10. Here, you can see how to use ML Scenario Manager's pipeline templates
to access some of the functional services. For instance, in this case, I created a template
of the translation service, which automatically creates the above pipeline for me, and if I open my script,
I can simply enter the parameters, such as source, and target, and use the translation
with minimal effort. Because this is a pipeline,
you can easily add more operations to it, and extend your business use case,
it makes it really easy for you. Let's dive deeper, and have a better understanding
of what the operators of pipeline are. The pipeline modeler is the processing power
behind SAP Data Intelligence. It provides the execution environment
for all processes and applications. An operator, however, is a single computation unit.
The idea is that we have a data stream, which flows through the operators,
and each of these operators performs some task on top of the data.
The input of each operator should have the same type as the output of the previous operator.
A graph of operators is called a pipeline. The operators can run internally in the pipeline,
or you can delegate their execution to their connected system, using the workflow operators.
For instance, you can use a workflow operator to delegate the execution to your BW
or Data Services system. The pipeline modeler is delivered
with around 300 pre-built operators, and you have already seen a demo of this in unit five.
If the operator that you need is not part of SAP Data Intelligence, you can either create
your own custom operator using your preferred language, such as JavaScript, Python, R, or Golang,
or if you have already developed lots of Python or C++ code, and you want to reuse them on your modeler,
you can compile them and run them on our C++ or Python subengine.
Using the operators gives you lots of flexibility, whereby you can easily replace one operator
with another, by simply dragging and dropping. Let's say you were reading from your S3 storage,
and now you want to read from the Semantic Data Lake, without disrupting your data flow.
That's really easy. You can also scale your operations by deciding
how to execute your processes, and because the pipeline modeler is developed
on top of Kubernetes, it will take care of the distributed executions
and resource management for you. The custom operators and the pipelines
that you have created are re-usable in your future pipelines. In the picture, as you can see,
I have added an operator to read from a file saved in HDFS, then I have added a Python 2 operator,
where I have written my code to run sentiment analysis, and at the end I use the Vora Avro Ingestor operator,
where I can write the result as an Avro file in SAP Data Intelligence's
scalable relational database called Vora. The wiretap operator acts as a terminal
to show you the result of your process. As we mentioned, SAP Data Intelligence
is developed on Kubernetes. To understand what happens under the hood,
we have a model repository, where we save all our pipeline artifacts,
such as the operators, the Dockerfiles, the tags, and so on. When we start a pipeline execution, a Docker image
will be created by the image composer, based on operator tags and on demand.
The Docker image will then be picked up by Kubernetes to be translated into containers, which run in pods,
and, of course, are managed by Kubernetes. This translation is sometimes one on one,
depending on your configuration. Here are two examples of using the R operator,
and also triggering some PAL execution on HANA, using the pipeline modeler.
Not only the training, but also the deployment, can happen on the pipeline modeler,
where the deployment operator will expose a REST API, which can be consumed with your business application.
You can also trace your pipeline execution by activating the trace capability in the pipeline,
and turning the trace level to DEBUG. You can also use the Kubernetes dashboard or the kubectl
to find the log for the particular pod you are running, and drill down to the problem.
For instance, the log here indicates that there is a problem with the HDFS connection.
Hope you enjoyed this unit. In this unit, we had a deep dive into the pipeline modeler.
In the next unit, we will particularly focus on the Python and R languages,
which are the two most preferred languages for data scientists, and show you a couple of demos
on how to run them on SAP Data Intelligence. Happy learning!