Hi, everyone and welcome to unit four of the SAP Data Intelligence course.
In this unit we will focus on some of the capabilities that are developed for the data scientist persona.
Some of the challenges which we face with our customers and the pain points, and the main reasons
behind developing SAP Data Intelligence, was that there was a missing link
between the customers' contextual data and their Big Data. Companies had their sales data in their ERP systems,
and at the same time they were collecting the click-throughs from the Web sites, and, for example,
wanted to study a customer's behavioral pattern browsing their Web site.
But this was not easy, because running code on both of these datasets
residing in different locations was not easy. So having a single platform that allows them to work
across multiple sources seamlessly was priceless. Also, it was easy for the data scientists
to open their Jupyter notebooks and run some Python or R code on top of some sample data.
Let's say 1,000 rows saved as CSV and getting great F1-score,
but then applying the same code on the real data, while data is changing, connecting to multiple sources,
surfacing the result in the business application, and so on, was the main challenge.
So, with SAP Data Intelligence we made the transition from the POC
to production as smooth as possible. Also, there are so many applications and libraries out there
that are focused on one single task, and they do a great job.
For example, one is developed for data preparation, another was developed
for running machine learning algorithms, another for visualizations.
But because these softwares are developed by different companies, they're not compatible
with each other. And if you try hard to get them to work together,
there will be a new version out which doesn't work in your old setup.
And what if the customer needs support? You can't ask the big enterprises
to seek support from the online forums. So, having a single platform
that takes care of the compatibility of different sources, versioning and support was a must for our customers.
A platform that also gives you the flexibility to run your open source libraries
and connect to different applications. Let's look at data science lifecycle management.
The first step for a data scientist is to define the business question they need to answer.
Sometimes this happens while they are exploring their data. They usually deal with the siloed landscapes
and various data types. So, after they have created the correct connections,
they need to look at their datasets. Usually they have to cleanse or enrich their datasets,
before they can define the method they need to approach the problem.
Maybe this is a classification problem. Then they need to run some experiments,
compare different algorithms results and find the best possible model.
And after they're happy with their result, they have to operationalize their implementation
in a production environment, and embed their result in a business application.
So, gaining intelligence is an isolated operation, but it should be a continuous process.
SAP Data Intelligence has multiple components that make the data science life cycle
easy for data scientists. So they don't need to go back and forth
between different applications to get their jobs done. To identify the business questions,
they can use the launchpad connection management component. Native connections are already provided
and for the connections that are not available yet, they can use an open connector to build that connection.
Data scientists can also use the Metadata Explorer for data discovery, cataloging, and profiling.
Metadata Explorer can be used to extract lineage or to define some quality and validation rules
which are applicable to the incoming data streams. There is also a self-service data preparation component
which allows them to interactively make changes in their dataset, create a recipe out of all the changes,
and apply that to a new dataset. After they're happy with their dataset,
data scientists can use a Machine Learning Scenario Manager, we call it ML Scenario Manager, to define their problems.
This component provides features for versioning, and it is a centralized place
for datasets, models, executions, and deployments. The benefits are that model performance and life cycle
are automatically managed, so IT and data scientists can focus on the tasks that matter.
And it provides one single interface to see all ML models across the organization
in a unified workspace, with tools to help manage governance, auditability, and transparency.
We will talk more about ML Scenario Manager in unit six, but here is a view of a registered dataset
in ML Scenario Manager, or the notebooks added to a single scenario, or even a managed pipeline
for the purpose of the operationalization. You can also execute or deploy your models
from the same scenario manager. In addition, you can access your lab, JupyterLab,
through the ML Scenario Manager to run your ML experiments or run some visualization.
And after you're happy, you can use the SAP Data Intelligence Modeler
to improve your model by adding new operators or connections, or just to deploy your model.
The good thing about the Modeler, is that it allows you to execute the operators
internally or externally. For instance, if you have some Python code,
you can run it internally on SAP Data Intelligence's Kubernetes cluster,
but if you have some SAP BW process chain, you can push that down
for the execution to your BW system. So you don't need to move the data around, if not needed.
And after you're happy with your implementation, you can decide how to run your processes.
Sometimes you have developed multiple pipelines. Then you can decide if you want to split your flow,
and run those two processes in parallel, or if they're depending on each other,
and you need to run them in sequence. Workflow capabilities in Modeler make orchestration easy.
Also, you have access to monitoring capabilities to drill down to the problems,
or just collect some analytics on the processes running in your landscape.
The monitoring capabilities are available for processes, instances, and also the scheduled tasks.
Now, let's take a look at some more capabilities which Data Intelligence offers for data scientists.
If you're one of our customers who has HANA in their landscape,
you may have developed some of your algorithms using SAP Predictive Analysis Library, or PAL,
or Automated Predictive Library, or APL. Or you may have used
some of the SAP Leonardo AI capabilities. The good news is that, with the SAP Data Intelligence,
you can still benefit from them. Let's say you used Leonardo image feature extraction
to extract some features from your IR images, and you want to run PAL on HANA
to classify your product as defective or not, the use case with the customer,
which we discussed in unit three. No problem, you can use SAP Data Intelligence
to trigger those executions and push them down to the HANA system, or delegate the job
to the Leonardo MLF clusters, get the result back, and do more.
You can also orchestrate, execute, schedule, and monitor the result in a production environment.
Please note that HANA ML is not part of Data Intelligence, but you can use it from Data Intelligence.
In the next three slides we will give you an introduction to APL, PAL and Leonardo AI, SAP's offerings for ML and AI.
SAP HANA Automated Predictive Library, or APL, capabilities are available to any application
that's able to use and consume HANA services, either on premise or in the cloud.
Which allows you to provide automated classification, regression, clustering, time series,
and descriptive analytics within HANA. With APL functions you can create your models,
train your models, apply your models, and also query your models,
for instance, retrieve variable descriptions, indicators, reporting information, and so on.
SAP HANA Predictive Analysis Library, or PAL, has an extensive list of machine learning algorithms implemented
and run natively on HANA. From classification, such as decision tree, random forests,
K nearest neighbors, Naive Bayes, and so on, to regression and clustering such as kmeans, LDA, GMM,
agglomerate hierarchical, and even association analysis, such as Apriori, FP-Growth.
Also probability distribution, outlier detection, recommender algorithms, link predictions, and so on.
So, if you have HANA, and your data is saved in the HANA system,
you can trigger the execution of your PAL algorithms on your HANA data and get the result back.
And that way you don't need to move your data around. Leonardo AI allows you to make use
of predefined machine learning services or use your own models.
There is an operator developed for you called Functional Services, which provides you
with the ML services out of the box. Services for image and text analysis,
such as image classification, or topic detection, and so on. You can also use TensorFlow models
for deep learning to train or bring your own model. We will talk more about this later in unit 10.
In this unit you learned that SAP Data Intelligence includes different components to support
your end-to-end data science requirements. And it is a flexible platform,
which allows you to use your preferred language or platform, while taking care of their compatibility,
versioning, and support for your enterprise. Hope you enjoyed this session.
In the next session we will see a demo of the components of the launchpad, thank you.