We've previously talked about
how to find a suitable problem
by understanding how you evaluate your business success.
In this video, you'll understand how to evaluate
the actual implementation of
your machine learning solution,
and how to compare it with
your existing business practices.
There are two different aspects of evaluating
the performance of your QuAM in your business.
There's the technical assessment of
your QuAM and there's also the business evaluation.
Later on we'll talk about
the technical assessment of QuAMs.
But in this video, we'll focus on
business considerations for evaluating QuAMs.
Business evaluation is looking at
QuAM performance through the lens of
things like return on investment,
key performance indicators, and management criteria.
Usually a machine learning scientist is
highly motivated to find the best QuAM,
how close to perfect can you get your predictions to be?
However, more often than not,
straining for that incremental gain isn't necessary.
It isn't always the case that
perfect predictions is an appropriate criteria for
business success and it might not be
the thing that the business is hoping to optimize.
For example, if I tell you
that a client of mine implemented
a QuAM with only 70 percent accuracy,
you might be appalled.
However, this client did not think the investment in
time to improve the accuracy of the QuAM was necessary.
Even with only 70 percent accuracy,
this QuAM helped replace
a predominantly manual and undesirable workflow
that took employees multiple weeks.
Implementing the QuAM shorten
the workflow to only a couple of days,
freeing up the team to do more of
the challenging tasks they enjoy.
So we need a way to assess the performance of
the QuAM in our business application.
To justify the use of the QuAM in production,
we need to compare it to some baseline.
The baseline is a metric indicating
how successful you are at
solving the problem today
without any machine learning solution.
The goal of building a QuAM should be to improve on this.
Let's go back to our earlier example,
but be more specific.
Employees tend to be correct
about 75 percent of the time.
However, they find that task extremely frustrating.
It takes about three weeks for
employees to manually complete the task,
and this manual process costs
the company around $1,000 per case.
Here we have three quantifiable metrics:
performance, time, and cost.
There's also the qualitative point indicating
the current solution is undesirable to the employees.
The goal is to create a new solution to
remove this task from the employees,
so they can do the work they prefer,
improving employee satisfaction while
saving time and consequently money.
As mentioned before,
the business decided that accuracy was
less important than removing
that unwanted workload and saving the time,
and so they chose to move the QuAM into
operation with only 70 percent accuracy.
The point of the story is to
understand if a QuAM is fit to be put into use,
you need to consider and balance multiple metrics.
The main three we consider here are
performance, time, and cost.
Let's start with performance.
Performance measures how good the solution
is that answering the questions correctly.
This metric depends on the type of
problem the QuAM is trying to solve.
Whichever metric is chosen for the QuAM should also
be used to calculate the performance baseline.
Another metric, time, is
the duration it takes to complete the task.
For the baseline, this is how long
the task takes without using a QuAM.
The third concern is the monetary impact of the task.
For the baseline, this could be related to how much
it costs to complete the task with the current solution.
A machine learning scientist
may get very fixated on performance,
maybe because this is what we have control over.
However, in order to get the QuAM used in production,
it's important that the other quantifiers
are evaluated and communicated.
Stakeholders can then make an informed decision
on whether to move forward
with the QuAM that's been built.
As another example, let's consider
a classifier QuAM that
finds fraudulent banking activities.
Currently, the bank has simple rules that
flag suspicious transactions such as
those above a particular threshold amount
or purchases made out of state.
For each flagged transaction,
an employer reviews the account owner's profile and
prior transactions to better
understand if the transaction seems unusual.
Then they use their best judgment to either allow
the transaction or take appropriate actions,
if they consider it to be fraudulent.
In order to improve this process,
the bank wants to build a classifier QuAM
that better identifies fraud.
The QuAM should reduce the number of times the employees
manually review transactions that
turned out not to be fraudulent,
as well as capture
all those transactions that are in fact fraudulent.
The bank needs to decide how
to use such a QuAM in production.
One option is to keep the employees in
the loop having them still evaluate the profiles,
but make sure they have fewer profiles to review.
Alternately, we could trust the QuAM to
move straight to the next step in the process,
calling the customer to verify
that transaction was theirs.
This is a decision that will
need to be made by the business,
but the following metrics can help guide that decision.
As before, let's start with the performance baseline.
What percentage of transactions flagged as
suspicious were in fact fraud?
How many of the transactions let
through were actually fraudulent?
We compare any QuAM with how well
fraudulent activities are identified today.
Next we have the time baseline.
What is the time between a transaction taking place and
when it's identified by the bank agent as fraud?
How long does it take for an agent to confirm with
a customer whether a transaction was valid or not?
There may be an opportunity to use a QuAM
to shorten the agent's time handling this task.
However, delaying the release of funds when
a bank agent is doing it may also have consequences.
Finally, we have the monetary base line.
What's the operational cost for
the financial institution to use a classifier QuAM,
including development and maintenance costs?
Conversely, how much money does it cost to
follow-up with the customer
to confirm their transactions?
Any occurrence of fraudulent transactions
costs money to the business.
At the same time, it's not feasible to hold
the funds until each transaction
is verified by the customer.
Understanding these associated costs can
allow you to decide the appropriate business action.
You have to take these three metrics; performance, time,
and cost, and put them into your business contexts.
For each metric, compare
the QuAM to your current baseline.
How much business value does that QuAM provide?
Those answers will form
a basis for your business decision.
So we just saw examples of how a QuAM
can be evaluated in a business context.
This evaluation gives you an idea of how
justifiable the use of the QuAM in your business is.