So how can we change the MDP reward function without changing
the optimal policy?
So in an MDP, we usually define it with the states, the actions,
the rewards, the transitions, and our friend gamma.
>> Mm-hm. [LAUGH] What we're going to mess
with here is the R, but we're going to leave everything else the same.
So what are some ways that we might be able to change a reward function that
actually wouldn't change what the agent would want to
do to maximize the reward function.
>> Well there's a couple of I think, obvious ones.
So one is if you just take the reward function and
multiply it by any positive constant.
Everything should be the same.
Right? >> Yes, exactly so.
So in some sense the rewards are kind of unit less, so
if you multiply them by 1,000, it still encourages the same behavior.
It just seems like you're making more reward points, but they're not,
it doesn't change the relative strength of different behaviors.
So, good, good. That's definitely one.
>> And I want to say this is true.
Just like by multiplying by a positive constant doesn't make a difference,
it seems to me that if you just shifted everything over by one or
by two or by three,
it would make a difference of adding a constant to the reward function.
Adding a constant as opposed to multiplying a constant.
>> Good, okay, good.
We'll talk about that one as well.
And there's a third one that we're going to talk about, that may not be so
obvious.
But do you have any other guesses?
>> This makes me think it desperately looks like what we did
in the other machine learning class with clustering.
There has to be non-linear ways of messing with the reward function to get
you the same answer.
But I can't think of a compact way of saying what I'm thinking.
>> Okay, so that's, that's fair.
And we'll get into that in more detail.
Ultimately, we're going to call it potential based modifications, or
potential based rewards.
And I'll define in a bit, but first I want to go through the,
as you pointed out, the simpler cases and develop a kind of methodology for
proving that we're not going to change the optimal policy.
>> Okay.