Everyone and welcome back to this class deep reinforcement learning deep learning in Python part 7
in this lecture we're going to review approximation methods we study dynamic programming Monte-Carlo
and temporal difference learning in the context of tabular methods.
In other words we treated VNU like a dictionary indexed by state and action.
This works fine but only for the smallest of problems.
It's not going to work for games with billions or trillions of states and it certainly won't work for
games with an infinite number of states.
This is where approximation methods enter the picture.
We know that supervised learning can be used for function approximation.
Specifically we are interested in approximating vear Q.
We can do this by converting each state s into a feature vector and then treating the return as the
target.
Since the rewards and hence the return are real valued numbers we are doing a regression.
And as you know the appropriate function for regression is squared error.
In this equation theta refers to the parameters of our prediction model.
If you've taken deep learning this sort of thing should look very familiar to you.
In the previous course we use linear models for function approximation.
Recall that the only requirement is that the function approximated is differentiable so that we can
use gradient descent as you know by now.
The gradient takes this form and the parameter we want to update is theta.
In general though Theta can be the parameters of a neural network as well.
And all of the techniques we've learned about neural networks can be applied.
That includes batch at descent using momentum to speed up gradient descent using dropout and other types
of regularization and so on.
And of course since the gradient can get quite complicated we don't want to calculate it ourselves.
We can use libraries like piano intensive flow to do automatic differentiation.