Q-learning (Watkins, 1989, Mnih et. al, 2013) algorithms estimate the optimal Q function, i.e the value of taking action A in state S under the optimal policy. Q-learning algorithms have an implicit policy (strategy for acting in the environment). This is typically
-greedy, in which the action with the maximum Q value is selected with probability
and a random action is taken with probability
, or boltzmann (see definition below). Random actions encourage exploration of the state space and help prevent algorithms from getting stuck in local minima.
Q-learning algorithms are off-policy algorithms because the target value used to train the network is independent of the policy used to generate the training data. This makes it possible to use experience replay to train an agent.
It is bootstrapped algorithm; updates to the Q function are based on existing estimates, and a temporal difference algorithm; the estimate in time
tis updated using an estimate from time
t+1. This allows Q-Learning algorithms to be online and incremental, so the agent can be trained during an episode.
action_policystring specifying which policy to use to act. "boltzmann" or "epsilon_greedy".
- "boltzmann" policy selects actions by sampling from a probability distribution over the actions. This is generated by taking a softmax over all the Q-values (estimated by a neural network) for a state, adjusted by the temperature parameter, tau.
- "epsilon_greedy" policy selects a random action with probability epsilon, and the action corresponding to the maximum Q-value with (1 - epsilon).
explore_var_startinitial value for the exploration parameters (tau or epsilon)
explore_var_endend value for the exploration parameters (tau or epsilon)
explore_anneal_epihow many episodes to take to reduce the exploration parameter value from start to end. Reduction is currently linear.
training_batch_epochhow many gradient updates to make per batch.
training_epochhow many batches to sample from the replay memory each time the agent is trained
training_frequencyhow often to train the algorithm. Value of 3 means train every 3 steps the agent takes in the environment.
batch_sizehow many examples to include in each batch when sampling from the replay memory.
max_sizemaximum size of the memory. Once the memory has reached maximum capacity, the oldest examples are deleted to make space for new examples.
training_min_timestephow many time steps to wait before starting to train. It can be useful to set this to 0.5 - 1x the batch size so that the
DQNhas a few examples to learn from in the first training iterations.
action_policy_updatehow to update the
explore_varparameter in the action policy each episode. Available options are "linear_decay", "rate_decay", and "periodic_decay". See policy_util.py for more details.
update_typemethod of updating
target_net. "replace" or "polyak". "replace" replaces
update_frequencytime steps. "polyak" updates
target_net+ (1 -
neteach time step.
update_frequencyhow often to update
netwhen using "replace"
polyak_weighthow much weight to give the old
target_netwhen updating the