Actor Critic
There are a variety of approaches to training the critic. Three options are provided with the baseline algorithms.
Learn the V function and use it to approximate the Q function. See ac.json for some examples specs.
Advantage with n-step forward returns from Mnih et. al. 2016. See a2c.json for some example specs.
Generalized advantage estimation from Schulman et. al, 2015. See a2c.json for some example specs.
The actor and critic can be trained separately or jointly, depending on whether the networks are structured to share parameters.
Actor-Critic algorithms are on policy. Only data that is gathered using the current policy can be used to update the parameters. Once the policy parameters have been updated all previous data gathered must be discarded and the collection process started again with the new policy.
Algorithm: Simple Actor Critic with separate actor and critic parameters
See ac.json and a2c.json for example specs of variations of the Actor-Critic algorithm.
Basic Parameters
algorithm
name
general paramaction_pdtype
general paramaction_policy
string specifying which policy to use to act. "Categorical" (for discrete action spaces), "Normal" (for continuous actions spaces with one dimension), or "default" to automatically switch between the two depending on the environment.gamma
general paramuse_gae
whether to calculate the advantage using generalized advantage estimation.use_nstep
: whether to calculate the advantage using n-step forward returns. Ifuse_gae
istrue
this parameter will be ignored.num_step_returns
number of forward steps to use when calculating the target for advantage estimation using nstep forward returns.add_entropy
whether to add entropy to the advantage to encourage explorationentropy_coef
coefficient to multiply the entropy of the distribution with when adding it to advantagetraining_frequency
when using episodic data storage (memory) such as "OnPolicyReplay", this means how many episodes of data to collect before each training iteration - a common value is 1; or when using batch data storage (memory) such as "OnPolicyBatchReplay", how often to train the algorithm. Value of 32 means train every 32 steps the agent takes in the environment using the 32 examples since the agent was previously trained.
memory
name
general param. Compatible types; "OnPolicyReplay", "OnPolicyBatchReplay"
net
type
general param. All networks are compatible.shared
whether the actor and the critic should share parametershid_layers
general paramhid_layers_activation
general paramactor_optim_spec
general param optimizer for the actorcritic_optim_spec
general param optimizer for the critic
Advanced Parameters
algorithm
training_epoch
how many gradient steps to take when training the critic. Only applies when the actor and critic have separate parameters.policy_loss_coef
how much weight to give to the policy (actor) component of the loss when the actor and critic have shared parameters, so are trained jointly.val_loss_coef
how much weight to give to the critic component of the loss when the actor and critic have shared parameters, so are trained jointly.
net
use_same_optim
whether to use theoptim_actor
for both the actor and critic. This can be useful when using conducting a parameter search.rnn_hidden_size
general paramrnn_num_layers
general paramseq_len
general paramclip_grad
: general paramclip_grad_val
: general paramlr_decay
: general paramlr_decay_frequency
: general paramlr_decay_min_timestep
: general paramlr_anneal_timestep
: general paramgpu
: general param
Last updated