Comment on page
Starting with random parameter values, the agent uses this policy to act in an environment and receive rewards. After an episode has finished, the "goodness" of each action, represented by,
, is calculated using the episode trajectory. The parameters of the policy are then updated in a direction which makes good actions
more likely, and bad actions
less likely. Good actions are reinforced, bad actions are discouraged.
The agent then uses the updated policy to act in the environment, and the training process repeats.
REINFORCE is an on policy algorithm. Only data that is gathered using the current policy can be used to update the parameters. Once the policy parameters have been updated all previous data gathered must be discarded and the collection process started again with the new policy.
There are a number of different approaches to calculating
. Method 3, outlined below, is common. It captures the idea that the absolute quality of the actions matters less than their quality relative to some baseline. One option for a baseline is the average of
over the training data (typically one episode trajectory).
Algorithm: REINFORCE with baseline
Methods for calculating
action_policystring specifying which policy to use to act. For example, "Categorical" (for discrete action spaces), "Normal" (for continuous actions spaces with one dimension), or "default" to automatically switch between the two depending on the environment.
training_frequencyhow many episodes of data to collect before each training iteration. A common value is 1.
entropywhether to add entropy to theto encourage exploration
entropy_coefcoefficient to multiply the entropy of the distribution with when adding it to