Chapter 2 REINFORCE

Page 26, Section 2.2 The Objective Function, Equation 2.1

Equation 2.1 misplaces the prime symbol due to a Latex formatting error. It was

$$R_t(\tau) = \sum_{t'=t}^T \gamma^{t'-t} r_t' \ \ \ \ \ \ (\text{misplaced prime on } r')$$

Instead it should have been

$$R_t(\tau) = \sum_{t'=t}^T \gamma^{t'-t} r_{t'}$$

Page 27, Section 2.3 The Policy Gradient, Equation 2.3

Equation 2.3 contains a typo. Following from equation 2.2, the max argument should be applied on both side of the equation. It was,

$$\max\limits_\theta \textit{J}(\pi_\theta) = \mathbb{E}_{\tau\thicksim\pi_\theta} [\textit{R}(\tau)] \ \ \ \ \ \ (\text{missing max on the right})$$

Instead it should have been

$$\max\limits_\theta \textit{J}(\pi_\theta) = \max\limits_\theta \mathbb{E}_{\tau\thicksim\pi_\theta} [\textit{R}(\tau)]$$

Page 28, Section 2.3.1 Policy Gradient Derivation, Equation 2.9

In the chain of derivation, equation (2.9) states that the step used is (chain-rule), but in fact it is (product-rule).

Page 30, Section 2.3.1 Policy Gradient Derivation, Equation 2.21

Equation 2.21 misses a step in derivation:

By substituting Equation 2.20 into 2.15 and bringing in the multiplier R(τ), we obtain

$$\nabla_\theta J(\pi_\theta) = E_{\tau \sim \pi_\theta} \big[\sum_{t=0}^T R_t(\tau) \nabla_\theta \log \pi_\theta(a_t|s_t) \big] \ \ \ \ \ \ (2.21)$$

Actually, the substitution yields:

$$\nabla_\theta J(\pi_\theta) = E_{\tau \sim \pi_\theta} \big[\sum_{t=0}^T R(\tau) \nabla_\theta \log \pi_\theta(a_t|s_t) \big] \ \ \ \ \ \ (\text{missing step 1})$$

There is an additional step which modifies R(τ) to give us equation 2.21. The form above has a high variance due to the many possible actions over a trajectory. One way to reduce the variance is to account for causality by only considering the future rewards for any given time step t. This makes sense since event occurring at time step t can only affect the future but not the past. To do so, we modify R(τ) as follows:

$$R(\tau) = R_0(\tau) = \sum_{t'=0}^T \gamma^{t'} r_{t'} \rightarrow \sum_{t'=t}^T \gamma^{t'-t} r_{t'} = R_t(\tau) \ \ \ \ \ \ (\text{missing step 2})$$

With this, we obtain equation 2.21.