Chapter 2 REINFORCE

Page 26, Section 2.2 The Objective Function, Equation 2.1

Equation 2.1 misplaces the prime symbol due to a Latex formatting error. It was

Rt(τ)=t=tTγttrt      (misplaced prime on r)R_t(\tau) = \sum_{t'=t}^T \gamma^{t'-t} r_t' \ \ \ \ \ \ (\text{misplaced prime on } r')

Instead it should have been

Rt(τ)=t=tTγttrtR_t(\tau) = \sum_{t'=t}^T \gamma^{t'-t} r_{t'}

Page 27, Section 2.3 The Policy Gradient, Equation 2.3

Equation 2.3 contains a typo. Following from equation 2.2, the max argument should be applied on both side of the equation. It was,

maxθJ(πθ)=Eτπθ[R(τ)]      (missing max on the right)\max\limits_\theta \textit{J}(\pi_\theta) = \mathbb{E}_{\tau\thicksim\pi_\theta} [\textit{R}(\tau)] \ \ \ \ \ \ (\text{missing max on the right})

Instead it should have been

maxθJ(πθ)=maxθEτπθ[R(τ)]\max\limits_\theta \textit{J}(\pi_\theta) = \max\limits_\theta \mathbb{E}_{\tau\thicksim\pi_\theta} [\textit{R}(\tau)]

Page 28, Section 2.3.1 Policy Gradient Derivation, Equation 2.9

In the chain of derivation, equation (2.9) states that the step used is (chain-rule), but in fact it is (product-rule).

Page 30, Section 2.3.1 Policy Gradient Derivation, Equation 2.21

Equation 2.21 misses a step in derivation:

By substituting Equation 2.20 into 2.15 and bringing in the multiplier R(τ), we obtain

θJ(πθ)=Eτπθ[t=0TRt(τ)θlogπθ(atst)]      (2.21)\nabla_\theta J(\pi_\theta) = E_{\tau \sim \pi_\theta} \big[\sum_{t=0}^T R_t(\tau) \nabla_\theta \log \pi_\theta(a_t|s_t) \big] \ \ \ \ \ \ (2.21)

Actually, the substitution yields:

θJ(πθ)=Eτπθ[t=0TR(τ)θlogπθ(atst)]      (missing step 1)\nabla_\theta J(\pi_\theta) = E_{\tau \sim \pi_\theta} \big[\sum_{t=0}^T R(\tau) \nabla_\theta \log \pi_\theta(a_t|s_t) \big] \ \ \ \ \ \ (\text{missing step 1})

There is an additional step which modifies R(τ) to give us equation 2.21. The form above has a high variance due to the many possible actions over a trajectory. One way to reduce the variance is to account for causality by only considering the future rewards for any given time step t. This makes sense since event occurring at time step t can only affect the future but not the past. To do so, we modify R(τ) as follows:

R(τ)=R0(τ)=t=0Tγtrtt=tTγttrt=Rt(τ)      (missing step 2)R(\tau) = R_0(\tau) = \sum_{t'=0}^T \gamma^{t'} r_{t'} \rightarrow \sum_{t'=t}^T \gamma^{t'-t} r_{t'} = R_t(\tau) \ \ \ \ \ \ (\text{missing step 2})

With this, we obtain equation 2.21.

Last updated

Was this helpful?