Replay

Replay

Code: master/slm_lab/agent/memory/replay.py

Experiences are stored in a circular buffer which grows until the memory has reached capacity. Once the memory is at capacity the oldest experiences are deleted to make space for the newest.

Batches of size batch_size are sampled from the entire memory. Sampling is random uniform unless "PrioritizedReplay" memory is used.

Suitable for off-policy algorithms.

Source Documentation

Refer to the class documentation and example memory spec from the source: slm_lab/agent/memory/replay.py#L43-L67

Example Memory Spec

This specification creates a Replay memory with a maximum capacity of 10,000 elements, i.e. it can store experiences for 10,000 time steps. When memory.sample() is called, it will return a batch of 32 elements. We also specify the use of CER (Combined Experience Replay), which guarantees the latest experience is included in the samples.

{
    ...
    "agent": [{
      "memory": {
        "name": "Replay",
        "batch_size": 32,
        "max_size": 10000,
        "use_cer": true
      }
    }],
    ...
}

For more concrete examples of memory spec specific to algorithms, refer to the existing spec files.