SLM Lab
v4.1.1
v4.1.1
  • SLM Lab
  • 🖥Setup
    • Installation
    • Quick Start
  • 🚀Using SLM Lab
    • Lab Command
    • Lab Organization
    • Train and Enjoy: REINFORCE CartPole
    • Agent Spec: DDQN+PER on LunarLander
    • Env Spec: A2C on Pong
    • GPU Usage: PPO on Pong
    • Parallelizing Training: Async SAC on Humanoid
    • Experiment and Search Spec: PPO on Breakout
    • Run Benchmark: A2C on Atari Games
    • Meta Spec: High Level Specifications
    • Post-Hoc Analysis
    • TensorBoard: Visualizing Models and Actions
    • Using SLM Lab In Your Project
  • 📈Analyzing Results
    • Data Locations
    • Graphs and Data
    • Performance Metrics
  • 🥇Benchmark Results
    • Public Benchmark Data
    • Discrete Environment Benchmark
    • Continuous Environment Benchmark
    • Atari Environment Benchmark
    • RL GIFs
  • 🔧Development
    • Modular Design
      • Algorithm Taxonomy
      • Class Inheritance: A2C > PPO
    • Algorithm
      • DQN
      • REINFORCE
      • Actor Critic
    • Memory
      • Replay
      • PrioritizedReplay
      • OnPolicyReplay
      • OnPolicyBatchReplay
    • Net
      • MLP
      • CNN
      • RNN
    • Profiling SLM Lab
  • 📖Publications and Talks
    • Book: Foundations of Deep Reinforcement Learning
    • Talks and Presentations
  • 🤓Resources
    • Deep RL Resources
    • Contributing
    • Motivation
    • Help
    • Contact
Powered by GitBook
On this page
  • Convolutional Neural Network
  • Source Documentation
  • Example Net Spec

Was this helpful?

  1. 🔧Development
  2. Net

CNN

PreviousMLPNextRNN

Last updated 5 years ago

Was this helpful?

Convolutional Neural Network

Code:

These networks take a single state as input and produce one or more outputs. It consists of zero or more convolutional layers, followed by zero or more dense layers. Finally, there are one or more dense output layers. CNNs excel at image processing and processing inputs with 2D or 3D structure. This makes them well suited for environments with pixel level inputs or inputs with a spatial component.

For more information on each of the elements see this on CNNs or the

Source Documentation

Refer to the class documentation and example net spec from the source:

Example Net Spec

This specification instantiates a typical CNN for deep RL with 3 convolutional layers (input channels are automatically inferred):

  • 32 output channels, kernel size (8 x 8), stride of 4, padding of 0, and dilation of (1 x 1)

  • 64 output channels, kernel size (4 x 4), stride of 2, padding of 0, and dilation of (1 x 1)

  • 32 output channels, kernel size (3 x 3), stride of 1, padding of 0, and dilation of (1 x 1)

Then, the output of the last convolutional layer is flattened and connected to the fully-connected hidden layers (this can be empty):

  • a layer of 512 units, then to a layer of 256 units

Lastly, the final layer is connected to an output layer that is automatically constructed by inferring the the algorithm and the environment.

The network uses linear (ReLU) activations, no activation for the output layer, and no weight initialization function. The rest of the spec is annotated below.

{
    ...
    "agent": [{
      ...
      "net": {
        "type": "ConvNet",
        "shared": false,  // whether to shared networks for Actor-Critic
        "conv_hid_layers": [
            [32, 8, 4, 0, 1],
            [64, 4, 2, 0, 1],
            [32, 3, 1, 0, 1]
        ],
        "fc_hid_layers": [512, 256],
        "hid_layers_activation": "relu",
        "out_layer_activation": null,
        "init_fn": null,  // weight initialization
        "normalize": false,  // whether to divide input by 255.0
        "batch_norm": false,  // whether to add batchnorm layers
        "clip_grad_val": 1.0,  // clip gradient by norm
        "loss_spec": {
          "name": "SmoothL1Loss"  // default loss function used for regression
        },
        "optim_spec": {  // the optimizer and its arguments
          "name": "Adam",
          "lr": 0.02
        },
        ...
      }
    }],
    ...
}

For more concrete examples of net spec specific to algorithms, refer to the existing .

slm_lab/agent/net/conv.py
tutorial
PyTorch documentation
slm_lab/agent/net/conv.py#L11-L75
spec files