Experiment and Search Spec: PPO on Breakout

​📂
The Search Spec
In this tutorial, we will learn how to run an experiment to study the following example question:
What values of lambda of PPO provide the fastest, most stable solution for Atari Breakout, if the other variables are held constant?
In SLM Lab, we can easily run experiments to answer questions about deep RL. An Experiment in SLM Lab runs a number of Trials using a search spec by generating different sets of hyperparameters to search over (using Ray Tune) and running a Trial for each one. 
The search spec has the following format:
{
  "{spec_name}": {
    "agent": [{...}],
    "env": [{...}],
    ...
    "meta": {
      ...
      "max_trial": int,
    },
    "search": {
      *spec
    }
  }
}
That is, the search spec can contain any components of a spec file. To perform search over a spec variable, simply:
mirror the spec for that variable
append a search mode to the variable key
define the config space for the search mode
Essentially, search spec defines the config space using "{key}__{space_type}": {v}, where {space_type} is grid_search of ray.tune, or any function name of np.random:
space_type
v
v type
grid_search
[value1, value2, ...]
 str|int|float
choice
[value1, value2, ...]
str|int|float
randint
[low, high)
int
uniform
[low, high)
float
normal
[low, high)
float
For example:
"explore_anneal_epi__randint": [10, 60] will sample integers uniformly from 10 to 60 for explore_anneal_epi
"lr__uniform": [0.001, 0.1] will sample lr using np.random.uniform(0.001, 0.1)
When constructing a new Trial, an Experiment samples an instance from the config space, then updates the original spec with the sampled values before passing it to the Trial constructor.
By default, an Experiment will run search for as many Trials as specified by "max_trial" in meta spec using Random sampling from the full config space. If any key uses grid_search, it will be combined exhaustively in combination with other random sampling, e.g. for max_trial = 1 with one grid search of 4 elements, this will yield 4 x 1 = 1 total trials.
​✍
 Search Spec for PPO
As an example, let's try to answer the question:
What values of lambda of PPO provide the fastest, most stable solution for Atari Breakout, if the other variables are held constant?
Let's look at the search spec for PPO on Breakout from slm_lab/spec/experimental/ppo/ppo_lam_search.json.
slm_lab/spec/experimental/ppo/ppo_lam_search.json
{
  "ppo_breakout": {
    "agent": [{
      "name": "PPO",
      "algorithm": {
        "name": "PPO",
        "action_pdtype": "default",
        "action_policy": "default",
        "explore_var_spec": null,
        "gamma": 0.99,
        "lam": 0.70,
        ...
      },
      ...
    }],
    "env": [{
      "name": "BreakoutNoFrameskip-v4",
      "frame_op": "concat",
      "frame_op_len": 4,
      "reward_scale": "sign",
      "num_envs": 16,
      "max_t": null,
      "max_frame": 1e7
    }],
    ...
    "meta": {
      "distributed": false,
      "log_frequency": 10000,
      "eval_frequency": 10000,
      "max_session": 4,
      "max_trial": 1,
    },
    "search": {
      "agent": [{
        "algorithm": {
          "lam__grid_search": [0.50, 0.70, 0.90, 0.95, 0.97, 0.99]
        }
      }]
    }
  }
}
This file defines the spec for PPO and Breakout as usual. Corresponding to the question, we are interested in finding out the effect of different values of agent[0].algorithm.lam. The search spec specifies a grid search over it, and we set "meta.max_trial" to 1 since we are only doing a grid search.
​🚀
 Running a PPO Search on Breakout
Let's run an Experiment using the spec file above by using the search lab mode:
python run_lab.py slm_lab/spec/experimental/ppo/ppo_lam_search.json ppo_breakout search
This will spawn 6 trials in queue using Ray Tune, which are then dequeued to run as computing resources free up. Since we specify a trial to run 4 sessions, it will take up 4 CPUs and 4 GPUs. If we run this on a machine with 32 CPUs and 8 GPUs, the experiment will run 2 trials at any given time.
At the end of the experiment, we will obtain the usual trial graphs. Additionally, it will also produce an experiment graph, as shown below:
Just as how we can plot the moving average version of a trial graph, we can do the same for experiment graph:
From the experiment graph, we can observe that trial 1 (red) with lam: 0.7 performs the best on Breakout with the fastest convergence and the best final result.

🚀Using SLM Lab - Previous

Parallelizing Training: Async SAC on Humanoid

Next - 🚀Using SLM Lab

Run Benchmark: A2C on Atari Games

Last modified 7mo ago

Copy link

Outline

The Search Spec

Search Spec for PPO

Running a PPO Search on Breakout

​📂The Search Spec

​✍ Search Spec for PPO

​🚀 Running a PPO Search on Breakout

📂
The Search Spec

✍
Search Spec for PPO

🚀
Running a PPO Search on Breakout