SLM Lab
Search…
Run Benchmark: A2C on Atari Games

Spec Params for A2C on Atari Games

Benchmark results for an algorithm requires running it for a number of environments. This can easily be done in SLM Lab by parametrizing the spec file, which is similar to how it was done in Experiment and Search Spec: PPO on Breakout. Running benchmark in SLM Lab is easy by using spec_params in the spec file, which uses the same underlying function as experiment search.
Let's run a benchmark for A2C on 4 Atari environments. We can look at an example spec from slm_lab/spec/benchmark/a2c/a2c_gae_atari.json
slm_lab/spec/benchmark/a2c/a2c_gae_atari.json
1
{
2
"a2c_gae_atari": {
3
"agent": [{
4
"name": "A2C",
5
"algorithm": {
6
"name": "ActorCritic",
7
...
8
},
9
...
10
}
11
}],
12
"env": [{
13
"name": "${env}",
14
"frame_op": "concat",
15
"frame_op_len": 4,
16
"reward_scale": "sign",
17
"num_envs": 16,
18
"max_t": null,
19
"max_frame": 1e7
20
}],
21
...
22
"spec_params": {
23
"env": [
24
"BreakoutNoFrameskip-v4", "PongNoFrameskip-v4", "QbertNoFrameskip-v4", "SeaquestNoFrameskip-v4"
25
]
26
}
27
}
28
}
Copied!
Spec param uses template string replacement to modify the spec and append to the spec name. Replace the value of the environment name with "${env}".

🚀
Running A2C Atari Benchmark

This benchmark will run 4 trials in total. The command to run it is familiar:
1
python run_lab.py slm_lab/spec/benchmark/a2c/a2c_gae_atari.json a2c_gae_atari train
Copied!
The spec name will be replaced with each value of the spec param, so the resulting trials will be named "a2c_gae_atari_BreakoutNoFrameskip-v4", "a2c_gae_atari_PongNoFrameskip-v4", "a2c_gae_atari_QbertNoFrameskip-v4", and "a2c_gae_atari_SeaquestNoFrameskip-v4".
All the SLM Lab benchmark results are run from files in slm_lab/spec/benchmark/.
Refer to the following pages for benchmark results in SLM Lab.