🌍Env Spec

This tutorial shows how to configure the env spec for continuous control environments. We'll train PPO on HalfCheetah—a MuJoCo locomotion task.

The Env Spec

The environment is specified using the env key in a spec file:

{
  "spec_name": {
    "agent": {...},
    "env": {
      // Environment name (must be in gymnasium registry)
      "name": str,

      // Number of parallel environment instances
      "num_envs": int,

      // Maximum timesteps per episode (null = use environment default)
      "max_t": int|null,

      // Total training frames
      "max_frame": int,

      // Optional: Online state normalization (recommended for MuJoCo)
      "normalize_obs": bool,

      // Optional: Online reward normalization (recommended for MuJoCo)
      "normalize_reward": bool,

      // Optional: Clip observations to [-bound, bound] (default: 10.0 if normalize_obs)
      "clip_obs": float,

      // Optional: Clip rewards to [-bound, bound] (default: 10.0 if normalize_reward)
      "clip_reward": float
    },
    ...
  }
}

Supported Environments

SLM Lab uses Gymnasium (the maintained fork of OpenAI Gym):

Category

Examples

Difficulty

Docs

Classic Control

CartPole, Pendulum, Acrobot

Easy

Gymnasium Classic

Box2D

LunarLander, BipedalWalker

Medium

Gymnasium Box2D

MuJoCo

Hopper, HalfCheetah, Humanoid

Hard

Gymnasium MuJoCo

Atari

Qbert, MsPacman, and 54 more

Varied

ALE

Any gymnasium-compatible environment works—just specify its name in the spec.

Environment-Specific Settings

Category

num_envs

max_frame

Normalization

GPU

Classic Control

2e5-3e5

Optional

Box2D

3e5

Optional

MuJoCo

1e6-10e6

normalize_obs, normalize_reward

Optional

Atari

10e6

Recommended

See Benchmark Specs for complete spec files for each environment.

Example: PPO on HalfCheetah

HalfCheetah-v5 is a classic MuJoCo benchmark—a 2D cheetah robot that learns to run forward. It has a 17-dimensional observation space and 6-dimensional continuous action space.

The PPO MuJoCo spec from slm_lab/spec/benchmark/ppo/ppo_mujoco.json:

slm_lab/spec/benchmark/ppo/ppo_mujoco.json (excerpt)

{
  "ppo_mujoco": {
    "agent": {
      "name": "PPO",
      "algorithm": {
        "name": "PPO",
        "gamma": 0.99,
        "lam": 0.95,
        "time_horizon": 2048,
        "minibatch_size": 64,
        "training_epoch": 10,
        "normalize_v_targets": true
      },
      "memory": {"name": "OnPolicyBatchReplay"},
      "net": {
        "type": "MLPNet",
        "hid_layers": [256, 256],
        "hid_layers_activation": "tanh",
        "init_fn": "orthogonal_",
        "gpu": "auto"
      }
    },
    "env": {
      "name": "${env}",
      "num_envs": 16,
      "max_frame": "${max_frame}",
      "normalize_obs": true,
      "normalize_reward": true
    },
    "meta": {
      "max_session": 4,
      "max_trial": 1,
      "log_frequency": 10000
    }
  }
}

Key env settings for MuJoCo:

Parameter

Value

Why

num_envs: 16

16 parallel environments

Faster data collection for on-policy learning

normalize_obs: true

Normalize observations

MuJoCo observations have varying scales

normalize_reward: true

Normalize rewards

Stabilizes value function learning

MuJoCo became free in 2022. No license needed—Gymnasium includes MuJoCo out of the box.

Running PPO on HalfCheetah

# Dev mode (quick test with rendering)
slm-lab run -s env=HalfCheetah-v5 -s max_frame=1e5 slm_lab/spec/benchmark/ppo/ppo_mujoco.json ppo_mujoco dev

# Full training (4M frames)
slm-lab run -s env=HalfCheetah-v5 -s max_frame=4e6 slm_lab/spec/benchmark/ppo/ppo_mujoco.json ppo_mujoco train

The variable substitution (-s env=...) lets you use the same spec for different MuJoCo environments.

Results

PPO achieves 5852 MA on HalfCheetah-v5 with this configuration.

Training curves (average of 4 sessions):

Trained models available on HuggingFace.

Using Other Environments

To use a different environment, find a spec for that environment category and modify env.name. Spec files are organized by algorithm in slm_lab/spec/benchmark/:

Environment Categories

Category

Environments

Spec Examples

Classic Control

CartPole-v1, Acrobot-v1, Pendulum-v1, MountainCar-v0

ppo_cartpole.json, dqn_cartpole.json

Box2D

LunarLander-v3, BipedalWalker-v3

ppo_lunar.json, ddqn_per_lunar.json

MuJoCo

Hopper-v5, HalfCheetah-v5, Walker2d-v5, Ant-v5, Humanoid-v5, Swimmer-v5, etc.

ppo_mujoco.json, sac_mujoco.json

Atari

54 games (ALE/Qbert-v5, ALE/MsPacman-v5, etc.)

ppo_atari.json, dqn_atari.json

Switching Environments

Find a spec for your target environment category
Change env.name to the Gymnasium environment name
Adjust settings as needed (num_envs, max_frame, normalization)

Example—use the same algorithm on different environments:

# PPO on CartPole (Classic Control)
slm-lab run slm_lab/spec/benchmark/ppo/ppo_cartpole.json ppo_cartpole train

# PPO on LunarLander (Box2D)
slm-lab run slm_lab/spec/benchmark/ppo/ppo_lunar.json ppo_lunar train

# PPO on HalfCheetah (MuJoCo) - uses variable substitution
slm-lab run -s env=HalfCheetah-v5 -s max_frame=4e6 slm_lab/spec/benchmark/ppo/ppo_mujoco.json ppo_mujoco train

# PPO on Breakout (Atari) - uses variable substitution
slm-lab run -s env=ALE/Breakout-v5 slm_lab/spec/benchmark/ppo/ppo_atari.json ppo_atari train

Template Specs with Variable Substitution

Some specs use ${var} placeholders for flexibility. Use -s var=value to substitute:

# MuJoCo template - works for any MuJoCo environment
slm-lab run -s env=Hopper-v5 -s max_frame=2e6 slm_lab/spec/benchmark/ppo/ppo_mujoco.json ppo_mujoco train
slm-lab run -s env=Walker2d-v5 -s max_frame=5e6 slm_lab/spec/benchmark/ppo/ppo_mujoco.json ppo_mujoco train

# Atari template - works for any ALE game
slm-lab run -s env=ALE/Qbert-v5 slm_lab/spec/benchmark/ppo/ppo_atari.json ppo_atari train
slm-lab run -s env=ALE/MsPacman-v5 slm_lab/spec/benchmark/ppo/ppo_atari.json ppo_atari train

Finding Environment Specs

# List all benchmark specs
ls slm_lab/spec/benchmark/

# Find specs for a specific environment
grep -r "CartPole" slm_lab/spec/benchmark/
grep -r "LunarLander" slm_lab/spec/benchmark/
grep -r "HalfCheetah" slm_lab/spec/benchmark/

Any Gymnasium environment works. Just set env.name to a valid Gymnasium environment ID. Use the benchmark specs as starting points for hyperparameters.

Standard Settings for Fair Comparison

When comparing algorithms, use consistent environment settings. Different num_envs or max_frame values make comparisons invalid.

Recommended Settings by Category

Category

num_envs

max_frame

log_frequency

Notes

Classic Control

2e5-3e5

500

Fast training

Box2D

3e5

1000

Medium complexity

MuJoCo

4e6-10e6

10000

Use normalization

Atari

10e6

10000

GPU recommended

What to Keep Consistent

When comparing algorithms on the same environment:

Parameter

Keep Same?

Why

num_envs

Yes

Affects data collection rate and batch statistics

max_frame

Yes

Total training budget must match

max_t

Yes

Episode length affects learning signal

normalize_obs

Yes

Changes observation distribution

normalize_reward

Yes

Changes reward scale

Example: Fair Algorithm Comparison

To compare DQN vs PPO on LunarLander fairly:

# Both use: num_envs=8, max_frame=3e5, max_session=4
slm-lab run slm_lab/spec/benchmark/dqn/dqn_lunar.json dqn_concat_lunar train
slm-lab run slm_lab/spec/benchmark/ppo/ppo_lunar.json ppo_lunar train

Check that both specs have matching env settings before comparing results.

Benchmark specs are pre-configured. The specs in slm_lab/spec/benchmark/ use standardized settings for each environment category. When creating custom specs, match these settings for comparable results.

Advanced Env Options

Environment Kwargs

Any additional keys in the env spec are passed to gymnasium.make():

"env": {
  "name": "HalfCheetah-v5",
  "num_envs": 16,
  "max_frame": 4e6,
  "exclude_current_positions_from_observation": false  // Passed to MuJoCo
}

Normalization Details

The normalization wrappers maintain running statistics:

Option

What It Does

When to Use

normalize_obs

Centers observations, scales to unit variance

MuJoCo, continuous control

normalize_reward

Scales rewards using running std

Environments with varying reward scales

Gymnasium API: SLM Lab v5 uses Gymnasium's (obs, reward, terminated, truncated, info) return format. This correctly distinguishes task completion (terminated) from time limits (truncated)—important for proper value estimation.

Next, we'll use GPU to train on Atari games where image processing is the bottleneck.

PreviousAgent Spec NextMeta Spec

Last updated 5 days ago

Was this helpful?

hashtagThe Env Spec

hashtagSupported Environments

hashtagEnvironment-Specific Settings

hashtagExample: PPO on HalfCheetah

hashtagRunning PPO on HalfCheetah

hashtagResults

hashtagUsing Other Environments

hashtagEnvironment Categories

hashtagSwitching Environments

hashtagTemplate Specs with Variable Substitution

hashtagFinding Environment Specs

hashtagStandard Settings for Fair Comparison

hashtagRecommended Settings by Category

hashtagWhat to Keep Consistent

hashtagExample: Fair Algorithm Comparison

hashtagAdvanced Env Options

hashtagEnvironment Kwargs

hashtagNormalization Details

The Env Spec

Supported Environments

Environment-Specific Settings

Example: PPO on HalfCheetah

Running PPO on HalfCheetah

Results

Using Other Environments

Environment Categories

Switching Environments

Template Specs with Variable Substitution

Finding Environment Specs

Standard Settings for Fair Comparison

Recommended Settings by Category

What to Keep Consistent

Example: Fair Algorithm Comparison

Advanced Env Options

Environment Kwargs

Normalization Details