Run RLE-Bench

Install, evaluate, and inspect results with the RLE-Bench CLI.

01Install

We test RLE-Bench on Linux with Git, Make, uv, and Docker. GPU tasks also require an NVIDIA GPU accessible from Docker.

Shell
git clone https://github.com/RLE-Bench/RLE-Bench.git
cd RLE-Bench
make install
source .venv/bin/activate

02Check your setup

List available tasks and check for missing dependencies.

Shell
rlebench list
rlebench doctor

03Run an evaluation

Prepare Mobile Base Design assets and images. No GPU required.

Shell
rlebench prepare task08

Choose your agent and credentials below. Replace MODEL_NAME with a model available to your account.

Anthropic · Subscription

Use a Claude Code subscription OAuth token, supplied as CLAUDE_CODE_OAUTH_TOKEN. This is different from ANTHROPIC_AUTH_TOKEN.

Authentication
export CLAUDE_CODE_OAUTH_TOKEN="YOUR_OAUTH_TOKEN"
Run
rlebench run task08 -a claude-code \
  -e anthropic/claude-code -m "anthropic/MODEL_NAME"
Anthropic · API

Use an Anthropic API key. ANTHROPIC_AUTH_TOKEN is also supported by this API configuration; use one credential variable.

Authentication
export ANTHROPIC_API_KEY="YOUR_API_KEY"
Run
rlebench run task08 -a claude-code \
  -e anthropic/api -m "anthropic/MODEL_NAME"
Codex · Subscription

Sign in with ChatGPT using file-based credential storage. The benchmark uses ~/.codex/auth.json.

Authentication
codex -c 'cli_auth_credentials_store="file"' login
export CODEX_FORCE_AUTH_JSON=1
Run
rlebench run task08 -a codex \
  -e openai/codex -m "openai/MODEL_NAME"
Codex · API

Run the Codex agent with an OpenAI API key.

Authentication
export OPENAI_API_KEY="YOUR_API_KEY"
Run
rlebench run task08 -a codex \
  -e openai/api -m "openai/MODEL_NAME"
Gemini · API

Use GEMINI_API_KEY to run Gemini on task09. Prepare the task first with rlebench prepare task09.

Authentication
export GEMINI_API_KEY="YOUR_API_KEY"
Run
rlebench run task09 -a agy \
  -e gemini/api -m gemini-3.7-flash
GLM · API with Claude Code

Run GLM through Claude Code using ZAI_API_KEY (or GLM_API_KEY). Use zai/api-cn for the China endpoint.

Authentication
export ZAI_API_KEY="YOUR_API_KEY"
Run
rlebench run task08 -a claude-code \
  -e zai/api -m "zai/MODEL_NAME"
GPU example

Prepare the Interactive Control task family.

Shell
rlebench prepare task01

With Anthropic API credentials configured, run the open-fridge task on GPU 0.

Shell
rlebench run task01/L1/01-open-fridge \
  -a claude-code -e anthropic/api -m "anthropic/MODEL_NAME" --device cuda:0

Use rlebench run --help for agents and options. Check task docs for required datasets and weights.

04Bring your own agent

You can use any Harbor-supported agent and model. Choose the agent with -a and the model with -m, then add Harbor configuration after --. RLE-Bench forwards everything after -- directly to Harbor.

Shell
rlebench run task08 -a AGENT_NAME -m "PROVIDER/MODEL_NAME" \
  -- --ak "OPTION=VALUE"

Replace the placeholders with your agent, model, and supported agent options. For agents without a built-in endpoint configuration, omit -e and supply credentials and allowed API hosts through Harbor as needed. Run harbor run --help for available options.

05View results

Generate reports with run statuses, rewards, and costs.

Shell
rlebench summarize jobs

Open the viewer to explore trajectories, scores, and recordings.

Shell
rlebench view jobs