Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions docs/get-started/debugging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
# Debugging Servers

This guide shows you how to debug NeMo Gym servers using VS Code's debugger.

## Prerequisites

Make sure you have the dev dependencies installed:

```bash
pip install -e ".[dev]"
```

This includes `debugpy`, which is required for remote debugging.

## VS Code Configuration

Add these configurations to your `.vscode/launch.json`:

```json
{
"version": "0.2.0",
"configurations": [
{
"name": "📎 Attach: Server Port 5678",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5678
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "${workspaceFolder}"
}
],
"justMyCode": false
},
{
"name": "📎 Attach: Server Port 5679",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5679
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "${workspaceFolder}"
}
],
"justMyCode": false
},
{
"name": "📎 Attach: Server Port 5680",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5680
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "${workspaceFolder}"
}
],
"justMyCode": false
},
{
"name": "📎 Attach: Server Port 5681",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5681
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "${workspaceFolder}"
}
],
"justMyCode": false
}
]
}
```

:::{note}
If you already have a `launch.json`, just add these configurations to your existing `"configurations"` array.
:::

## Debugging Workflow

### Step 1: Start Servers with Debug Mode

In your terminal, enable debug mode and start your servers:

```bash
# Enable subprocess debugging
export NEMO_GYM_DEBUG_SUBPROCESS=1
export NEMO_GYM_DEBUG_PORT=5678

# Start your servers (adjust config paths as needed)
ng_run "+config_paths=[resources_servers/example_simple_weather/configs/simple_weather.yaml,responses_api_models/openai_model/configs/openai_model.yaml]"
```

You'll see output like:
```
🐛 Subprocess debug mode enabled - servers will use ports starting from 5678
[openai_model] listening for debugger on port 5678
[weather_server] listening for debugger on port 5679
```

The servers are now running and waiting for debugger attachment.

### Step 2: Attach VS Code Debugger

1. **Set your breakpoints** in the server code (e.g., `responses_api_models/openai_model/app.py`)

2. **Attach the debugger:**
- Press `F5` or `Ctrl+Shift+D` (Cmd+Shift+D on Mac)
- Select **"📎 Attach: Server Port 5678"** (for the first server)
- Or select the port corresponding to the server you want to debug

3. You'll see the debugger attach successfully in VS Code's debug panel.

### Step 3: Trigger Your Code

Run your client or test to trigger requests to the servers:

```bash
python responses_api_agents/simple_agent/client.py
```

Your breakpoints will now hit! 🎉

## Quick Reference

### Environment Variables

| Variable | Description |
|----------|-------------|
| `NEMO_GYM_DEBUG_SUBPROCESS` | Set to `1` to enable debug mode for subprocesses |
| `NEMO_GYM_DEBUG_PORT` | Base port number (default: 5678). Each server gets sequential ports |

### Port Assignment

When debug mode is enabled, servers are assigned sequential ports:
- First server: 5678 (base port)
- Second server: 5679 (base + 1)
- Third server: 5680 (base + 2)
- And so on...

### Debugging Multiple Servers

You can attach to multiple servers simultaneously:

1. Attach to port 5678 (press F5 → select port 5678)
2. Attach to port 5679 (press F5 again → select port 5679)
3. Both servers are now being debugged!

VS Code will show multiple debug sessions in the Call Stack panel.

## Troubleshooting

### "Failed to attach"

- Make sure servers are running with `ng_run`
- Check the port numbers in the terminal output
- Verify `NEMO_GYM_DEBUG_SUBPROCESS=1` is set

### Breakpoints not hitting

- Verify you attached to the correct port
- Make sure you're triggering the right endpoint
- Check that the request is actually reaching the server
- Ensure breakpoints are set before the code executes

### "debugpy not found"

Install dev dependencies:
```bash
pip install -e ".[dev]"
```

## How It Works

When `NEMO_GYM_DEBUG_SUBPROCESS=1` is set, the CLI automatically starts each server subprocess with:

```bash
python -m debugpy --listen 0.0.0.0:PORT app.py
```

This enables remote debugging without modifying your application code. VS Code connects to these debug ports using the attach configurations.

11 changes: 10 additions & 1 deletion docs/get-started/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Make sure you have these prerequisites ready before beginning the tutorials:

## Tutorial Path

Follow these four tutorials in sequence to build your first AI agent from scratch:
Follow these tutorials to get started with NeMo Gym:

::::{grid} 1 1 1 1
:gutter: 3
Expand All @@ -46,6 +46,15 @@ Generate your first batch of rollouts and understand how they become training da
{bdg-secondary}`training-data` {bdg-secondary}`scale`
:::

:::{grid-item-card} {octicon}`bug;1.5em;sd-mr-1` 3. Debugging Servers
:link: debugging
:link-type: doc

Learn how to debug NeMo Gym servers using VS Code's debugger.
+++
{bdg-secondary}`development` {bdg-secondary}`debugging`
:::

::::

---
Expand Down
10 changes: 10 additions & 0 deletions nemo_gym/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
"""
Entry point for running nemo_gym as a module: python -m nemo_gym

This allows debugging with: python -m nemo_gym
"""

if __name__ == "__main__":
from nemo_gym.cli import run
run()

18 changes: 17 additions & 1 deletion nemo_gym/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,13 @@ def start(self, global_config_dict_parser_config: GlobalConfigDictParserConfig)
self._processes: Dict[str, Popen] = dict()
self._server_instance_display_configs: List[ServerInstanceDisplayConfig] = []

# Check if debug mode is enabled for subprocess debugging
debug_enabled = environ.get("NEMO_GYM_DEBUG_SUBPROCESS", "").lower() in ("1", "true", "yes")
if debug_enabled:
debug_base_port = int(environ.get("NEMO_GYM_DEBUG_PORT", "5678"))
server_index = 0
print(f"🐛 Subprocess debug mode enabled - servers will use ports starting from {debug_base_port}")

# TODO there is a better way to resolve this that uses nemo_gym/global_config.py::ServerInstanceConfig
for top_level_path in top_level_paths:
server_config_dict = global_config_dict[top_level_path]
Expand All @@ -161,10 +168,19 @@ def start(self, global_config_dict_parser_config: GlobalConfigDictParserConfig)

dir_path = PARENT_DIR / Path(first_key, second_key)

# Set up debugging for this subprocess if enabled
if debug_enabled:
debug_port = debug_base_port + server_index
python_cmd = f"python -m debugpy --listen 0.0.0.0:{debug_port}"
print(f" [{top_level_path}] listening for debugger on port {debug_port}")
server_index += 1
else:
python_cmd = "python"

command = f"""{_setup_env_command(dir_path, global_config_dict)} \\
&& {NEMO_GYM_CONFIG_DICT_ENV_VAR_NAME}={escaped_config_dict_yaml_str} \\
{NEMO_GYM_CONFIG_PATH_ENV_VAR_NAME}={shlex.quote(top_level_path)} \\
python {str(entrypoint_fpath)}"""
{python_cmd} {str(entrypoint_fpath)}"""

process = _run_command(command, dir_path)
self._processes[top_level_path] = process
Expand Down
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,11 @@ dev = [
# Updated: Fri Jul 25, 2025 and requests-mock==1.12.1
# License: Apache 2.0 https://github.com/jamielennox/requests-mock/blob/9742d02a8cad17276dbeba7b300b6d27ae1b6fb1/LICENSE
"requests-mock",

# debugpy: Used for VS Code remote debugging of server processes
# Updated: Nov 16, 2025 with debugpy>=1.8.0
# License: MIT https://github.com/microsoft/debugpy/blob/main/LICENSE
"debugpy>=1.8.0",
]

[build-system]
Expand Down
Loading