Skip to content

more control over hostname used when defining remote rundir #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bernstei merged 1 commit into from
Mar 20, 2025
Merged

more control over hostname used when defining remote rundir #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/source/config.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ File format
**********************************


The top level structure is a single dict with one ``"systems"`` key with a dict value, containing
The top level structure is a single dict. Most of the content is in the ``"systems"`` key, which contains
a dict value. The `"systems"` dict contains
one key-value pair for each system. The key is the name of the system (used to select
the system when evaluating the function). See ``expyre.System`` docstring for full list. The
value is a dict with the following keys:
Expand All @@ -26,6 +27,12 @@ value is a dict with the following keys:
- ``"rundir"``: string for a path where the remote jobs should be run
- ``"partitions"`` or ``"queues"``: dict with partitions/queues/node-type names as keys and dict of node properties as values.

In addition, there is an optional `"remote_rundir_submit_hostname"` which overrides the hostname used
when constructing the remote rundir, for use by people who run their scripts from different
hostnames, e.g. multiple HPC login nodes. Because the hostname is embedded into the remote rundir, if the script run that submits
the job and the script run that gathers the results are run on machines with different hostnames, the second
run will look in a different remote rundir, and not find the remote rundir.

===========================
Node property dict includes
===========================
Expand Down
3 changes: 2 additions & 1 deletion expyre/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ def init(root_dir, verbose=False):
use_local_stage_dir = local_stage_dir.parent
else:
use_local_stage_dir = local_stage_dir
_rundir_extra = os.environ.get('HOSTNAME', 'unkownhost') + '-' + str(use_local_stage_dir).replace('/', '_')
hostname_use = _config_data.get('remote_rundir_submit_hostname', os.environ.get('HOSTNAME', 'unknown_host'))
_rundir_extra = hostname_use + '-' + str(use_local_stage_dir).replace('/', '_')

systems = {}
for _sys_name in _config_data['systems']:
Expand Down