Local vs remote env vars in root-dir config.

[hjoliver]

NIWA is moving to use of communal (project member) project areas, including for suite output, so we need to configure rose suite-run to symlink suite run directories to the right project directories.

Some users have more than one project on the go at once though, so hardwiring the project in login scripts isn't practical. Instead, I figured we could use a shell variable $PROJECT like this:

# rose.conf
[rose suite-run]
root-dir=*=/project/${PROJECT}/${USER}

...and advise users to set $PROJECT to the appropriate value in their local terminal session before executing rose suite-run.

However, this fails if the suite has a remote host (as most do) because rose suite-run re-invokes itself on the remote host to install files, and $PROJECT (for root-dir) is not defined there:

[oliverh@w-nwp01 ~/suites/suitex]$ export PROJECT=niwa00004
[oliverh@w-nwp01 ~/suites/suitex]$ rose suite-run -i
[INFO] export CYLC_VERSION=7.8.3
[INFO] export ROSE_ORIG_HOST=w-nwp01.maui.niwa.co.nz
[INFO] export ROSE_SITE=
[INFO] export ROSE_VERSION=2019.01.2
[INFO] create: /project/niwa00004/oliverh/cylc-run/suitex
[INFO] symlink: /project/niwa00004/oliverh/cylc-run/suitex <= /home/oliverh/cylc-run/suitex
...
[FAIL] ssh -oBatchMode=yes -n remote.maui.niwa.co.nz env\ ROSE_VERSION=2019.01.2\ 
   CYLC_VERSION=7.8.3\ bash\ -l\ -c\ \'\"$0\"\ \"$@\"\'\ rose\ suite-run\ -vv\ -n\ suitex\ --install-only\ 
   --run=run\ --remote=uuid=8fb5aaf8-9692-447a-b0c4-92a4726c3302,
   root-dir=\'/nesi/nobackup/${PROJECT}/${USER}\'
   # return-code=1, stderr=
[FAIL] [FAIL] 2019-07-22T04:16:01+0000 [UNDEFINED ENVIRONMENT VARIABLE] PROJECT

As a possibly-temporary measure, the following patch works with recent releases, by getting rose suite-run to interpolate local environment variables into "host config" values before re-invoking itself on remote hosts:

--- a/lib/python/rose/suite_run.py
+++ b/lib/python/rose/suite_run.py
@@ -400,6 +400,9 @@ class SuiteRunner(Runner):
                 value = self._run_conf(key, host=host, conf_tree=conf_tree)
                 if value is not None:
                     val = self.popen.list_to_shell_str([str(value)])
+                    # NIWA: interpolate local env vars into host_confs values.
+                    # (We use root-dir=/projects/$PROJECT/$USER, for example.)
+                    val = env_var_process(val)
                     shcommand += ",%s=%s" % (key, pipes.quote(val))
                     locs_conf.set([auth, key], value)
             command.append(shcommand)

But ... is this the appropriate permanent fix? Mabye others need different root-dir (etc.) on different hosts configured according to different login environments on those hosts (as per the status quo)?

[jonnyhtw]

Thanks for this @hjoliver

Works here at NIWA.

Jonny

[matthewrmshin]

Hi @hjoliver I am inclined to do nothing about this - as we are still looking at moving this sort of logic to Cylc in the Cylc 8 time scale.

[hjoliver]

@matthewrmshin - OK that's fine. Unfortunately we need this right now for cylc-7.8.x-compatible releases, but I've already patched (as above) our recent installed Rose versions, and we can live with local patching for any further pre-migration releases.

[matthewrmshin]

In which case, feel free to raise a PR against 2019.01.x.

[hjoliver]

I'm happy to do that, but it might be pointless depending on your response to this question above:

But ... is this the appropriate permanent fix? Mabye others need different root-dir (etc.) on different hosts configured according to different login environments on those hosts (as per the status quo)?

To rephrase that more clearly, if root-dir (etc.) config contains environment variables, where should they be evaluated for remote hosts?

• on the rose suite-run host - as we need for the reasons described
• or on the job host - as currently in the code

If both might be required depending on use case, I'll probably need some advice on how to do it in Rose.

[matthewrmshin]

I guess both required.

The reason for the current logic... I figured that only the job host's login profiles would have the latest correct configuration for those environment variables. Clearly, this is insufficient for your use case!

OK. Given that you have already patched your local environment, I agree that it is best to concentrate our effort to deliver a more flexible logic for these stuffs in Cylc 8.

[sadielbartholomew]

Is this on your radar, @wxtim? It may be relevant to cylc/cylc-admin#40?

[wxtim]

Is this on your radar, @wxtim? It may be relevant to cylc/cylc-admin#40?

It's not been on my radar, but it should be: Thank you!

[dpmatthews]

Related to #2162 ?

[oliver-sanders]

@hjoliver is there anything here that needs to be transferred to Cylc Flow?

[hjoliver]

The case in the description above is now handled by cylc platform-specific run dir config, right? In which case, this can be closed (and there's nothing else to transfer over).

[oliver-sanders]

Largely addressed by - https://github.com/cylc/cylc-flow

[oliver-sanders]

Reopening for now, need to go through and test in more detail.

[oliver-sanders]

@hjoliver

I think this is a duplicate of #2238?

The symlink functionality has now moved to Cylc 8 where it is provided by [install][symlink dirs]. This can be configured at the site or user level but not the workflow level (since this doesn't generally make sense and impedes workflow portability).

However, the cylc-rose plugin gets run before the cylc install script so this issue may have been passively solved?

https://github.com/cylc/cylc-flow/blob/6565dc05e8b2cecb07e1849b129784d81eb1bab0/cylc/flow/scripts/install.py#L210-L213

[jonnyhtw]

Thanks @oliver-sanders 😄

[hjoliver]

I think this is a duplicate of #2238?

Yes

[hjoliver]

I'll have to find the time to go test this (and to see what users are currently doing about it at NIWA ... presumably we're still using my patched Rose version.

[dpmatthews]

@hjoliver - I did some testing and I can't currently see a way to do this with Cylc 8 other than creating separate platforms for each project which isn't nice.

[oliver-sanders]

Issue transferred to cylc-flow -> cylc/cylc-flow#4612