Symlink dirs

[datamel]

This PR is part of the rose suite run migration.
New section added to the global config - [symlink dirs]. Configuration as documented in https://github.com/cylc/cylc-admin/blob/master/docs/proposal-platforms.md
Configurable directories are: run, share, share/cycle, work, log.

Symlinks are now created locally on cylc register and cylc run.
Symlinks are created remotely per install target during remote initialisation.

These changes close #3688

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes - I will tidy/squash after any review changes.
• Appropriate tests are included (unit and/or functional).
• Appropriate change log entry included.
• (master branch) I will open a documentation PR at cylc/cylc-doc. Issue opened and self-assigned.
• No dependency changes.

[oliver-sanders]

Looks good, few small comments.

[oliver-sanders]

Is this one needed? Perhaps it is, but on line 103?

[datamel]

I think you are referring to the shellcheck disable? I have tried removing them and it complains if any of them are removed - I guess SSHing resets, unless there is a nifty way of globally setting it for the whole test.

[oliver-sanders]

Looks good

Tried with this configuration:

[symlink dirs]
    [[<remote-install-target>]]
        run = $HOME/cylc-ran
        log = $HOME/cylc-log
        share = $HOME/cylc-share
        share/cycle = $HOME/cylc-share-cycle
        work = $HOME/cylc-work

And got this result:

$ tree cylc-*
cylc-log
└── cylc-run
    └── one
        └── log
            └── job
                └── 1
                    └── foo
                        ├── 01
                        │   ├── job
                        │   ├── job.err
                        │   ├── job.out
                        │   └── job.status
                        └── NN -> 01
cylc-ran
└── cylc-run
    └── one
        ├── log -> /home/oliver/cylc-log/cylc-run/one/log
        ├── share -> /home/oliver/cylc-share/cylc-run/one/share
        └── work -> /home/oliver/cylc-work/cylc-run/one/work
cylc-run
└── one -> /home/oliver/cylc-ran/cylc-run/one
cylc-share
└── cylc-run
    └── one
        └── share
            └── cycle -> /home/oliver/cylc-share-cycle/cylc-run/one/share/cycle
cylc-share-cycle
└── cylc-run
    └── one
        └── share
            └── cycle
cylc-work
└── cylc-run
    └── one
        └── work
            └── 1

Nice.

Spotted a couple of minor issues.

1. I misspelt $HOME as $HOMe` and got cryptic failures.
2. I think this might be messing with the on-shutdown tidy up logic, see cylc run --no-detach traceback below:

[edit] cannot reproduce, ignore for now unless anyone else spots it.

Exception ignored in: <function BaseEventLoop.__del__ at 0x7fcbb7d89e60>
Traceback (most recent call last):
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/asyncio/base_events.py", line 628, in __del__
    self.close()
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/asyncio/unix_events.py", line 55, in close
    super().close()
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/asyncio/selector_events.py", line 91, in close
    self._close_self_pipe()
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/asyncio/selector_events.py", line 98, in _close_self_pipe
    self._remove_reader(self._ssock.fileno())
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/asyncio/selector_events.py", line 271, in _remove_reader
    key = self._selector.get_key(fd)
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/selectors.py", line 190, in get_key
    return mapping[fileobj]
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/selectors.py", line 71, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/selectors.py", line 225, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/Users/oliver/miniconda3/envs/cylc8/lib/python3.7/selectors.py", line 42, in _fileobj_to_fd
    raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1

[oliver-sanders]

Conflicts.

[oliver-sanders]

👍

[MetRonnie]

Looks good 👍 I just had one comment, up to you whether or not to address it

[oliver-sanders]

Woohoo.