[FIX] Cross-device hard links exception

[anibalsolon]

I just had the same problem #17 with my resource.

The error happens when accessing the output of another job from the same process, in the same resource. So the command binds -B /work/<process>/<job> and tries to hard-link e.g. /work/<process>/<another job>

From what I could investigate, by binding the job directory, it considers to be in another device even if one is a subtree of another (-B /work/<process> -B /work/<process>/<job>):

/dev/mapper/ubuntu--vg-ubuntu--lv on /home/bl/abcd-spec/hooks type ext4 (rw,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /home/bl/workdir/62b2095ae0370ed92af3a543 type ext4 (rw,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /home/bl/workdir/62b2095ae0370ed92af3a543/62b27175e0370ed92afc9ee4 type ext4 (rw,nosuid,nodev,relatime)

One solution would be to not bind anything. However it will rely on the cluster Singularity configuration, which is not ideal.

Another solution would be to only bind the process directory (in this case 62b2095ae0370ed92af3a543). This will allow access to the output from the jobs within the same process, in the same machine.

In the solution here, I implement the binding change. It is now working on my resource.

[soichih]

The error happens when accessing the output of another job from the same process, in the same resource.
All other resources do this with no issue. Is this caused by something specific to stampede2? Maybe singularity version / configuration is weird on stampede2

[anibalsolon]

I think something like that- it is the same for my installation, which is vanilla, so I would assume the others have a specific config? Maybe mount fs or overlay.