TC Analysis failing in Unified RC

[forsyth2]

Error: slurmstepd: error: execve(): DetectNodes: No such file or directory
Task: tc_analysis_1850-1851 from zppy -c tests/integration/generated/test_complete_run.cfg
Machine: Perlmutter (pm-cpu)
Unified RC: source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.8.0rc5_pm-cpu.sh

The relevant command in tc_analysis_1850-1851.bash is:

srun -n 32 DetectNodes --verbosity 0 --in_connect ${result_dir}connect_CSne${res}_v2.dat --closedcontourcmd "PSL,300.0,4.0,0;_AVG(T200,T500),-0.6,4,0.30" --mergedist 6.0 --searchbymin PSL --outputcmd "PSL,min,0;_VECMAG(UBOT,VBOT),max,2" --timestride 1 --in_data_list ${result_dir}inputfile_${file_name}.txt --out ${result_dir}out.dat

Cori had a similar problem, that was solved by adding srun in #278.

[forsyth2]

It looks like the srun -n 32 may need to be removed for Perlmutter, as in #279.

[forsyth2]

When I remove that, I instead get

mv: inter-device move failed: '/pscratch/sd/f/forsyth//tc-analysis_1850_1851/' to '/global/cfs/cdirs/e3sm/forsyth/zppy_test_complete_run_output/v2.LR.historical_0201/post/atm/tc-analysis_1850_1851'; unable to remove target: Directory not empty

This occurs even if I delete /pscratch/sd/f/forsyth//tc-analysis_1850_1851/ prior to running.

[forsyth2]

This appears to be resolved with the following in #362:

# Avoid "unable to remove target: Directory not empty" errors on `mv $result_dir $result_dir_fin` below.
rm -rf ${result_dir}
rm -rf ${result_dir_fin}/tc-analysis_${Y1}_${Y2}