Include symlinks #261

forsyth2 · 2023-03-28T17:03:44Z

Include symlinks. Resolves #247

forsyth2 · 2023-03-28T17:17:54Z

I tried experimenting with symlinks in zstash yesterday.

With the changes in this branch:

$ cd /global/homes/f/forsyth/zstash_test/n247
$ emacs setup.sh
mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo 'file0 stuff' > zstash_demo/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt
ln zstash_demo/file0.txt zstash_demo/file2.txt
$ chmod 755 setup.sh
$ ./setup.sh
$ zstash create --hpss=zstash_archive zstash_demo
$ mkdir zstash_extraction
$ cd zstash_extraction
$ zstash extract --hpss=zstash_archive
$ ls
dir  empty_dir  file0.txt  file2.txt  file_empty.txt  zstash
$ zstash ls --hpss=zstash_archive
file0.txt
file2.txt
file_empty.txt
dir/file1.txt
empty_dir

On the main branch:

$ cd /global/homes/f/forsyth/zstash_test/n247v2
$ emacs setup.sh
mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo 'file0 stuff' > zstash_demo/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt
ln zstash_demo/file0.txt zstash_demo/file2.txt
$ chmod 755 setup.sh
$ ./setup.sh
$ zstash create --hpss=zstash_archive2 zstash_demo
$ mkdir zstash_extraction
$ cd zstash_extraction
$ zstash extract --hpss=zstash_archive2
$ ls
dir  empty_dir  file0.txt  file2.txt  file_empty.txt  zstash
$ zstash ls --hpss=zstash_archive2
file0.txt
file2.txt
file_empty.txt
dir/file1.txt
empty_dir

The results are the same, which is not what I expected. I expected the sym-linked file2.txt to be present when running zstash on this branch, but not when running on main. Do I have something wrong in my reasoning? @TonyB9000 @golaz

(Note: this tests sym-linking to a file in the directory being archived. I believe there would be differences if we sym-linked to a file in a different directory).

TonyB9000 · 2023-03-28T20:19:01Z

The final "zstash ls --hpss=zstash_archive" (and "zstash ls --hpss=zstash_archive2") does not reveal whether the content exists for the files named. Use ordinary "ls -l" to ensure they are as expected.

Also, the symlink "zstash_demo/file2.txt" points to a file "zstash_demo/file0.txt" that is present in the archive anyway. There is no opportunity for that link to be broken. The issue does not involve empty files. There is nothing wrong (necessarily) with archiving a file with no content. The problem involves (somehow) archiving a symlink to a file that does not exist - either before the archive is created, or afterwards WHEN that path to the real file was NOT already part of the archive (and will not be an available path where the archive is extracted.)

One scenario is to zstash a directory of regular files, containing a broken link (add a link-file to a real-file in a separate directory, then delete that latter real-file BEFORE making the archive. What happens when zstash tries to archive the directory containing a broken link? Does it complain about the broken link? Does it quietly include the broken link in the archive?

Another scenario is to leave the real-file (in a separate directory) intact when forming the zstash archive, then destroying the real file, and attempting to zstash extract. This (I think) would simulate taking the zstash archive to a new file-system where the "symlink" (link-file) should no longer work. Was the sym-link exercised (is the real-file present in the archive? Or is it now just a broken link?)

As you mentioned under "issues 247", you could call tarinfo.issym (https://docs.python.org/3/library/tarfile.html#tarfile.TarInfo) to see if a file is a symlink. If so, replace "link-file" with os.path.realpath(link-file) (https://docs.python.org/3/library/os.path.html).

The remaining issue would be to ensure that a file "link-named" as "FILE_A", that was a symlink to a distant-path to "FILE_B", not only gets copied over, but remains named FILE_A and is not tarred as "distant_path/FILE_B" (the value returned by "realpath").

forsyth2 · 2023-03-31T20:15:49Z

The final "zstash ls --hpss=zstash_archive" (and "zstash ls --hpss=zstash_archive2") does not reveal whether the content exists for the files named. Use ordinary "ls -l" to ensure they are as expected.

In both cases, ls -l shows a size of 12 for file2.txt.

One scenario is to zstash a directory of regular files, containing a broken link (add a link-file to a real-file in a separate directory, then delete that latter real-file BEFORE making the archive. What happens when zstash tries to archive the directory containing a broken link?

It appears to just work somehow? On both branches (main and n247-symlinks), ls -l shows a size of 12 for file2.txt.

Another scenario is to leave the real-file (in a separate directory) intact when forming the zstash archive, then destroying the real file, and attempting to zstash extract.

This also appears to work. On both branches, ls -l shows a size of 12 for file2.txt.

TonyB9000 · 2023-03-31T20:42:45Z

One scenario is to zstash a directory of regular files, containing a broken link (add a link-file to a real-file in a separate directory, then delete that latter real-file BEFORE making the archive. What happens when zstash tries to archive the directory containing a broken link?

It appears to just work somehow? On both branches (main and n247-symlinks), ls -l shows a size of 12 for file2.txt.

Are you sure that is the actual file, and not just a "link-name"?

That should be impossible. If I create a symlink to a file in a distant directory, then delete that target file, the symlink becomes broken. It cannot be used to access the missing data.

If you then zstash the directory containing the symlink, *something" may end up in the archive, but it cannot be the target file that does not exist,

We have a zstash archive: /p/user_pub/e3sm/archive/2_0/DECK-v2/v2.NARRM.amip_0201/

If I conduct "zstash ls -l --hpss=none archive/atm/hist/v2.NARRM.amip_0201.eam.h0.*.nc" what I see includes:

994     archive/atm/hist/v2.NARRM.amip_0201.eam.h0.1952-10.nc   1190877200      2022-03-18 14:41:36.222648      b5aec1e99fea718a88be4363b091e29f        000008.tar      86934183936
995     archive/atm/hist/v2.NARRM.amip_0201.eam.h0.1952-11.nc   1190877200      2022-03-18 14:51:13.824008      4d092038bfd91e2a3a279ac37ecfd053        000008.tar      88125063168
996     archive/atm/hist/v2.NARRM.amip_0201.eam.h0.1952-12.nc   0       2022-03-21 04:23:09.129694      None    000008.tar      89315942400
997     archive/atm/hist/v2.NARRM.amip_0201.eam.h0.1953-01.nc   1190877712      2022-03-21 04:35:19.980703      d17425dfe969040ba9d20f3c6484553e        000008.tar      89315943936
998     archive/atm/hist/v2.NARRM.amip_0201.eam.h0.1953-02.nc   1190877712      2022-03-21 04:44:17.562973      5cc26b53af9e33b09d92601b7a6ded9c        000008.tar      90506823680

Somehow, the sim-team managed to create a 0-length file (archive/atm/hist/v2.NARRM.amip_0201.eam.h0.1952-12.nc) in the archive. I don't know how they did it, but there must be a way to avoid this - throw up a warning or exception at archive-time. When it is discovered only months later, it leads to far more recovery work.

I will try to develop a test regime to duplicate the process. But your example ("appears to just work somehow?") demonstrates that zstash is actually NOT working. Otherwise, zstash is just inventing files that do not exist.

forsyth2 · 2023-03-31T20:50:48Z

The commands I ran were essentially the same as above, but with the following differences:

setup.sh: putting file0.txt in a directory that won't actually be archived

mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
mkdir non_archived
echo 'file0 stuff' > non_archived/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt
ln non_archived/file0.txt zstash_demo/file2.txt

Then running rm non_archived/file0.txt either before or after zstash create --hpss=<archive-name> zstash_demo

Are you sure that is the actual file, and not just a "link-name"?

Is there a way to determine that?

I will try to develop a test regime to duplicate the process.

Yes, that would be great. Thank you!

TonyB9000 · 2023-03-31T20:59:57Z

Are you sure that is the actual file, and not just a "link-name"?
Is there a way to determine that?

If you delete the file "non_archived/file0.txt" BEFORE creating the archive, the link "zstash_demo/file2.txt" should be broken.

If you extract the file "zstash_demo/file2.txt", from the later completed archive, and then issue

cat zstash_demo/file2.txt

You should get an error. You should not get 'file0 stuff' because the actual file no longer existed when the archive was created.

Otherwise - magic is present...

P.S. Echo more text into file0 before starting:

echo 'file0_supercalifragilisticexpialadocious_stuff' > non_archived/file0.txt

forsyth2 · 2023-03-31T21:06:53Z

Hmm cat zstash_extraction/file2.txt shows file0 stuff for both the before-create and after-create cases.

TonyB9000 · 2023-03-31T22:50:00Z

Then perhaps the archive was merely being "appended to" instead of being "recreated"?

Here is my test:

(dsm_dev) -bash-4.2$ pwd
/p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests

(dsm_dev) -bash-4.2$ mkdir other_data sourcedata
(dsm_dev) -bash-4.2$ echo "This is remote target data" > other_data/other_file
(dsm_dev) -bash-4.2$ echo "This is local data" > sourcedata/file_1
(dsm_dev) -bash-4.2$ cd sourcedata
(dsm_dev) -bash-4.2$ ln -s /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/other_data/other_file file_2
(dsm_dev) -bash-4.2$ ls -l
total 2
-rw-rw-r--. 1 bartoletti1 publishers 19 Mar 31 15:00 file_1
lrwxrwxrwx. 1 bartoletti1 publishers 70 Mar 31 15:05 file_2 -> /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/other_data/other_file
(dsm_dev) -bash-4.2$ cd ..
(dsm_dev) -bash-4.2$ cat sourcedata/file_1
This is local data
(dsm_dev) -bash-4.2$ cat sourcedata/file_2
This is remote target data

(dsm_dev) -bash-4.2$ rm other_data/other_file
(dsm_dev) -bash-4.2$ ls -l sourcedata/
total 2
-rw-rw-r--. 1 bartoletti1 publishers 19 Mar 31 15:00 file_1
lrwxrwxrwx. 1 bartoletti1 publishers 70 Mar 31 15:05 file_2 -> /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/other_data/other_file
(dsm_dev) -bash-4.2$ cat sourcedata/file_1
This is local data
(dsm_dev) -bash-4.2$ cat sourcedata/file_2
cat: sourcedata/file_2: No such file or directory

(NOW - I don't have HPSS, so ...)

(dsm_dev) -bash-4.2$ mkdir archive_test
(dsm_dev) -bash-4.2$ zstash create --hpss none --cache /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/archive_test/zstash /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/sourcedata

INFO: Gathering list of files to archive
INFO: Creating new tar archive 000000.tar
INFO: Archiving file_1
INFO: Archiving file_2
INFO: tar name=000000.tar, tar size=10240, tar md5=cbc423fbb99c7fc10655b038df522b19
INFO: put: HPSS is unavailable
INFO: put: Keeping tar files locally and removing write permissions
INFO: '/p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/archive_test/zstash/000000.tar' original mode=b"'664'"
INFO: '/p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/archive_test/zstash/000000.tar' new mode=b"'444'"
INFO: put: HPSS is unavailable
(dsm_dev) -bash-4.2$ cd archive_test/zstash/
(dsm_dev) -bash-4.2$ ls -l
total 32
-r--r--r--. 1 bartoletti1 publishers 10240 Mar 31 15:32 000000.tar
-rw-r--r--. 1 bartoletti1 publishers 20480 Mar 31 15:32 index.db

(dsm_dev) -bash-4.2$ tar tvf *.tar
-rw-rw-r-- bartoletti1/publishers 19 2023-03-31 15:00 file_1
lrwxrwxrwx bartoletti1/publishers  0 2023-03-31 15:05 file_2 -> /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/other_data/other_file

(for some reason, I cannot read the "index.db" file.)

(dsm_dev) -bash-4.2$ cd ..
(dsm_dev) -bash-4.2$ ls -l
total 1
drwxrwxr-x. 2 bartoletti1 publishers 4096 Mar 31 15:32 zstash
(dsm_dev) -bash-4.2$ zstash extract --hpss=none *
INFO: No failures detected when extracting the files. If you have a log file, run "grep -i Exception <log-file>" to double check.
(dsm_dev) -bash-4.2$ ls -l
total 1
drwxrwxr-x. 2 bartoletti1 publishers 4096 Mar 31 15:32 zstash

At least, there were no errors . . .

I am not very practiced at making archives. Can you try this (make an archive of that "sourcedata" directory?)

TonyB9000 · 2023-03-31T22:56:54Z

Addendum: Here is "zstash ls -l"

(dsm_dev) -bash-4.2$ ll
total 1
drwxrwxr-x. 2 bartoletti1 publishers 4096 Mar 31 15:32 zstash
(dsm_dev) -bash-4.2$ zstash ls -l --hpss=none *
id      name    size    mtime   md5     tar     offset

(Edit): Also:

(dsm_dev) -bash-4.2$ zstash version
v1.2.0

forsyth2 · 2023-04-03T17:36:14Z

Ah, so it turns out I was using ln wrong. I didn't realize you need the -s to make it a symbolic link instead of a hard link.

Updated results (ls -l size, ls -l link, zstash ls -l ... size) from my testing:

Case	`main`	`n247-symlinks`
Don't delete original file	22, file2.txt -> non_archived/file0.txt, 0	On create: `FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash_diagnostic/n247_branch_no_delete_3/zstash_demo/non_archived/file0.txt'`
Delete before create	22, file2.txt -> non_archived/file0.txt, 0	On create: `FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash_diagnostic/n247_branch_before_create_3/zstash_demo/non_archived/file0.txt'`
Delete after create	22, file2.txt -> non_archived/file0.txt, 0	On create: `FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash_diagnostic/n247_branch_after_create_3/zstash_demo/non_archived/file0.txt'`

Unfortunately, this doesn't appear to illuminate much. I'm not seeing why the results are the same in all 3 cases.... I'll keep debugging and try out your test commands as well.

TonyB9000 · 2023-04-03T18:24:59Z

Also, I am only using "zstash . . . --hpss=none", which is not what the simulation teams will use when creating their archives. But (in principle) that should not matter with regard to treatment of broken symlinks, etc. (Perhaps different execution paths are exercised?)

One way or another, broken links are being archived - and we need to prevent it (detect early).

forsyth2 · 2023-04-03T21:40:54Z

@TonyB9000 I tried using hpss=none and got the same results as in my table above. What machine are you running on?
cd /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests fails on both Chrysalis and Perlmutter. (I've been running on Perlmutter).

TonyB9000 · 2023-04-03T21:46:57Z

I do all of my work on acme1, and run all jobs from the mounted /p/user_pub/ space.

forsyth2 · 2023-04-03T23:51:00Z

I don't have access to acme1, but I was able to more-or-less replicate your results on Perlmutter. Your test case essentially matches my Delete before create case above.

I'm assuming the following behavior is what we would actually want after extraction:

Case	`ls -l` size	`ls -l` link	`zstash ls -l ...` size	Notes
Don't delete original file	22	file2.txt (without the `-> non_archived/file0.txt`)	22	Extraction pulls down a copy, even though file is never deleted
Delete before create				Failure on `zstash create`
Delete after create	22	file2.txt (without the `-> non_archived/file0.txt`)	22	Extraction pulls down a copy of the since-deleted file

Is that right?

TonyB9000 · 2023-04-04T00:52:38Z

Not quite: On the "delete before" test, I do NOT get a zstash create failure. What I get:

INFO: Gathering list of files to archive
INFO: Creating new tar archive 000000.tar
INFO: Archiving file_1
INFO: Archiving file_2
INFO: tar name=000000.tar, tar size=10240, tar md5=cbc423fbb99c7fc10655b038df522b19
INFO: put: HPSS is unavailable
INFO: put: Keeping tar files locally and removing write permissions
INFO: '/p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/archive_test/zstash/000000.tar' original mode=b"'664'"
INFO: '/p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/archive_test/zstash/000000.tar' new mode=b"'444'"
INFO: put: HPSS is unavailable
(dsm_dev) -bash-4.2$ cd archive_test/zstash/
(dsm_dev) -bash-4.2$ ls -l
total 32
-r--r--r--. 1 bartoletti1 publishers 10240 Mar 31 15:32 000000.tar
-rw-r--r--. 1 bartoletti1 publishers 20480 Mar 31 15:32 index.db

If I examine the raw tar file that zstash created, it contains:

(dsm_dev) -bash-4.2$ tar tvf *.tar
-rw-rw-r-- bartoletti1/publishers 19 2023-03-31 15:00 file_1
lrwxrwxrwx bartoletti1/publishers  0 2023-03-31 15:05 file_2 -> /p/user_pub/e3sm/bartoletti1/ISSUES/zstash_tests/other_data/other_file

if I use zstash to extract all content of the archive, I get:

(dsm_dev) -bash-4.2$ zstash extract --hpss=none *
INFO: No failures detected when extracting the files. If you have a log file, run "grep -i Exception <log-file>" to double check.
(dsm_dev) -bash-4.2$ ls -l
total 1
drwxrwxr-x. 2 bartoletti1 publishers 4096 Mar 31 15:32 zstash

As you can see - nothing was extracted. But zstash is happy.

Addendum: Here is "zstash ls -l"

(dsm_dev) -bash-4.2$ ll
total 1
drwxrwxr-x. 2 bartoletti1 publishers 4096 Mar 31 15:32 zstash
(dsm_dev) -bash-4.2$ zstash ls -l --hpss=none *
id      name    size    mtime   md5     tar     offset

(Edit): Also:

(dsm_dev) -bash-4.2$ zstash version
v1.2.0

forsyth2 · 2023-04-04T16:18:36Z

@TonyB9000 Sorry, in the table above, I meant that those are the results we would want to see (i.e., not what the current implementation of zstash does).

Also, it looks like you're running zstash extract directly out of the archive. When I cd to my equivalent of the directory above archive_test and then create a separate directory zstash_extract to run the extraction in, I get:

Case	`ls -l` size	`ls -l` link	`zstash ls -l ...` size	Notes
Modified version of your test case, `main`	85	file_2 -> /global/homes/f/forsyth/zstash_diagnostic/tony_test_case_main_5/other_data/other_file	0
Modified version of your test case, `n247-symlinks`				`ERROR: Archiving file_2` on `zstash create`

TonyB9000 · 2023-04-04T18:21:59Z

Actually, when I create a zstash archive, like

/path/to/archive_name/(contains tarfiles and index.db)

I create a "Holodeck" directory elsewhere, as follows:

/path/to/Holodeck/zstash/ (contains symlinks to the real tarfiles and index.db under /path/to/archive_name/)

and I "reside" in Holodeck (above the "zstash" directory) when issuing zstash commands.

I employ this "Holodeck" method to insulate myself from the fact that - depending on parameters, zstash would delete the very archived from which files are extracted. When HPSS is present, the tape-extracted tar-files are only temporary. But for our purposes, they ARE the archives. This way, at worst, only the symlinks are deleted.

forsyth2 · 2023-04-07T23:28:37Z

@TonyB9000 I updated the run script and results:

Updated run script:

#!/bin/bash

checkout_branch()
{
  branch=$1
  cd /global/homes/f/forsyth/zstash
  git checkout ${branch}
  pip install .
}

setup()
{
  archive_name=$1
  echo "##########################################################################################################"
  echo ${archive_name}
  cd /global/homes/f/forsyth/zstash_diagnostic
  rm -rf n247_${archive_name}
  rm -rf run_n247_non_archived
  mkdir n247_${archive_name}
  mkdir run_n247_non_archived
  cd n247_${archive_name}

  mkdir zstash_demo
  mkdir zstash_demo/empty_dir
  mkdir zstash_demo/dir
  mkdir non_archived
  echo '' > zstash_demo/file_empty.txt
  echo 'file0 stuff' > zstash_demo/dir/file0.txt
  echo 'file1 stuff' > non_archived/file1.txt
  echo 'file2 stuff' > ../run_n247_non_archived/file2.txt
  # NOTE `ln -s` appears to require absolute paths for the source files
  #ln -s non_archived/file1.txt zstash_demo/file3.txt
  #ln -s ../run_n247_non_archived/file2.txt zstash_demo/file4.txt
  ln -s /global/homes/f/forsyth/zstash_diagnostic/n247_${archive_name}/non_archived/file1.txt zstash_demo/file3.txt
  ln -s /global/homes/f/forsyth/zstash_diagnostic/run_n247_non_archived/file2.txt zstash_demo/file4.txt
  cat zstash_demo/file3.txt
  cat zstash_demo/file4.txt
}

zstash_create()
{
  archive_name=$1
  echo "Starting zstash create"
  zstash create --hpss=zstash_n247_${archive_name} zstash_demo  
}

zstash_extract()
{
  archive_name=$1
  mkdir zstash_extraction
  cd zstash_extraction
  echo "Starting zstash extract"
  zstash extract --hpss=zstash_n247_${archive_name}
  cat file3.txt
  cat file4.txt
  ls
  ls -l
  zstash ls --hpss=zstash_n247_main_${archive_name}
  zstash ls -l --hpss=zstash_n247_main_${archive_name}
  cd ..
}

test_cases()
{
  branch=$1
  archive_num=$2

  # Checkout branch
  checkout_branch ${branch}
  
  # No delete
  archive_name=${branch}_no_delete_${archive_num}
  setup ${archive_name}
  zstash_create ${archive_name}
  zstash_extract ${archive_name}

  # Delete before create
  archive_name=${branch}_before_create_${archive_num}
  setup ${archive_name}
  rm non_archived/file1.txt
  rm ../run_n247_non_archived/file2.txt
  zstash_create ${archive_name}
  zstash_extract ${archive_name}
  
  # Delete after create
  archive_name=${branch}_after_create_${archive_num}
  setup ${archive_name}
  zstash_create ${archive_name}
  rm non_archived/file1.txt
  rm ../run_n247_non_archived/file2.txt
  zstash_extract ${archive_name}

}

archive_num=17
source /global/homes/f/forsyth/miniconda3/etc/profile.d/conda.sh
conda activate zstash_dev_n247
test_cases main ${archive_num}
test_cases n247-symlinks ${archive_num}

Note that file3.txt is sym-linked to a non-archived directory under where we're running zstash from. file4.txt, however is sym-linked to a non-archived directory not under where we're running zstash from.

Updated results (ls -l size, ls -l link, zstash ls -l ... size) [scroll right to see rest of table]:

Case	`main` file3	`main` file4	`n247-symlinks` file3	`n247-symlinks` file4	Notes
Don't delete original file	87, file3.txt -> /global/homes/f/forsyth/zstash_diagnostic/n247_main_no_delete_17/non_archived/file1.txt, 0	73, file4.txt -> /global/homes/f/forsyth/zstash_diagnostic/run_n247_non_archived/file2.txt, 0	96, file3.txt -> /global/homes/f/forsyth/zstash_diagnostic/n247_n247-symlinks_no_delete_17/non_archived/file1.txt, 0	N/A, N/A, 0	Multiple instances of `tarfile.InvalidHeaderError: bad checksum` on `extract` in `n247-symlinks` version
Delete before create	91, file3.txt -> /global/homes/f/forsyth/zstash_diagnostic/n247_main_before_create_17/non_archived/file1.txt, 0	73, /global/homes/f/forsyth/zstash_diagnostic/run_n247_non_archived/file2.txt, 0	N/A	N/A	`FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash_diagnostic/n247_n247-symlinks_before_create_17/non_archived/file1.txt'`,`FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash_diagnostic/run_n247_non_archived/file2.txt'` on `create` in `n247-symlinks` version
Delete after create	90, file3.txt -> /global/homes/f/forsyth/zstash_diagnostic/n247_main_after_create_17/non_archived/file1.txt, 0	73, file4.txt -> /global/homes/f/forsyth/zstash_diagnostic/run_n247_non_archived/file2.txt, 0	99, file3.txt -> /global/homes/f/forsyth/zstash_diagnostic/n247_n247-symlinks_after_create_17/non_archived/file1.txt, 0	N/A, N/A, 0	Multiple instances of `tarfile.InvalidHeaderError: bad checksum` on `extract` in `n247-symlinks` version

forsyth2 · 2023-04-07T23:35:05Z

Results of running cat on the sym-links, in the zstash_extraction directory:

Case	`main` file3	`main` file4	`n247-symlinks` file3	`n247-symlinks` file4
Don't delete original file	`file1 stuff`	`file2 stuff`	`file1 stuff`	`cat: file4.txt: No such file or directory`
Delete before create	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`
Delete after create	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`

forsyth2 · 2023-04-07T23:42:54Z

The results we would actually want from cat, I believe:

Case	file3	file4
Don't delete original file	`file1 stuff`	`file2 stuff`
Delete before create	`file1 stuff`	`file2 stuff`
Delete after create	`file1 stuff`	`file2 stuff`

That is, because we'd be making hard copies of the sym links, we'd always expect to have the data available.

forsyth2

@TonyB9000 Results of running cat on the sym-links, in the zstash_extraction directory, using the latest commit:

Case	`main` file3	`main` file4	`n247-symlinks` file3	`n247-symlinks` file4
Don't delete original file	`file1 stuff`	`file2 stuff`	`file1 stuff`	`file2 stuff`
Delete before create	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`
Delete after create	`cat: file3.txt: No such file or directory`	`cat: file4.txt: No such file or directory`	`file1 stuff`	`file2 stuff`

I think that's actually what we want, right? If the file being linked to is deleted before zstash create, there's nothing we can do in that command to get it back.

If this is the correct behavior, I just need to make a command line flag to use the old (current) behavior instead.

TonyB9000 · 2023-04-14T17:52:07Z

FYI:
echo '' > zstash_demo/file_empty.txt
Does not create an empty file. It creates a file with a single blank line (1 byte newline char). Use
touch zstash_demo/file_empty.txt
to create an empty file.

If I employ the name "TOP" to mean the full path to the test directory (zstash_diagnostic), and run "setup.sh XXX", the following is created:

    TOP/n247_XXX/non_archived/file1.txt       (file1 stuff)
    TOP/n247_XXX/zstash_demo/dir/file0.txt    (file0 stuff)
    TOP/n247_XXX/zstash_demo/empty_dir/
    TOP/n247_XXX/zstash_demo/file3.txt -> TOP/n247_XXX/non_archived/file1.txt
    TOP/n247_XXX/zstash_demo/file4.txt -> TOP/run_n247_non_archived/file2.txt
    TOP/n247_XXX/zstash_demo/file_empty.txt

    TOP/run_n247_non_archived/
    TOP/run_n247_non_archived/file2.txt             (file2 stuff)

In short, your "file3" and "file4" are always symlinks to "file1" and "file2", respectively.

Let me refer to your table by its rows:

Row 2: It is not enough that we can't recover a file that was not (could not be) archived. We must abort the archiving itself, so we do not produce an archive that has broken links as content. What does "zstash ls -l *" say about the archive? Do the "broken" links appear at all? There should be a flag that allows "broken links" to be archived, if needed.

Row 3: That is the expected behavior. The archive contains the files themselves (existing at archive-creation time, but named according to the symlink names), and not merely archiving the symlinks.

forsyth2 · 2023-04-14T20:53:07Z

Does not create an empty file

Good to know, thanks.

In short, your "file3" and "file4" are always symlinks to "file1" and "file2", respectively.

Yes, that's how I set up the initial conditions.

We must abort the archiving itself

I think this does do that by propagation of the FileNotFoundError from the removed file.

What does "zstash ls -l *" say about the archive?

We can't get there because the archive doesn't get created. Instead, there's a confusing ERROR: Error=Transferring file from HPSS, which I now realize is because there's no archive to extract from. Perhaps I should add a note to that error that the archive may not exist.

Row 3: That is the expected behavior.

Awesome.

TonyB9000 · 2023-04-14T22:32:55Z

FYI, I just discovered you can use

echo -n "" > empty_file

since the "-n" omits the newline.

TonyB9000 · 2023-04-14T22:45:07Z

there's a confusing ERROR: Error=Transferring file from HPSS, which I now realize is because there's no archive to extract from.

I think the behavior for "--hpss=<archive_name>" and "--hpss=none" should be distinct. Error messages for one are not always appropriate for the other. But if there is no archive found with a given name, then "zstash ls" and "zstash extract" should say so., and not say "Error transferring file from HPSS". That would be weird.

If I issue in bash, "cp fileA fileB" and fileA does not exist, I don't want to see "Error copying fileA to fileB, FileA may not exist". I want to see "Error: 'FileA' not found.

forsyth2 · 2023-04-19T00:35:12Z

@TonyB9000 Results from latests changes, on n247-symlinks branch:

With HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash_diagnostic/n247_zstash_n247_n247-symlinks_before_create_37/non_archived/file1.txt' => Exception: Archive creation failed due to missing file	success
`zstash extract`	success	RuntimeError: Error=Transferring file from HPSS: index.db, Command was `hsi -q "cd zstash_n247_n247-symlinks_before_create_37; get index.db"`. This command includes `hsi`. Be sure that you have logged into `hsi`.	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`file1 stuff`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`file2 stuff`
`zstash ls` name, size	`file3.txt 12`, `file4.txt 12`	RuntimeError: Error=Transferring file from HPSS: index.db, Command was `hsi -q "cd zstash_n247_n247-symlinks_before_create_37; get index.db"`. This command includes `hsi`. Be sure that you have logged into `hsi`.	`file3.txt 12`, `file4.txt 12`

Without HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	FileNotFoundError: [Errno 2] No such file or directory: 'file3.txt' => Exception: Archive creation failed due to missing file	success
`zstash extract`	success	FileNotFoundError: There was nothing to extract.	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`file1 stuff`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`file2 stuff`
`zstash ls` name, size	`file3.txt 12`, `file4.txt 12`	FileNotFoundError: There was nothing to extract.	`file3.txt 12`, `file4.txt 12`

Ok, I think the "Without HPSS" cases are functioning as we want, correct? (Although I suppose I should change the error message for zstash ls to say nothing to ls).

As for the "With HPSS?" cases, it looks like zstash extract and zstash ls could have more useful error messages for "Delete before create".

TonyB9000 · 2023-04-19T16:01:48Z

This looks great Ryan! This behavior should prevent "bad archives" from slipping out into the open.

For "zstash ls", I agree it should behave like (bash) ls. Programmers (like me) will sometimes use "filecount=`ls dir | wc -l`", and expect filecount to be 0 when no files (no "lines") are returned.

forsyth2 · 2023-04-21T22:38:17Z

@TonyB9000 Results from latest changes, on n247-symlinks branch below.

I decided to make the new behavior (rather than the behavior in main) the command-line option, for backwards compatibility and for ease of testing (all the existing unit tests would have to be updated otherwise). That said, I think the changes I've put in extract and ls may technically break backwards compatibility.

I think I've copied everything over correctly. We're looking at 12 cases here - [using copy-symlinks, not] x [using HPSS, not] x [no deletion, delete before, delete after]. Does all this behavior look like what we want it to be?

With --copy-symlinks, with HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	FileNotFoundError: [Errno 2] No such file or directory: 'file3.txt' => Exception: Archive creation failed due to broken symlink.	success
`zstash extract`	success	RuntimeError: Error=Transferring file from HPSS: index.db, Command was `hsi -q "cd zstash_n247_n247-symlinks_before_create_42_true; get index.db"`. This command includes `hsi`. Be sure that you have logged into `hsi`. This command includes `cd`. Check that this directory exists and contains the needed files	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`file1 stuff`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`file2 stuff`
`zstash ls` name, size	`file3.txt 12`, `file4.txt 12`	ERROR: Error=Transferring file from HPSS: index.db, Command was `hsi -q "cd zstash_n247_n247-symlinks_before_create_42_true; get index.db"`. This command includes `hsi`. Be sure that you have logged into `hsi`. This command includes `cd`. Check that this directory exists and contains the needed files	`file3.txt 12`, `file4.txt 12`

With --copy-symlinks, without HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	FileNotFoundError: [Errno 2] No such file or directory: 'file3.txt'=> Exception: Archive creation failed due to broken symlink.	success
`zstash extract`	success	FileNotFoundError: There was nothing to extract.	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`file1 stuff`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`file2 stuff`
`zstash ls` name, size	`file3.txt 12`, `file4.txt 12`	FileNotFoundError: There was nothing to ls.	`file3.txt 12`, `file4.txt 12`

Without --copy-symlinks, with HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	success	success
`zstash extract`	success	success	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`cat: file3.txt: No such file or directory`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`cat: file4.txt: No such file or directory`
`zstash ls` name, size	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`

Without --copy-symlinks, without HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	success	success
`zstash extract`	success	success	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`cat: file3.txt: No such file or directory`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`cat: file4.txt: No such file or directory`
`zstash ls` name, size	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`

forsyth2 · 2023-04-21T22:40:11Z

zstash/ls.py

+                # Ignore failed commands
+                # If the match list is empty, then `zstash ls` will display nothing,
+                # just as `ls` does.
+                return []


I don't think this is getting used. The "nothing to ls" message still gets printed in some cases.

TonyB9000 · 2023-04-25T15:52:42Z

The changes look good to me, although I am personally unfamiliar with using zstash with HPSS, in either "create" or "extract" (or for that matter, "ls") modes. (Graphical/pictorial representations of steps depicting tape-archive, local caches, final destinations, etc with create and extract would probably be useful someday, for HPSS and non-HPSS usages.)

When you say the "new behavior" will be the command-line option, does this apply only to create? I am not familiar with the "short-term-archiving" process (post simulation-runs), so I don't know where this would be applied. Would I need to apply the command-line option when perusing or extracting from "caches" that are our "zstash archives"?

golaz · 2023-04-25T22:21:08Z

@forsyth2 and @TonyB9000: great new feature, thanks for working on it.

I would suggest to reconsider the naming of the command line option: --copy-symlinks is not terribly intuitive in my view. It could be argued that this is what zstash already does: it copies the symlinks into the archive and extracts them as symlinks. An alternate option would be to follow the naming convention from tar and use --dereference instead. From man tar:

       -h, --dereference
              Follow symlinks; archive and dump the files they point to.

TonyB9000 · 2023-04-25T22:55:20Z

I second the motion. --dereference, (or --follow-symlinks) would be better than "--copy-symlinks".

It should be clear in the usage statement that a failure to dereference a symlink will result in a failed archive. I just hope that folks won't avoid it on that account. I'd prefer not to receive broken archives...

forsyth2 · 2023-04-26T19:45:52Z

(Graphical/pictorial representations of steps depicting tape-archive, local caches, final destinations, etc with create and extract would probably be useful someday, for HPSS and non-HPSS usages.)

I created #262 for this.

When you say the "new behavior" will be the command-line option, does this apply only to create?

It will be a command-line option for create and update. However, other functions will be impacted downstream -- i.e., if create prevents an archive from being created in the first place, then extract will fail with the new error about the archive (possibly) not existing.

I am not familiar with the "short-term-archiving" process (post simulation-runs), so I don't know where this would be applied.

Short-term archiving is independent of, and done well before, any calls to zstash. Notice that in https://acme-climate.atlassian.net/wiki/spaces/ED/pages/2309226536/Running+E3SM+step-by-step+guide, "Short Term Archiving" is listed well before "Long Term Archiving with zstash".

Would I need to apply the command-line option when perusing or extracting from "caches" that are our "zstash archives"?

See my second comment above.

I would suggest to reconsider the naming of the command line option

Sure, I can change copy-symlinks to dereference.

It should be clear in the usage statement that a failure to dereference a symlink will result in a failed archive

I can add that as well.

forsyth2 · 2023-04-28T13:58:09Z

Ok, I think this is ready to merge. New behavior is as follows:

With --follow-symlinks, with HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash/tests/test_follow_symlinks/non_archived/file1.txt' & FileNotFoundError: [Errno 2] No such file or directory: 'file3.txt' => Exception: Archive creation failed due to broken symlink.	success
`zstash extract`	success	RuntimeError: Error=Transferring file from HPSS: index.db, Command was `hsi -q "cd /home/f/forsyth/zstash_test_follow_symlinks; get index.db"`. This command includes `hsi`. Be sure that you have logged into `hsi`. This command includes `cd`. Check that this directory exists and contains the needed files	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`file1 stuff`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`file2 stuff`
`zstash ls` name, size	`file3.txt 12`, `file4.txt 12`	RuntimeError: Error=Transferring file from HPSS: index.db, Command was `hsi -q "cd /home/f/forsyth/zstash_test_follow_symlinks; get index.db"`. This command includes `hsi`. Be sure that you have logged into `hsi`. This command includes `cd`. Check that this directory exists and contains the needed files => FileNotFoundError: There was nothing to ls.	`file3.txt 12`, `file4.txt 12`

With --follow-symlinks, without HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	FileNotFoundError: [Errno 2] No such file or directory: '/global/u1/f/forsyth/zstash/tests/test_follow_symlinks/non_archived/file1.txt' & FileNotFoundError: [Errno 2] No such file or directory: 'file3.txt' => Exception: Archive creation failed due to broken symlink.	success
`zstash extract`	success	FileNotFoundError: There was nothing to extract.	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`file1 stuff`
`cat file4.txt`	`file2 stuff`	`cat: file4.txt: No such file or directory`	`file2 stuff`
`zstash ls` name, size	`file3.txt 12`, `file4.txt 12`	FileNotFoundError: There was nothing to ls.	`file3.txt 12`, `file4.txt 12`

Without --follow-symlinks, with HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	success	success
`zstash extract`	success	success	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`cat: file3.txt: No such file or directory`
`cat file4.txt`	`file2 stuff`	`file2 stuff`	`file2 stuff`
`zstash ls` name, size	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`

Without --follow-symlinks, without HPSS:

	Don't delete original file	Delete before create	Delete after create
`zstash create`	success	success	success
`zstash extract`	success	success	success
`cat file3.txt`	`file1 stuff`	`cat: file3.txt: No such file or directory`	`cat: file3.txt: No such file or directory`
`cat file4.txt`	`file2 stuff`	`file2 stuff`	`file2 stuff`
`zstash ls` name, size	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`

TonyB9000 · 2023-04-28T16:23:54Z

Looks good to me Ryan - Excellent work and thorough coverage.

I do wonder how it will run for me (zstash ls) on an archive with broken symlinks created using the older zstash. Of course, "zstash ls -l" always worked correctly (demonstrated file length 0, md5 hash = None), but that was an inconvenient way to discover broken links. Ideally, the new "zstash ls --follow-symlinks filename" on an old archive should not list "filename" if it is a broken symlink (i.e. just a symlink, "not followed").

I guess this begs the question: If people create new archives without "--follow-symlinks" (original behavior), and I receive the "cache" as a non-HPSS archive, and "file" is a broken symlink, will the new "zstash ls file --follow-symlinks" simply list the file as if present? Or will it behave differently...

forsyth2 · 2023-04-28T17:06:52Z

Thanks @TonyB9000!

"zstash ls --follow-symlinks filename"

Ah, so this does not exist. The --follow-symlinks is only new to the create and update commands. (Which reminds me, I need to add those to the docs at https://e3sm-project.github.io/zstash/_build/html/main/usage.html). I didn't explicitly test update but the relevant logic is similar to create.

If people create new archives without "--follow-symlinks" (original behavior), and I receive the "cache" as a non-HPSS archive, and "file" is a broken symlink, will the new "zstash ls file --follow-symlinks" simply list the file as if present?

If the archive is created without --follow-symlinks and without HPSS, we'd be in the 4th table of my comment above. So, the results will be as indicated for the last row (zstash ls) of that table. That is:

	Don't delete original file	Delete before create	Delete after create
`zstash ls` name, size	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`	`file3.txt 0`, `file4.txt 0`

TonyB9000 · 2023-04-28T17:52:00Z

OK, understood. It would probably be a hassle for every "zstash ls" to do an internal "ls -l" and test for "md5=None" or "len=0", which was the only way I confirmed I was looking at a broken link. I can probably use (and parse) "zstash ls -l" instead of just calling "zstash ls" to test for file existing/location stuff.

forsyth2 · 2023-04-28T17:54:06Z

I need to add those to the docs

#263

I can probably use (and parse) "zstash ls -l" instead of just calling "zstash ls" to test for file existing/location stuff.

Sounds good. If it turns out to be too much of a problem, we can create a new issue to handle that.

forsyth2 marked this pull request as draft March 28, 2023 17:04

forsyth2 force-pushed the n247-symlinks branch from 87c91c1 to 8e9a4fa Compare April 13, 2023 23:44

forsyth2 commented Apr 13, 2023

View reviewed changes

forsyth2 force-pushed the n247-symlinks branch from 8e9a4fa to c92453a Compare April 18, 2023 23:56

forsyth2 force-pushed the n247-symlinks branch from 04b113b to d29bcd2 Compare April 21, 2023 22:08

forsyth2 commented Apr 21, 2023

View reviewed changes

forsyth2 mentioned this pull request Apr 26, 2023

Add high-level overview to docs #262

Open

forsyth2 force-pushed the n247-symlinks branch 2 times, most recently from c952c91 to 400be37 Compare April 28, 2023 13:46

Include symlinks

a6dee77

forsyth2 force-pushed the n247-symlinks branch from 400be37 to a6dee77 Compare April 28, 2023 13:50

forsyth2 marked this pull request as ready for review April 28, 2023 14:01

forsyth2 merged commit a094796 into main Apr 28, 2023

forsyth2 deleted the n247-symlinks branch April 28, 2023 14:01

forsyth2 mentioned this pull request Apr 28, 2023

Docs for follow-symlinks #263

Merged

forsyth2 mentioned this pull request Aug 2, 2024

[Bug]: zstash with --follow-symlinks clobbers the source directory #341

Closed

Include symlinks #261

Include symlinks #261

Conversation

forsyth2 commented Mar 28, 2023

forsyth2 commented Mar 28, 2023

TonyB9000 commented Mar 28, 2023

forsyth2 commented Mar 31, 2023

TonyB9000 commented Mar 31, 2023

forsyth2 commented Mar 31, 2023

TonyB9000 commented Mar 31, 2023 • edited Loading

forsyth2 commented Mar 31, 2023

TonyB9000 commented Mar 31, 2023

TonyB9000 commented Mar 31, 2023 • edited Loading

forsyth2 commented Apr 3, 2023

TonyB9000 commented Apr 3, 2023

forsyth2 commented Apr 3, 2023

TonyB9000 commented Apr 3, 2023

forsyth2 commented Apr 3, 2023 • edited Loading

TonyB9000 commented Apr 4, 2023 • edited Loading

forsyth2 commented Apr 4, 2023

TonyB9000 commented Apr 4, 2023 • edited Loading

forsyth2 commented Apr 7, 2023

forsyth2 commented Apr 7, 2023

forsyth2 commented Apr 7, 2023

forsyth2 left a comment

Choose a reason for hiding this comment

TonyB9000 commented Apr 14, 2023

forsyth2 commented Apr 14, 2023

TonyB9000 commented Apr 14, 2023

TonyB9000 commented Apr 14, 2023

forsyth2 commented Apr 19, 2023

TonyB9000 commented Apr 19, 2023 • edited Loading

forsyth2 commented Apr 21, 2023 • edited Loading

forsyth2 Apr 21, 2023

Choose a reason for hiding this comment

TonyB9000 commented Apr 25, 2023

golaz commented Apr 25, 2023

TonyB9000 commented Apr 25, 2023

forsyth2 commented Apr 26, 2023

forsyth2 commented Apr 28, 2023

TonyB9000 commented Apr 28, 2023 • edited Loading

forsyth2 commented Apr 28, 2023

TonyB9000 commented Apr 28, 2023

forsyth2 commented Apr 28, 2023

TonyB9000 commented Mar 31, 2023 •

edited

Loading

TonyB9000 commented Mar 31, 2023 •

edited

Loading

forsyth2 commented Apr 3, 2023 •

edited

Loading

TonyB9000 commented Apr 4, 2023 •

edited

Loading

TonyB9000 commented Apr 4, 2023 •

edited

Loading

TonyB9000 commented Apr 19, 2023 •

edited

Loading

forsyth2 commented Apr 21, 2023 •

edited

Loading

TonyB9000 commented Apr 28, 2023 •

edited

Loading