Add Compy endpoint to Globus file #274

forsyth2 · 2023-06-19T12:28:51Z

Add Compy endpoint to Globus file.

In https://github.com/E3SM-Project/zstash/blob/main/zstash/globus.py, add Compy's Globus endpoint.

mahf708 · 2023-06-19T13:30:35Z

See #275 and please feel free to edit the PR without consultation :)

mahf708 · 2023-06-19T13:52:05Z

Note that, socket.getfqdn() returns 'n0001.local'... on compy compute nodes

forsyth2 · 2023-07-05T23:13:45Z

@mahf708 Did you (or can you) run the following on Compy?

$ cd zstash
# <Activate a conda dev environment>
$ pip install .
$ python -m unittest tests/test_globus.py

I'm getting the following error on both main and this branch.

======================================================================
ERROR: testLs (tests.test_globus.TestGlobus)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 183, in testLs
    self.helperLsGlobus(
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 161, in helperLsGlobus
    self.preactivate_globus()
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 62, in preactivate_globus
    native_client.login(no_local_server=True, refresh_tokens=True)
  File "/qfs/people/fors729/mambaforge/envs/zstash_dev/lib/python3.9/site-packages/fair_research_login/client.py", line 157, in login
    auth_code = self.get_code(requested_scopes, refresh_tokens,
  File "/qfs/people/fors729/mambaforge/envs/zstash_dev/lib/python3.9/site-packages/fair_research_login/client.py", line 197, in get_code
    auth_url = self.client.oauth2_get_authorize_url(
TypeError: oauth2_get_authorize_url() got an unexpected keyword argument 'additional_params'

That error message isn't terribly helpful. (It looks like my Globus endpoint is activated: https://app.globus.org/file-manager/collections/68fbd2fa-83d7-11e9-8e63-029d279f7e24/overview)

Perhaps related, using the latest Unified RC, I get:

$ source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.9.0rc6_compy.sh
$ python -m unittest tests/test_globus.py
Removing test files, both locally and at the HPSS repo
E
======================================================================
ERROR: testLs (tests.test_globus.TestGlobus)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 183, in testLs
    self.helperLsGlobus(
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 161, in helperLsGlobus
    self.preactivate_globus()
  File "/qfs/people/fors729/zstash/tests/test_globus.py", line 69, in preactivate_globus
    r = self.transfer_client.endpoint_autoactivate(ep_id, if_expires_in=600)
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/services/transfer/client.py", line 458, in endpoint_autoactivate
    return self.post(
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/client.py", line 157, in post
    return self.request(
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/client.py", line 294, in request
    r = self.transport.request(
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/transport/requests.py", line 320, in request
    self._set_authz_header(authorizer, req)
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/transport/requests.py", line 252, in _set_authz_header
    authz_header = authorizer.get_authorization_header()
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 171, in get_authorization_header
    self.ensure_valid_token()
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 165, in ensure_valid_token
    self._get_new_access_token()
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 125, in _get_new_access_token
    res = self._get_token_response()
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/authorizers/refresh_token.py", line 80, in _get_token_response
    return self.auth_client.oauth2_refresh_token(self.refresh_token)
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/services/auth/client/native_client.py", line 130, in oauth2_refresh_token
    return self.oauth2_token(form_data, body_params=body_params)
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/services/auth/client/base.py", line 474, in oauth2_token
    self.post(
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/client.py", line 157, in post
    return self.request(
  File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.9.0rc6_login/lib/python3.10/site-packages/globus_sdk/client.py", line 312, in request
    raise self.error_class(r)
globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')

mahf708 · 2023-07-06T02:57:52Z

@forsyth2 I will test tomorrow or Friday; if you don't hear from me, please nudge me again (I actually don't like using compy because it is so annoying lol ... but I also need to get some files from compy to NERSC so this is a good chance to test)

forsyth2 · 2023-07-11T21:56:50Z

Ah, I knew I had seen this error before. Searching GitHub, I found additional_params occurs in #201 (comment) and #250 (comment).

In #250 (comment), Xylar says:

it seems like the version of fair-research-login` is most likely to be responsible

$ git grep -n fair
conda/dev.yml:12:  - fair-research-login=0.2.0
conda/meta.yaml:24:    - fair-research-login >=0.2.6,<0.3.0
tests/test_globus.py:8:from fair_research_login.client import NativeClient
zstash/globus.py:9:from fair_research_login.client import NativeClient

Changing fair-research-login=0.2.0 to 0.2.6 (and creating a new mamba environment) just seems to make the test hang however.

@mahf708 If you can run the test on Compy and see if you run into the same issue, that would be helpful in my debugging. Thanks!

mahf708 · 2023-07-11T22:41:50Z

Sorry, I let you down on this. I am on it right now. I will report back shortly.

mahf708 · 2023-07-11T22:46:05Z

{
  "context": [
    {
      "operation": "Directory List / File Scan",
      "path": "/qfs/people/mahf708/e3sm/zstash/zstash_test/zstash/index.db"
    }
  ],
  "error": {
    "body": "550-GlobusError: v=1 c=PATH_NOT_FOUND%0D%0A550-GridFTP-Errno: 2%0D%0A550-GridFTP-Reason: System error in stat%0D%0A550-GridFTP-Error-String: No such file or directory%0D%0A550 End.%0D%0A",
    "code": 550,
    "endpoint": "pic#compy-dtn (68fbd2fa-83d7-11e9-8e63-029d279f7e24)",
    "server": "gsiftp://compy-dtn.pnl.gov:2811",
    "type": "FTPServerError"
  }
}

I cannot reproduce your errors, but I get the above...

env:

(e3sm_unified_1.9.0rc6_login) [mahf708@compy01 ~]$ mamba list | grep zstash
zstash                    1.4.0rc2           pyh9caca29_0    conda-forge/label/zstash_dev
(e3sm_unified_1.9.0rc6_login) [mahf708@compy01 ~]$ mamba list | grep fair
fair-research-login       0.2.6              pyhd8ed1ab_0    conda-forge
(e3sm_unified_1.9.0rc6_login) [mahf708@compy01 ~]$

mahf708 · 2023-07-11T22:49:10Z

@forsyth2

can you please check your https://app.globus.org/activity to see if the file transfers are hanging there?
is this an error you've seen before? Where some files are missing?

forsyth2 · 2023-07-11T22:56:26Z

Sorry, I let you down on this. I am on it right now.

No worries, thanks!

I cannot reproduce your errors, but I get the above...

can you please check your https://app.globus.org/activity to see if the file transfers are hanging there?

I get the same errors as in your image. Clicking "view details", I get:

{
  "context": [
    {
      "operation": "Directory List / File Scan",
      "path": "/qfs/people/fors729/zstash/zstash_test/zstash/000000.tar"
    }
  ],
  "error": {
    "body": "550-GlobusError: v=1 c=PATH_NOT_FOUND%0D%0A550-GridFTP-Errno: 2%0D%0A550-GridFTP-Reason: System error in stat%0D%0A550-GridFTP-Error-String: No such file or directory%0D%0A550 End.%0D%0A",
    "code": 550,
    "endpoint": "pic#compy-dtn (68fbd2fa-83d7-11e9-8e63-029d279f7e24)",
    "server": "gsiftp://compy-dtn.pnl.gov:2811",
    "type": "FTPServerError"
  }
}

I wasn't aware of the extra detail there; that matches what you got.

is this an error you've seen before? Where some files are missing?

I don't think so? At least, it wasn't a big enough problem to remember. And almost certainly not on this small test case.

mahf708 · 2023-07-11T23:00:31Z

Do you see anything in the diffs that may be helpful? v1.3.0...v1.4.0rc2

mahf708 · 2023-07-11T23:02:22Z

The type of error tells me it's either you changed settings/names or it is an external auth library...

mahf708 · 2023-07-11T23:03:56Z

The fair-research-login diff:

-  fair-research-login            0.3.0         pyhd8ed1ab_0                   conda-forge
+  fair-research-login            0.2.6         pyhd8ed1ab_0                   conda-forge

forsyth2 · 2023-07-11T23:12:46Z

@mahf708 Thank you for pointing out the diff from the latest release version. I just ran on the latest official release of Unified, and the Globus test actually gives the same result.

That seems to indicate something happened with the way Globus, fair-research-login, or perhaps another third-party package work internally.

forsyth2 · 2023-07-11T23:13:45Z

Actually, anything involving a package wouldn't make sense, because we have the version pinned. That means it must be something that changed with Globus, right?

mahf708 · 2023-07-11T23:14:16Z

I see, please see below the globus changes:

- globus-cli                      3.10.1        pyhd8ed1ab_0                   conda-forge
- globus-sdk                      3.15.0        pyhd8ed1ab_0                   conda-forge
+ globus-cli                      3.15.0        pyhd8ed1ab_0                   conda-forge
+ globus-sdk                      3.21.0        pyhd8ed1ab_0                   conda-forge

forsyth2 · 2023-07-11T23:15:41Z

But these are still packages. We know zstash worked on the last release. And now that same exact code (and dependencies!) does not work.

mahf708 · 2023-07-11T23:18:59Z

Let me test something for you, just give me a second.

Does this fail similarly anywhere else or just here?

forsyth2 · 2023-07-11T23:20:51Z

The Globus test on Compy is the only issue I'm having with the upcoming zstash release. This test appears to fail consistently regardless of what commit or dev environment is being used.

forsyth2 · 2023-07-11T23:21:51Z

(I suppose now I should have made these comments in a separate issue, but I thought this endpoint change might be the necessary fix. Apparently that's not the case).

mahf708 · 2023-07-11T23:26:31Z

Btw, I just tested and it fails similarly with older release zstash / e3sm-u... not sure how to help 😢

mahf708 · 2023-07-11T23:26:51Z

(I also tried different permutations of the libraries to no avail)

forsyth2 · 2023-07-11T23:30:06Z

@mahf708 Thank you for helping to debug. I'm going to open an issue specifically to address this, since it's now clearly unrelated to this pull request.

forsyth2 · 2023-07-11T23:36:46Z

I created #279. Unfortunately, I can't test this pull request on Compy until that is resolved.

mahf708 · 2023-07-12T02:01:35Z

No worries about this PR. It's not essential and the workaround is pretty straightforward.

forsyth2 added this to forsyth2 current tasks Jun 19, 2023

github-project-automation bot moved this to Todo in forsyth2 current tasks Jun 19, 2023

mahf708 mentioned this issue Jun 19, 2023

add compy (login and compute) and chrlogin to globus endpoints #275

Merged

1 task

forsyth2 mentioned this issue Jul 11, 2023

Globus fails on Compy, regardless of version #279

Closed

forsyth2 mentioned this issue Jul 21, 2023

Update fair-research-login #284

Merged

forsyth2 closed this as completed in #275 Jul 21, 2023

github-project-automation bot moved this from Todo to Done in forsyth2 current tasks Jul 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Compy endpoint to Globus file #274

Add Compy endpoint to Globus file #274

forsyth2 commented Jun 19, 2023

mahf708 commented Jun 19, 2023

mahf708 commented Jun 19, 2023 •

edited

Loading

forsyth2 commented Jul 5, 2023

mahf708 commented Jul 6, 2023

forsyth2 commented Jul 11, 2023 •

edited

Loading

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023 •

edited

Loading

mahf708 commented Jul 11, 2023 •

edited

Loading

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023 •

edited

Loading

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023 •

edited

Loading

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 12, 2023

Add Compy endpoint to Globus file #274

Add Compy endpoint to Globus file #274

Comments

forsyth2 commented Jun 19, 2023

mahf708 commented Jun 19, 2023

mahf708 commented Jun 19, 2023 • edited Loading

forsyth2 commented Jul 5, 2023

mahf708 commented Jul 6, 2023

forsyth2 commented Jul 11, 2023 • edited Loading

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023 • edited Loading

mahf708 commented Jul 11, 2023 • edited Loading

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023 • edited Loading

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

mahf708 commented Jul 11, 2023

forsyth2 commented Jul 11, 2023 • edited Loading

forsyth2 commented Jul 11, 2023

mahf708 commented Jul 12, 2023

mahf708 commented Jun 19, 2023 •

edited

Loading

forsyth2 commented Jul 11, 2023 •

edited

Loading

mahf708 commented Jul 11, 2023 •

edited

Loading

mahf708 commented Jul 11, 2023 •

edited

Loading

mahf708 commented Jul 11, 2023 •

edited

Loading

forsyth2 commented Jul 11, 2023 •

edited

Loading