Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add session_cert and session_verify parameters to WebHDFS class (enables mTLS) #1472

Merged
merged 1 commit into from
Jan 4, 2024

Conversation

markhatch
Copy link
Contributor

@markhatch markhatch commented Dec 18, 2023

Provides mechanism to supply certificates in the WebHDFS session.

@martindurant
Copy link
Member

I feel like this could be done simpler with optional kwargs to the existing webHDFS rather than making a new class. Since fsspec keys URLs by their protocol prefixes, this would ensure that "webhdfs" would serve both uses, but allow for the extra arguments to be defined in config.

@markhatch
Copy link
Contributor Author

I feel like this could be done simpler with optional kwargs to the existing webHDFS rather than making a new class. Since fsspec keys URLs by their protocol prefixes, this would ensure that "webhdfs" would serve both uses, but allow for the extra arguments to be defined in config.

Sounds good, I'll take a stab at that!

@martindurant
Copy link
Member

(please ping me when you want another look)

@markhatch markhatch changed the title Add subclass of WebHDFS that supports mTLS Add session_cert and session_verify parameters to WebHDFS class (enables mTLS) Dec 26, 2023
@markhatch
Copy link
Contributor Author

Had to add #noqa to the url line on 105 to pass pre-commit, error didn't make sense to me (and I didn't change the line)

fsspec/implementations/webhdfs.py:105:56: E231 missing whitespace after ':'
fsspec/implementations/webhdfs.py:105:65: E231 missing whitespace after ':'

@markhatch
Copy link
Contributor Author

@martindurant think the changes are ready. Initially thought I could just pass session_kwargs through, but guess that requests.Session doesn't have init kwargs so used explicit parameters instead.

@martindurant
Copy link
Member

Had to add #noqa

I suppose it's OK, so long as the code is legible. I'm surprised it wasn't simply fixed by black rather than being picked up by pep8.

@martindurant martindurant merged commit 0ca1762 into fsspec:master Jan 4, 2024
10 checks passed
@markhatch markhatch deleted the webhdfs_mtls branch January 8, 2024 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants