Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Remove resolving ViewFs file path from scan validation #8829

Merged
merged 1 commit into from
Feb 26, 2025

Conversation

PHILO-HE
Copy link
Contributor

@PHILO-HE PHILO-HE commented Feb 26, 2025

What changes were proposed in this pull request?

I note Velox's HDFS filesystem allows ViewFs scheme.
https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp#L112.

Then, we can just rely on the uniform JNI call through allSupportedByRegisteredFileSystems, instead of resolving the ViewFS file path in Gluten scan validation.

How was this patch tested?

Customer validation.

@github-actions github-actions bot added the VELOX label Feb 26, 2025
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@PHILO-HE PHILO-HE changed the title [VL] Remove resolving viewFs file path from scan validation [VL] Remove resolving ViewFs file path from scan validation Feb 26, 2025
@PHILO-HE
Copy link
Contributor Author

@wangyum, @JkSelf, could you take a look?

@JkSelf
Copy link
Contributor

JkSelf commented Feb 26, 2025

@PHILO-HE We still require this support on the Gluten side. Velox's ViewFS is only supported on a single node and does not extend to cross-node clusters, making it unsuitable for @wangyum environment.

@PHILO-HE
Copy link
Contributor Author

@PHILO-HE We still require this support on the Gluten side. Velox's ViewFS is only supported on a single node and does not extend to cross-node clusters, making it unsuitable for @wangyum environment.

@JkSelf, I understand Velox doesn't fully support ViewFS, so path resolving is required before plan execution. But during scan validation, as ViewFS path will always be supported by HDFS file system if it's registered (it's guaranteed by allSupportedByRegisteredFileSystems), it seems unnecessary to do a path resolving here.

@JkSelf
Copy link
Contributor

JkSelf commented Feb 26, 2025

@PHILO-HE Oh, I understand. It makes sense to remove this conversion during the validation phase. Thanks.

Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@jinchengchenghh jinchengchenghh merged commit 46229b4 into apache:main Feb 26, 2025
50 checks passed
@wangyum
Copy link
Member

wangyum commented Feb 26, 2025

lgmt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants