Spark 3.5: Use File.separator instead of hardcode in RewriteTablePath
#12066
Conversation
| if (!path.startsWith(toRemove)) { | ||
| throw new IllegalArgumentException( | ||
| String.format("Path %s does not start with %s", path, toRemove)); | ||
| } | ||
| return path.substring(toRemove.length()); | ||
| } | ||
|
|
||
| public static String appendFileSeparatorIfNotExists(String path) { |
There was a problem hiding this comment.
maybe maybeAppendFileSeparator?
|
Thanks @manuzhang , i saw this too when working on OSSing the original implementation and thought about this but never got around to simplifying it. One doubt I had was, File.separator is potentially different when running on different platform, do we care? Shouldnt we preserve the path/ make the same path if we run on different plartform? |
You are right @szehon-ho! This is a mistake on my part. We probably can't do this because the Java ENV in use by the application doesn't necessarily match the files system being used. (almost certainly does not) For example should my path on S3 be different if I make my Table using an application running from a windows machine vs a linux machine? |
No description provided.