-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make AzureProperties w/ shared-key creds serializable #10045
Conversation
@snazy could you add a test please? |
any eta on when this would be merged? |
e4e3fed
to
72fd23e
Compare
}) | ||
.doesNotThrowAnyException(); | ||
|
||
byte[] serialized = out.toByteArray(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it would be good to actually create an instance of AzureProperties
and verify that this matches props
. Additionally, you can use
public static <T> T roundTripSerialize(T type) throws IOException, ClassNotFoundException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
something like
AzureProperties props =
new AzureProperties(
ImmutableMap.<String, String>builder()
.put(ADLS_SAS_TOKEN_PREFIX + "foo", "bar")
.put(ADLS_CONNECTION_STRING_PREFIX + "foo", "bar")
.put(ADLS_READ_BLOCK_SIZE, "42")
.put(ADLS_WRITE_BLOCK_SIZE, "42")
.put(ADLS_SHARED_KEY_ACCOUNT_NAME, "me")
.put(ADLS_SHARED_KEY_ACCOUNT_KEY, "secret")
.build());
AzureProperties serdedProps = TestHelpers.roundTripSerialize(props);
assertThat(serdedProps.adlsReadBlockSize()).isEqualTo(props.adlsReadBlockSize());
assertThat(serdedProps.adlsWriteBlockSize()).isEqualTo(props.adlsWriteBlockSize());
assertThat(serdedProps.adlsSasTokens()).isEqualTo(props.adlsSasTokens());
assertThat(serdedProps.namedKeyCreds()).isEqualTo(props.namedKeyCreds());
assertThat(serdedProps.adlsConnectionStrings()).isEqualTo(props.adlsConnectionStrings());
22629aa
to
c069f11
Compare
As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark.
c069f11
to
99c1d2f
Compare
…10045) As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark.
…10045) As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark.
As
StorageSharedKeyCredential
is not serializable, shared key auth doesn't work with Spark.