Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Client encryption]: Drops RentArrayBufferWriter #4847

Open
wants to merge 100 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
d6f89ea
Add baseline benchmarks for `Microsoft.Azure.Cosmos.Encryption.Custom`
juraj-blazek Sep 12, 2024
8928b88
Cleanup
juraj-blazek Sep 12, 2024
9dc7d54
Use set of static test data for benchmarks
juraj-blazek Sep 12, 2024
368dc25
Merge branch 'master' into users/juraj-blazek/encryption-benchmark-ba…
juraj-blazek Sep 12, 2024
38a198a
Add non-allocating APIs to encryptors
juraj-blazek Sep 16, 2024
e0bb8bf
WIP
juraj-blazek Sep 16, 2024
9c3c276
Revert solution update
juraj-blazek Sep 16, 2024
c37a092
Merge branch 'users/juraj-blazek/encryption-benchmark-baseline' of ht…
juraj-blazek Sep 16, 2024
f4db1c6
Merge branch 'master' into users/juraj-blazek/encryption-benchmark-ba…
juraj-blazek Sep 16, 2024
91714f9
Merge branch 'master' into users/juraj-blazek/encryption-benchmark-ba…
juraj-blazek Sep 17, 2024
2058f8b
Merge remote-tracking branch 'origin/master' into users/juraj-blazek/…
juraj-blazek Sep 17, 2024
f33538c
Add implementation, fix tests
juraj-blazek Sep 17, 2024
6923fd2
Switch to randomized encryption for benchmarks
juraj-blazek Sep 17, 2024
4ea8690
Merge branch 'master' into users/juraj-blazek/encryption-benchmark-ba…
kr-santosh Sep 18, 2024
03ef682
Some more array pooling
juraj-blazek Sep 17, 2024
1044a89
Streaming deserialization
juraj-blazek Sep 19, 2024
f1db54d
Merge remote-tracking branch 'origin/master' into users/juraj-blazek/…
juraj-blazek Sep 19, 2024
05bfc50
Cleanup
juraj-blazek Sep 20, 2024
5629f74
Update MDE and rerun benchmarks
juraj-blazek Sep 20, 2024
495d2c4
Add non-allocating APIs to encryptors
juraj-blazek Sep 16, 2024
b3f74b1
Merge branch 'master' into users/juraj-blazek/non-allocating-encryptor
Sep 30, 2024
14bce37
~ drop repeated DEK calls
Sep 30, 2024
4ff1601
! typo
Sep 30, 2024
d8a345c
~ update benchmark
Sep 30, 2024
03c06e0
~ fix tests
Sep 30, 2024
a1025c4
Merge branch 'Azure:master' into users/juraj-blazek/non-allocating-en…
JanHyka Oct 1, 2024
3bf77c8
~ cleanup
Oct 1, 2024
ceaa8b5
+ refresh benchmark
Oct 1, 2024
611b3ac
+ unit test
Oct 1, 2024
9ca89d4
~ merge predecessor PR
Oct 1, 2024
8a78fe8
~ merge fixes and initial cleanup
Oct 1, 2024
8ed2135
~ write directly to output document instead of copying
Oct 1, 2024
bbe9845
! tests
Oct 1, 2024
a107f62
~ retrieve DataEncryptionKey only once per document
Oct 1, 2024
a1ad02b
! fix tests
Oct 1, 2024
4f2f072
~ update Aes algorithm to reuse GetEncryptedByteCount
Oct 2, 2024
cbbeee2
~ refactor EncryptionProcessor
Oct 4, 2024
b6c851c
! names
Oct 4, 2024
72ccae7
~ less static
Oct 4, 2024
8ea5879
Merge branch 'master' into users/juraj-blazek/encryption-array-pooling
Oct 6, 2024
8c60e69
Merge branch 'master' into users/juraj-blazek/encryption-array-pooling
Oct 6, 2024
5554aa0
~ merge fixes
Oct 6, 2024
28620ed
~ cleanup
Oct 6, 2024
eb059c8
~ unwanted changes
Oct 6, 2024
cc2eab5
- unused method
Oct 6, 2024
c9ba300
~ updates (PR)
Oct 6, 2024
9f9cbca
~ add stable vs preview release duplicity
Oct 7, 2024
c347e71
Merge branch 'users/juraj-blazek/encryption-array-pooling' into users…
Oct 7, 2024
64172b8
~ cleanup and parent branch merge
Oct 7, 2024
ff583f9
Merge branch 'master' into users/jan-hyka/refactor-encryption-processor
Oct 7, 2024
0ba8c34
Merge branch 'master' into users/jan-hyka/refactor-encryption-processor
Oct 7, 2024
326b1be
~ master merges
Oct 7, 2024
c520e16
- duplicate
Oct 7, 2024
5c40821
~ cleanup
Oct 7, 2024
99c7a75
+ Add .NET8.0 target for Cosmos.Encryption.Custom
Oct 8, 2024
31c20e7
- remove implicit IsPreview from csproj
Oct 8, 2024
90b2cec
+ JsonNodeSqlSerializer
Oct 9, 2024
6f27d9d
+ initial commit
Oct 9, 2024
d7c06d6
! EncryptioProperties System.Text.Json annotations
Oct 9, 2024
cb9d166
+ bump benchmark
Oct 9, 2024
5fc5165
~ wip
Oct 9, 2024
c9e7f30
+ initial for JsonNode decryption
Oct 9, 2024
9215f67
~ Merge branch 'master' into users/jan-hyka/jsonnode_serializer
Oct 10, 2024
fed6a3a
~ polishing and benchmark refresh
Oct 10, 2024
38cc928
- remove explicit System.Text.Json 8.0.5
Oct 10, 2024
ba4f73b
Merge branch 'users/jan-hyka/jsonnode_serializer' into users/jan-hyka…
Oct 10, 2024
424028c
Merge branch 'master' into users/jan-hyka/jsonnode_encryptionProcessor
Oct 10, 2024
2af9ddb
~ propagate changes from master
Oct 10, 2024
30c18d0
Merge branch 'users/jan-hyka/jsonnode_encryptionProcessor' into users…
Oct 10, 2024
54b0ad9
~ fixes for preview x non-preview branching
Oct 10, 2024
69c8978
Merge branch 'master' into users/jan-hyka/jsonnode_encryptionProcessor
JanHyka Oct 11, 2024
ad15ceb
+ tests
Oct 11, 2024
a0ccd44
Merge branch 'master' into users/jan-hyka/jsonnode_encryptionProcessor
Oct 13, 2024
d9c315d
~ complete merge from master & update benchmark
Oct 13, 2024
b11db28
Merge branch 'users/jan-hyka/jsonnode_encryptionProcessor' into users…
Oct 13, 2024
996252b
+ initial
Oct 14, 2024
758e98d
+ tests, fixes and benchmark
Oct 14, 2024
420a35d
! bugfix
Oct 14, 2024
d08be05
~ benchmark update
Oct 14, 2024
0bf29b8
+ cleanup
Oct 15, 2024
2b2209f
+ stream serializer
Oct 15, 2024
f03c92d
Merge branch 'master' into users/jan-hyka/stream_deserializer
Oct 15, 2024
42462d5
~ preview branching fixes
Oct 15, 2024
703830f
+ add support for Stream deserialization of obsoleted encryption algo…
Oct 15, 2024
4319524
~ final touches
Oct 16, 2024
940d667
Merge branch 'master' into users/jan-hyka/stream_deserializer
Oct 17, 2024
fa0020c
~ PR based changes
Oct 18, 2024
8e45769
Merge branch 'master' into users/jan-hyka/stream_deserializer
kirankumarkolli Oct 18, 2024
c98f2bf
- drop JsonNode serializer/deserializer
Oct 21, 2024
3be6843
+ rms
Oct 21, 2024
de555a8
- drop rentArrayBufferWriter
Oct 21, 2024
428f2e1
Merge branch 'master' into users/jan-hyka/drop_jsonnode
Oct 24, 2024
8ce3622
~ fix the mess after merge
Oct 24, 2024
10ad1d4
~ fix merge mess
Oct 24, 2024
81394b2
- drop JsonNode based benchmarks
Oct 24, 2024
a439cf2
Merge branch 'users/jan-hyka/drop_jsonnode' into users/jan-hyka/drop_…
Oct 24, 2024
b4da46c
- drop files we don't need yet
Oct 24, 2024
98adb5c
Merge branch 'master' into users/jan-hyka/drop_rentarraybufferwriter
kirankumarkolli Oct 25, 2024
ff5a844
~ reference RecyclableMemoryStream 3.0.1 package instead
JanHyka Oct 25, 2024
8a20aa1
Merge branch 'master' into users/jan-hyka/drop_rentarraybufferwriter
juraj-blazek Nov 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@

<ItemGroup Condition=" '$(IsPreview)' == 'True' ">
<PackageReference Include="Microsoft.Data.Encryption.Cryptography" Version="2.0.0-pre007" />
<PackageReference Include="Microsoft.IO.RecyclableMemoryStream" Version="3.0.1" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecyclableMemoryStream is outside of Azure core has allowed list of dependencies.
One options is to inline the code as Jan has before.

https://azure.github.io/azure-sdk/dotnet_introduction.html#dotnet-dependencies

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see where that requirement comes from... to avoid dependency hell. The question is whether avoiding one evil doesn't invite much greater one instead:

  1. Copypasted code leads to bloating of code base and rises maintenance costs
  2. Nobody maintains it or keeps it up to date (or rather, nobody can afford to)
  3. Security issues within original code will be entirely missed in such copies. While affected library gets marked by security tools (or .NET9 analyzers that catch those already at compile), hidden copy will go unnoticed... until it gets exploited.

I am not saying referencing some random 3rd party library would be a good idea, otoh entirely blacklisting pretty much everything including Microsoft.* doesn't seem neither safe nor future proof either. Might be worth consideration.

</ItemGroup>

<ItemGroup>
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,21 @@
namespace Microsoft.Azure.Cosmos.Encryption.Custom.Transformation
{
using System;
using System.Buffers;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.IO;

internal partial class StreamProcessor
{
private readonly byte[] encryptionPropertiesNameBytes = Encoding.UTF8.GetBytes(Constants.EncryptedInfo);

private readonly RecyclableMemoryStreamManager streamManager = new ();

internal async Task EncryptStreamAsync(
Stream inputStream,
Stream outputStream,
Expand Down Expand Up @@ -51,7 +55,7 @@ internal async Task EncryptStreamAsync(

Utf8JsonWriter encryptionPayloadWriter = null;
string encryptPropertyName = null;
RentArrayBufferWriter bufferWriter = null;
RecyclableMemoryStream bufferWriter = null;

while (!isFinalBlock)
{
Expand Down Expand Up @@ -112,8 +116,8 @@ long TransformEncryptBuffer(ReadOnlySpan<byte> buffer)
case JsonTokenType.StartObject:
if (encryptPropertyName != null && encryptionPayloadWriter == null)
{
bufferWriter = new RentArrayBufferWriter();
encryptionPayloadWriter = new Utf8JsonWriter(bufferWriter);
bufferWriter = new RecyclableMemoryStream(this.streamManager);
encryptionPayloadWriter = new Utf8JsonWriter((IBufferWriter<byte>)bufferWriter);
encryptionPayloadWriter.WriteStartObject();
}
else
Expand All @@ -132,25 +136,26 @@ long TransformEncryptBuffer(ReadOnlySpan<byte> buffer)
if (reader.CurrentDepth == 1 && encryptionPayloadWriter != null)
{
currentWriter.Flush();
(byte[] bytes, int length) = bufferWriter.WrittenBuffer;
byte[] bytes = bufferWriter.GetBuffer();
int length = (int)bufferWriter.Length;
ReadOnlySpan<byte> encryptedBytes = TransformEncryptPayload(bytes, length, TypeMarker.Object);
writer.WriteBase64StringValue(encryptedBytes);

encryptPropertyName = null;
#pragma warning disable VSTHRD103 // Call async methods when in an async method - this method cannot be async, Utf8JsonReader is ref struct
encryptionPayloadWriter.Dispose();
bufferWriter.Dispose();
#pragma warning restore VSTHRD103 // Call async methods when in an async method
encryptionPayloadWriter = null;
bufferWriter.Dispose();
bufferWriter = null;
}

break;
case JsonTokenType.StartArray:
if (encryptPropertyName != null && encryptionPayloadWriter == null)
{
bufferWriter = new RentArrayBufferWriter();
encryptionPayloadWriter = new Utf8JsonWriter(bufferWriter);
bufferWriter = new RecyclableMemoryStream(this.streamManager);
encryptionPayloadWriter = new Utf8JsonWriter((IBufferWriter<byte>)bufferWriter);
encryptionPayloadWriter.WriteStartArray();
}
else
Expand All @@ -164,16 +169,17 @@ long TransformEncryptBuffer(ReadOnlySpan<byte> buffer)
if (reader.CurrentDepth == 1 && encryptionPayloadWriter != null)
{
currentWriter.Flush();
(byte[] bytes, int length) = bufferWriter.WrittenBuffer;
byte[] bytes = bufferWriter.GetBuffer();
int length = (int)bufferWriter.Length;
ReadOnlySpan<byte> encryptedBytes = TransformEncryptPayload(bytes, length, TypeMarker.Array);
writer.WriteBase64StringValue(encryptedBytes);

encryptPropertyName = null;
#pragma warning disable VSTHRD103 // Call async methods when in an async method - this method cannot be async, Utf8JsonReader is ref struct
encryptionPayloadWriter.Dispose();
bufferWriter.Dispose();
#pragma warning restore VSTHRD103 // Call async methods when in an async method
encryptionPayloadWriter = null;
bufferWriter.Dispose();
bufferWriter = null;
}

Expand Down
Loading