Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[No Merge] Update sdk #35480

Closed
wants to merge 1 commit into from
Closed

Conversation

ViktorHofer
Copy link
Member

@ViktorHofer ViktorHofer commented Apr 26, 2020

When using the latest SDK, Helix fails to upload the workitems because of malformed headers.

@alexperovich @lukas-lansky @MattGal can someone please take a look at the helix failures? We want to upgrade to a newer SDK with #35202.


```.packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20221.14/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : (NETCORE_ENGINEERING_TELEMETRY=Helix) StorageException: The value for one of the HTTP headers is not in the correct format.
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass28_0.<<UploadFromStreamAsyncHelper>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.DotNet.Helix.Client.ContainerBase.UploadFileAsync(Stream stream, String blobName, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/StorageHelpers/ContainerBase.cs:line 23
   at Microsoft.DotNet.Helix.Client.DirectoryPayload.DoUploadAsync(IBlobContainer payloadContainer, Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/Payloads/DirectoryPayload.cs:line 100
   at Microsoft.DotNet.Helix.Client.DirectoryPayload.UploadAsync(IBlobContainer payloadContainer, Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/Payloads/DirectoryPayload.cs:line 55
   at Microsoft.DotNet.Helix.Client.JobDefinition.<>c__DisplayClass74_0.<<SendAsync>b__0>d.MoveNext() in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 160
--- End of stack trace from previous location ---
   at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 159
   at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs:line 214
   at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 52

@ViktorHofer ViktorHofer added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Apr 26, 2020
@ghost
Copy link

ghost commented Apr 26, 2020

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

@ViktorHofer
Copy link
Member Author

cc @dotnet/ncl in case something changed in the header handling.

@wfurt
Copy link
Member

wfurt commented Apr 26, 2020

Is the failure consistent @ViktorHofer? There are some changes in known headers that may be relevant. cc @stephentoub

@ViktorHofer
Copy link
Member Author

ViktorHofer commented Apr 26, 2020

Yes, failures are consistent enough to appear in multiple legs in every attempt.

@stephentoub
Copy link
Member

stephentoub commented Apr 26, 2020

That failure message comes from the Azure storage service. We may be mishandling one of its headers.

@wfurt
Copy link
Member

wfurt commented Apr 26, 2020

I can reproduce it locally. I will take a look.

MININT-TRCM1I5:wfurt-runtime2 furt$ ./eng/common/msbuild.sh src/libraries/sendtohelix.proj /t:test /p:HelixTargetQueues=OSX.1014.Amd64.Open   /p:TimeoutInSeconds=1200  /p:WaitForWorkItemCompletion=true /p:HelixSource=tlsAlert /p:creator=wfurt
  Using Queues: OSX.1014.Amd64.Open
  Build TargetFramework: netcoreapp5.0
  Using Queues: OSX.1014.Amd64.Open
  Build TargetFramework: netcoreapp5.0
  Uploading payloads for Job on OSX.1014.Amd64.Open...
  Finished uploading payloads for Job on OSX.1014.Amd64.Open...
  Sending Job to OSX.1014.Amd64.Open...
/Users/furt/.nuget/packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20221.14/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : StorageException: The value for one of the HTTP headers is not in the correct format. [/Users/furt/github/wfurt-runtime2/src/libraries/sendtohelix.proj]
/Users/furt/.nuget/packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20221.14/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) [/Users/furt/github/wfurt-runtime2/src/libraries/sendtohelix.proj]
/Users/furt/.nuget/packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20221.14/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass28_0.<<UploadFromStreamAsyncHelper>b__0>d.MoveNext() [/Users/furt/github/wfurt-runtime2/src/libraries/sendtohelix.proj]
/Users/furt/.nuget/packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20221.14/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : --- End of stack trace from previous location --- [/Users/furt/github/wfurt-runtime2/src/libraries/sendtohelix.proj]
/Users/furt/.nuget/packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20221.14/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.DotNet.Helix.Client.ContainerBase.UploadFileAsync(Stream stream, String blobName, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/StorageHelpers/ContainerBase.cs:line 23 [/Users/furt/github/wfurt-runtime2/src/libraries/sendtohelix.pr

it should be easy to trace it down now.

@stephentoub
Copy link
Member

Thanks.

@wfurt
Copy link
Member

wfurt commented Apr 27, 2020

This is failing because we fail to send x-ms-version header and that breaks the blob storage protocol.

@ViktorHofer
Copy link
Member Author

Closing as @wfurt found the issue.

@ViktorHofer ViktorHofer deleted the UpdateSdk branch April 27, 2020 20:49
@ghost ghost locked as resolved and limited conversation to collaborators Dec 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants