CryptoNative_GetX509NameRawBytes assert failure in CI

[stephentoub]

This assert:

runtime/src/libraries/Native/Unix/System.Security.Cryptography.Native/openssl.c

Lines 391 to 404 in 57bfe47

	/*
	* length is size_t on some platforms and int on others, so the comparisons
	* are not tautological everywhere. We can let the compiler optimize away
	* any part of the check that is. We split the size checks into two checks
	* so we can get around the warnings on Linux where the Length is unsigned
	* whereas Length is signed on OS X. The first check makes sure the variable
	* value is less than INT_MAX in it's native format; once we know it is not
	* too large, we can safely cast to an int to make sure it is not negative
	*/
	if (nameBufLen > INT_MAX)
	{
	assert(0 && "Huge length X509_NAME");
	return 0;
	}

failed here:

net6.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open
https://dev.azure.com/dnceng/public/_build/results?buildId=1215774&view=ms.vss-test-web.build-test-results-tab&runId=36284970&paneView=debug
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-51653-merge-c7295d0c68554d41ae/System.Security.Cryptography.X509Certificates.Tests/console.6e780ff5.log?sv=2019-07-07&se=2021-07-21T14%3A13%3A18Z&sr=c&sp=rl&sig=82j5HfZOmJPAZLgdXSXr5OMinlqsbJ1YQcdl9J1mYdk%3D

  Discovering: System.Security.Cryptography.X509Certificates.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.X509Certificates.Tests (found 606 of 700 test cases)
  Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 2)
dotnet: /__w/1/s/src/libraries/Native/Unix/System.Security.Cryptography.Native/openssl.c:402: int32_t CryptoNative_GetX509NameRawBytes(X509_NAME *, uint8_t *, int32_t): Assertion `0 && "Huge length X509_NAME"' failed.
./RunTests.sh: line 162: 14474 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.X509Certificates.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.X509Certificates.Tests.deps.json xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This assert:

runtime/src/libraries/Native/Unix/System.Security.Cryptography.Native/openssl.c

Lines 391 to 404 in 57bfe47

	/*
	* length is size_t on some platforms and int on others, so the comparisons
	* are not tautological everywhere. We can let the compiler optimize away
	* any part of the check that is. We split the size checks into two checks
	* so we can get around the warnings on Linux where the Length is unsigned
	* whereas Length is signed on OS X. The first check makes sure the variable
	* value is less than INT_MAX in it's native format; once we know it is not
	* too large, we can safely cast to an int to make sure it is not negative
	*/
	if (nameBufLen > INT_MAX)
	{
	assert(0 && "Huge length X509_NAME");
	return 0;
	}

failed here:

net6.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open
https://dev.azure.com/dnceng/public/_build/results?buildId=1215774&view=ms.vss-test-web.build-test-results-tab&runId=36284970&paneView=debug
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-51653-merge-c7295d0c68554d41ae/System.Security.Cryptography.X509Certificates.Tests/console.6e780ff5.log?sv=2019-07-07&se=2021-07-21T14%3A13%3A18Z&sr=c&sp=rl&sig=82j5HfZOmJPAZLgdXSXr5OMinlqsbJ1YQcdl9J1mYdk%3D

  Discovering: System.Security.Cryptography.X509Certificates.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.X509Certificates.Tests (found 606 of 700 test cases)
  Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 2)
dotnet: /__w/1/s/src/libraries/Native/Unix/System.Security.Cryptography.Native/openssl.c:402: int32_t CryptoNative_GetX509NameRawBytes(X509_NAME *, uint8_t *, int32_t): Assertion `0 && "Huge length X509_NAME"' failed.
./RunTests.sh: line 162: 14474 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.X509Certificates.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.X509Certificates.Tests.deps.json xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

Author:	stephentoub
Assignees:	-
Labels:	area-System.Security, bug
Milestone:	6.0.0

[vcsjones]

Looks like a dupe of #49732.

[stephentoub]

Closing as dup of #49732.