Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove LittleEndian asserts #75159

Merged
merged 1 commit into from
Oct 16, 2024
Merged

Remove LittleEndian asserts #75159

merged 1 commit into from
Oct 16, 2024

Conversation

giritrivedi
Copy link
Contributor

The functionality work both on little endian and bigendian architecture. Hence removing the assert statements. These can be verified by running CommandLine test suite.

The functionality work both on little endian and bigendian architecture.
Hence removing the assert statements. These can be verified by running
CommandLine test suite.
@giritrivedi giritrivedi requested a review from a team as a code owner September 18, 2024 14:27
@dotnet-issue-labeler dotnet-issue-labeler bot added Area-Compilers untriaged Issues and PRs which have not yet been triaged by a lead labels Sep 18, 2024
@dotnet-policy-service dotnet-policy-service bot added the Community The pull request was submitted by a contributor who is not a Microsoft employee. label Sep 18, 2024
@giritrivedi
Copy link
Contributor Author

I am not sure if these failures are because of my commit. As I can see from below
link
is that they are failing with error "Insufficient memory to continue the execution of the program."

@AlekseyTs
Copy link
Contributor

I am curious what prompted this change. Are you running into the asserts in some scenarios? Could you provide details?

@giritrivedi
Copy link
Contributor Author

giritrivedi commented Sep 19, 2024

Well, if "Command line" test suits from Csharp/Test/ are run on S390x architecture , these asserts are triggered.

@AlekseyTs
Copy link
Contributor

The functionality work both on little endian and bigendian architecture. Hence removing the assert statements. These can be verified by running CommandLine test suite.

From a quick conversation with @VSadov, we probably do some bit/byte manipulations that we are not sure are correct on a big-endian machine and could never test that. Unfortunately, CommandLine tests is a very limited set of tests that do not really provide coverage for full breadth of the scenarios. If, let's say, all tests were passing, that would add more confidence. In addition, we think it would be good to explicitly go over the code in the affected components to confirm it is correct for big-endian platforms. There is no confidence that we have existing unit-tests covering the "interesting" scenarios.

@giritrivedi
Copy link
Contributor Author

The command Line test cases were failing on big-endian architectures. Upon investigating the cause of the problem, got to know that asserts were triggered. After removing these asserts, the test cases started passing.
For example taking away assert statement from ILBuilder.cs reduces around 403 failures.
Below is the trace of one such failure.

Is there anything needed to be verified on big endian architecture to ensure that these test cases are not only meant for little endian machines ?

      Microsoft.CodeAnalysis.UnitTests.LocalizationInfraTests.VerifyCulture (59ms): Error Message: System.InvalidOperationException : Assertion failed                                                                                                                                            
      Stack Trace:                                                                                                                                  
         at Microsoft.CodeAnalysis.ThrowingTraceListener.Fail(String message, String detailMessage) in 
        roslyn/src/Compilers/Test/Core/ThrowingTraceListener.cs:line 26                                                                                                             
         at System.Diagnostics.TraceInternal.Fail(String message, String detailMessage)                                                             
         at System.Diagnostics.TraceInternal.TraceProvider.Fail(String message, String detailMessage)                                               
         at System.Diagnostics.Debug.Fail(String message, String detailMessage)                                                                     
         at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage)                                                
         at System.Diagnostics.Debug.Assert(Boolean condition)                                                                                      
         at Microsoft.CodeAnalysis.CodeGen.ILBuilder..ctor(ITokenDeferral module, LocalSlotManager localSlotManager, OptimizationLevel optimizations, Boolean areLocalsZeroed) in roslyn/src/Compilers/Core/Portable/CodeGen/ILBuilder.cs:line 74                                  
         at Microsoft.CodeAnalysis.CSharp.MethodCompiler.GenerateMethodBody(PEModuleBuilder moduleBuilder, MethodSymbol method, Int32 methodOrdinal,BoundStatement block, ImmutableArray`1 lambdaDebugInfo, ImmutableArray`1 orderedLambdaRuntimeRudeEdits, ImmutableArray`1 closureDebugInfo, ImmutableArray`1 stateMachineStateDebugInfos, StateMachineTypeSymbol stateMachineTypeOpt, VariableSlotAllocator variableSlotAllocatorOpt, BindingDiagnosticBag diagnostics, DebugDocumentProvider debugDocumentProvider, ImportChain importChainOpt, Boolean emittingPdb, ImmutableArray`1 codeCoverageSpans, AsyncForwardEntryPoint entryPointOpt) in roslyn/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 1493                                                                                                                                           
         at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethod(MethodSymbol methodSymbol, Int32 methodOrdinal, ProcessedFieldInitializers& processedInitializers, SynthesizedSubmissionFields previousSubmissionFields, TypeCompilationState compilationState) in src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 1305                                                                              
         at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileNamedType(NamedTypeSymbol containingType) in roslyn/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 530                                                                                              
         at Microsoft.CodeAnalysis.CSharp.MethodCompiler.<>c__DisplayClass25_0.<CompileNamedTypeAsync>b__0() in roslyn/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 432                                                                                           
         at Roslyn.Utilities.UICultureUtilities.<>c__DisplayClass5_0.<WithCurrentUICulture>b__0() in roslyn/src/Compilers/Core/Portable/InternalUtilities/UICultureUtilities.cs:line 139                                                                                           
         at System.Threading.Tasks.Task.InnerInvoke()                                                                                               
         at System.Threading.Tasks.Task.<>c.<.cctor>b__292_0(Object obj)                                                                            
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)   
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
      --- End of stack trace from previous location ---
         at Microsoft.CodeAnalysis.CSharp.MethodCompiler.WaitForWorkers() in roslyn/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 331
         at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethodBodies(CSharpCompilation compilation, PEModuleBuilder moduleBeingBuiltOpt, Boolean emittingPdb, Boolean hasDeclarationErrors, Boolean emitMethodBodies, BindingDiagnosticBag diagnostics, Predicate`1 filterOpt, CancellationToken cancellationToken) in roslyn/src/Compilers/CSharp/Portable/Compiler/MethodCompiler.cs:line 160
         at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CompileMethods(CommonPEModuleBuilder moduleBuilder, Boolean emittingPdb, DiagnosticBag diagnostics, Predicate`1 filterOpt, CancellationToken cancellationToken) in roslyn/src/Compilers/CSharp/Portable/Compilation/CSharpCompilation.cs:line 3505
         at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream metadataPEStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, Stream sourceLinkStream, IEnumerable`1 embeddedTexts, RebuildData rebuildData, CompilationTestData testData, CancellationToken cancellationToken) in roslyn/src/Compilers/Core/Portable/Compilation/Compilation.cs:line 2952
         at Roslyn.Test.Utilities.RuntimeEnvironmentUtilities.EmitCompilationCore(Compilation compilation, IEnumerable`1 manifestResources, DiagnosticBag diagnostics, CompilationTestData testData, EmitOptions emitOptions) in roslyn/src/Compilers/Test/Core/Compilation/IRuntimeEnvironment.cs:line 260
         at Roslyn.Test.Utilities.RuntimeEnvironmentUtilities.EmitCompilation(Compilation compilation, IEnumerable`1 manifestResources, List`1 dependencies, DiagnosticBag diagnostics, CompilationTestData testData, EmitOptionsemitOptions) in roslyn/src/Compilers/Test/Core/Compilation/IRuntimeEnvironment.cs:line 232
         at Roslyn.Test.Utilities.CoreClr.CoreCLRRuntimeEnvironment.Emit(Compilation mainCompilation, IEnumerable`1 manifestResources, EmitOptions emitOptions, Boolean usePdbForDebugging) in /roslyn/src/Compilers/Test/Core/Platform/CoreClr/CoreCLRRuntimeEnvironment.cs:line 46
         at Microsoft.CodeAnalysis.Test.Utilities.CompilationVerifier.Emit(IRuntimeEnvironment testEnvironment, IEnumerable`1manifestResources, EmitOptions emitOptions) in roslyn/src/Compilers/Test/Core/CompilationVerifier.cs:line 491
         at Microsoft.CodeAnalysis.Test.Utilities.CompilationVerifier.Emit(String expectedOutput, Boolean trimOutput, Nullable`1 expectedReturnCode,String[] args, IEnumerable`1 manifestResources, EmitOptions emitOptions, Verification peVerify, SignatureDescription[] expectedSignatures) in
      roslyn/src/Compilers/Test/Core/CompilationVerifier.cs:line 269
         at Microsoft.CodeAnalysis.Test.Utilities.CommonTestBase.Emit(Compilation compilation, IEnumerable`1 dependencies, IEnumerable`1 manifestResources, SignatureDescription[] expectedSignatures, String expectedOutput, Boolean trimOutput, Nullable`1 expectedReturnCode, String[] args, Action`1 assemblyValidator, Action`1 symbolValidator, EmitOptions emitOptions, Verification verify) in /roslyn/src/Compilers/Test/Core/CommonTestBase.cs:line 191
      at Microsoft.CodeAnalysis.Test.Utilities.CommonTestBase.CompileAndVerifyCommon(Compilation compilation,IEnumerable`1 manifestResou[2/1881]numerable`1 dependencies, Action`1 sourceSymbolValidator, Action`1 assemblyValidator, Action`1 symbolValidator, SignatureDescription[] expectedSignatures, String expectedOutput, Boolean trimOutput, Nullable`1 expectedReturnCode, String[] args, EmitOptions emitOptions, Verification verify) in roslyn/src/Compilers/Test/Core/CommonTestBase.cs:line 103
         at Microsoft.CodeAnalysis.CSharp.Test.Utilities.CSharpTestBase.CompileAndVerify(Compilation compilation,IEnumerable`1 manifestResources, IEnumerable`1 dependencies, Action`1 sourceSymbolValidator, Action`1 validator, Action`1 symbolValidator, SignatureDescription[] expectedSignatures, String expectedOutput, Boolean trimOutput, Nullable`1 expectedReturnCode, String[] args, EmitOptions emitOptions, Verification verify) in roslyn/src/Compilers/Test/Utilities/CSharp/CSharpTestBase.cs:line 1048
         at Microsoft.CodeAnalysis.CSharp.Test.Utilities.CSharpTestBase.CompileAndVerify(CSharpTestSource source, IEnumerable`1 references, IEnumerable`1 manifestResources, IEnumerable`1 dependencies, Action`1 sourceSymbolValidator, Action`1 assemblyValidator, Action`1 symbolValidator, SignatureDescription[] expectedSignatures, String expectedOutput, Boolean trimOutput, Nullable`1 expectedReturnCode, String[] args, CSharpCompilationOptions options, CSharpParseOptions parseOptions, EmitOptions emitOptions, TargetFramework targetFramework, Verification verify) in roslyn/src/Compilers/Test/Utilities/CSharp/CSharpTestBase.cs:line 1005
    at Microsoft.CodeAnalysis.UnitTests.LocalizationInfraTests.VerifyCulture() in /roslyn/src/Compilers/CSharp/Test/CommandLine/LocalizationInfraTests.cs:line 48      ```
   

@giritrivedi
Copy link
Contributor Author

cc @uweigand

@jaredpar
Copy link
Member

jaredpar commented Sep 20, 2024

Is there anything needed to be verified on big endian architecture to ensure that these test cases are not only meant for little endian machines ?

As a starting point I would remove those asserts locally and re-run the entire test suite, not just CommandLineTests, and see if that revealed any issues. The tests passing here doesn't mean much with the attribute removed.

Overall though I think it would help if you gave some context here. There are now a few PRs out where it appears you're attempting to get Roslyn to run in a new environment. Could you tell us a bit more about what you're doing here? Maybe open an issue / discussion where we can see what the bigger goals are. Otherwise this just seems like one off PRs that look a bit off to us and without context we're not sure what to do with them.

@uweigand
Copy link
Contributor

Overall though I think it would help if you gave some context here. There are now a few PRs out where it appears you're attempting to get Roslyn to run in a new environment. Could you tell us a bit more about what you're doing here? Maybe open an issue / discussion where we can see what the bigger goals are. Otherwise this just seems like one off PRs that look a bit off to us and without context we're not sure what to do with them.

Hi @jaredpar , I understand @giritrivedi is out this week, so let me try to answer this. This effort is not about supporting any new platforms as such, but it is about the IBM community-supported platforms IBM Z (s390x) and POWER (ppc64le). We already support .NET using the Mono runtime on these platforms since .NET 6 (or 7), and roslyn in general is working fine - e.g. we can use roslyn natively on these platforms to build itself, or even build the full source-build .NET. Also, most of the roslyn test suite is passing - any major issue has been fixed back in the .NET 6 days.

However, we never got around to making the roslyn test suite fully green on our platforms. To ensure ongoing support going forward, we're now trying to fix that - get to a state where the test suite passes successfully 100%, and then set up an ongoing CI to keep it that way. Note that the vast majority of tests already pass, but there's a small number (around 50 out of the 30000 tests) that fail due to various issues, which look to be either test suite problems (incompatibility with big-endian and/or the Mono runtime), or some minor differences in behavior between the Mono and CoreCLR runtime that doesn't cause any problems for actual code, but are visible to some of the low-level roslyn tests.

What I've asked @giritrivedi to do is to go through that set of remaining test case failures and address them, typically by fixing some of those incompatibilities where feasible, or else disabling the tests if appropriate. (Anything that should turn out to be some actual problem we had overlooked so far would of course also have to be addressed.)

@giritrivedi
Copy link
Contributor Author

Thanks @uweigand for clarifying @jaredpar's concerns in my absense.
@jaredpar I hope you got some clarity about this now.
Since all tests are passing, can this be reviewed and merged ?

Copy link
Member

@jcouv jcouv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM Thanks (iteration 1)

@jcouv
Copy link
Member

jcouv commented Oct 16, 2024

@jaredpar @AlekseyTs for second review. Thanks

@jcouv jcouv self-assigned this Oct 16, 2024
@jcouv jcouv merged commit df6cccf into dotnet:main Oct 16, 2024
24 checks passed
@dotnet-policy-service dotnet-policy-service bot added this to the Next milestone Oct 16, 2024
@jcouv
Copy link
Member

jcouv commented Oct 16, 2024

Merged/squashed. Thanks @giritrivedi for the contribution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Compilers Community The pull request was submitted by a contributor who is not a Microsoft employee. Platform - Mono CoreClr Bugs specific to mono coreclr untriaged Issues and PRs which have not yet been triaged by a lead
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants