NDK crashes symbolification #8832
Comments
akravch
added
Area: App+Library Build
|
@grendello might be able to comment on how to get better crashes for native code. For managed code, there used to be the These are the open issues:
@steveisok @tommcdon are there any plans to implement something like |
jonathanpeppers
added
enhancement
|
NDK contains a utility called |
|
Hi @grendello, thank you for the response, but as I initially mentioned, I have already tried using Also, So basically my question is how do I get the symbols for our own |
|
@akravch I'm sorry, I should have read the context... :) The AOT .so files are weird in that they contain code as data, so most symbols will be useless for you. The code there is loaded into the application as if it were JIT-ed but without actually running the JIT and it's not executed directly. The generated code looks like this (you can see it if you set the The |
|
Thanks for the details, that was very enlightening! So does it mean there is no way at all to trace addresses back to source code in this case?
I guess the addresses are still following some rules, right? In our crash reporting system I see different crashes with completely same addresses and offsets, so the code must be loaded in the same way. |
|
The source code is, eventually, some managed one and you should be able to see it in the trace as a regular managed frame (or a trampoline to one). AOT is merely a shortcut to avoid JIT-ing at run time, the managed symbols should be preserved. You shouldn't think of it as about "native code running in a shared library", but rather as "serialized native code carried by a shared library" - the code will be running in the same place it would run if it was normally JIT-ed. Addresses have a very good chance of being similar/the same, because your application doesn't change on the device between runs, only between installations/updates. Base pointers may be the same (but don't have to be), offsets off of them will be the same in all cases, until the app's updated. If you want to try to map addresses to managed symbols, I think you'd need to do it manually. Having the |
|
Hi @akravch, I looked a bit deeper into this issue and the problem seems to related to how native crashes during PInvokes are reported. From Android's native perspective if we take a look the stack trace you are seeing: The unknown stack frame is shown because the call to libc comes from jitted methods. When we JIT compile a managed method (from your example: managed However, what is possible is to get such information from the mono runtime when the crash happens. Sadly, to make all this a bit more difficult, an additional problem here is that there does not seem to be a simple way to redirect stdout/stderr to android logcats (ref: android/ndk#671). But by using the solution provided here and your example code I managed to redirect these messages to logcat and after inspection it indeed shows managed stack trace: Potential workarounds/solutions
Let me know if this helps and we will find a more robust solution on our end. |
|
Hi @ivanpovazan, thanks a lot for the idea, we tried it and it actually worked even in production, and the stack traces were readable! I think it would be really great to have it better integrated into the normal logging process somehow. And if there was an out of the box option to save this output to a file before the app crash, it would almost be an NDK crash handler already. |
|
For anyone interested here is code that redirects stdout (1) to Logcat. using Java.IO;
using System.Text;
using static Android.Systems.Os;
namespace MauiApp;
// https://codelab.wordpress.com/2014/11/03/how-to-use-standard-output-streams-for-logging-in-android-apps/
// https://stackoverflow.com/questions/23352592/redirecting-stdin-and-stdout-of-c-program-in-android
// https://android.googlesource.com/platform/libcore/+/cff1616/luni/src/main/java/java/io/FileDescriptor.java
/// <summary>
/// On Android stdout are redirected to /dev/null by default.
/// This class redirects stdout to LogCat.
/// </summary>
public static class AndroidRedirectStdToLogCat
{
private static Thread? Thread;
private static bool IsStarted => Thread is not null;
public static void Start(string applicationName)
{
if (IsStarted)
{
return;
}
/* create the pipe and redirect stdout */
var pipe = Pipe();
if (pipe is null)
{
throw new Exception("Could not create pipe");
}
var pipeReader = pipe[0];
var pipeWriter = pipe[1];
// NOTE: file descriptors for 0, 1 and 2 are stdin, stdout and stderr (unix)
Dup2(pipeWriter, 1); // redirect stdout to pipe
/* spawn the logging thread */
var thread = new Thread(ThreadLoop);
Thread = thread;
thread.Start(new ThreadParameters(applicationName, pipeReader));
}
/// <summary>
/// For testing.
/// </summary>
internal static void WriteToStdout(string text)
{
// NOTE: System.Console.OpenStandardOutput() points to LogcatStream
// https://github.com/dotnet/runtime/blob/main/src/libraries/System.Console/src/System/ConsolePal.Android.cs
// NOTE: Java.IO.FileDescriptor.Out does NOT point to fd 1 (original stdout)
// use reflection to create a new FileDescriptor pointing to stdout (1)
using var stdout = new FileDescriptor();
var field = stdout.Class.GetDeclaredField("descriptor");
field.Accessible = true;
field.SetInt(stdout, 1);
// write to stdout
using var stream = new FileOutputStream(stdout);
var bytes = Encoding.UTF8.GetBytes(text);
stream.Write(bytes);
stream.Flush();
}
private sealed record ThreadParameters(
string ApplicationName,
FileDescriptor PipeReaderFileDescriptor);
private static void ThreadLoop(object? parameter)
{
try
{
if (parameter is not ThreadParameters parameters)
{
throw new Exception("Parameter object expected.");
}
var applicationName = parameters.ApplicationName;
const int bufferSize = 512;
var buffer = new byte[bufferSize];
var bufferSpan = buffer.AsSpan();
var lineTerminator = (byte)'\n';
int usedBufferLength = 0;
int readCount;
using var stream = new FileInputStream(parameters.PipeReaderFileDescriptor);
// read blocks until it gets input, -1 signifies EOF
while ((readCount = stream.Read(buffer, usedBufferLength, buffer.Length - usedBufferLength)) > 0)
{
var length = usedBufferLength + readCount;
ReadOnlySpan<byte> rest = bufferSpan[..length];
// iterate lines and write to LogCat
while (rest.IndexOf(lineTerminator) is var index && index != -1)
{
var lineBytes = rest[..index]; // skip null terminator
WriteToLogCat(lineBytes, applicationName);
rest = rest[(index + 1)..]; // skip null terminator
}
// line terminator not found, but there may still be some bytes left
switch (rest.Length)
{
// buffer full, write to LogCat
case bufferSize:
WriteToLogCat(rest, applicationName);
usedBufferLength = 0;
break;
// copy to start of buffer
case > 0 when (rest.Length != usedBufferLength):
rest.CopyTo(bufferSpan); // CopyTo deals with overlapping regions
usedBufferLength = rest.Length;
break;
case 0:
usedBufferLength = 0;
break;
}
}
// write any last content in buffer to LogCat
if (usedBufferLength > 0)
{
var lineBytes = bufferSpan.Slice(0, usedBufferLength);
WriteToLogCat(lineBytes, applicationName);
}
// cleanup
Thread = null;
}
catch (Exception e)
{
Android.Util.Log.Debug(nameof(AndroidRedirectStdToLogCat), e.ToString());
}
}
private static void WriteToLogCat(ReadOnlySpan<byte> bytes, string applicationName)
{
var lineStr = Encoding.UTF8.GetString(bytes);
Android.Util.Log.Debug(applicationName, lineStr);
}
} |
Android application type
.NET Android (net7.0-android, net8.0-android, etc.)
Affected platform version
.NET 8.0.200, workload 34.0.52/8.0.100
Description
Since updating from Xamarin to .NET 8 we are getting a lot of native crashes from production users. In order to investigate the reason, I'm trying to find a way to symbolificate them.
Our crash monitoring system is based on AppCenter, which uses Google Breakpad internally, but at this point I would be happy with any manual way of symbolification, either with Breakpad's
minidump_stackwalk,ndk-stack, or anything else.First, I noticed that by default there are no debug symbols at all in the
.dll.sofiles produced by AOT under . Runningreadelf -Wsusually gives something like that:I have found that setting
_AndroidAotStripLibrariestofalsedisables library stripping, but judging by the underscore in its name it's private API:https://github.com/xamarin/xamarin-android/blob/51bcefdbf08a2c17ab0633d5aee37f22acf5164d/src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.Aot.targets#L93
Now
readelfshows that there are debug symbols inside:However, if I crash the app and try to symbolificate the dump using the
.dll.sowith symbols e.g. withndk-stack, it doesn't map the addresses to the symbols:I have also tried disabling assembly compression with
<AndroidEnableAssemblyCompression>false</AndroidEnableAssemblyCompression>, but it didn't change anything in terms of symbolification.I am assuming I am still not building the app correctly for symbolification, so I would appreciate any help with that.
Steps to Reproduce
I have a repro project here, that crashes with an NDK crash on launch (run in Release): https://github.com/akravch/AndroidSymbols.
_AndroidAotStripLibrariesis set tofalse.Also attaching
readelfoutput for the unstripped.dll.so,ndk-stackoutput and the.dll.soitself.Did you find any workaround?
No.
Relevant log output
No response
The text was updated successfully, but these errors were encountered: