System.AccessViolationException at System.Text.UTF8Encoding.GetCharCount(Byte*, Int32, System.Text.DecoderNLS) #321
Comments
|
Well that's interesting.
But the tail should point to the same memory given, which was a utf8 string allocated and created by that same overload of So there's a bit of a mystery here. Do you know what SQL string is being passed down? I'm wondering if anything about it looks weird. I'm a little surprised that Microsoft.Data.Sqlite is calling this particular It seems possible you have found a bug, but (1) this piece of code has been pretty well tested, and (2) if there's a bug, I'm not seeing it yet. |
|
I shall try to see if I can get the SQL string. With SQLite version 1.1.14 it ran for months and it didn't occur. We are now only upgrading to SQLite 2.0.2 and in every hour this crash occurs. |
That area of code definitely changed between 1.1 and 2.0. |
|
Yesterday a LoggerFactory was set on the DbContextOptionsBuilder, so we could log the SQL statements, as shown below. It ran all day without any exception. Today I removed the LoggerFactory again and after a few minutes the application crashed. var optionsBuilder = We currently don't use a LoggerFactory on the DbContext because it is a database with one table that is just used for intensive logging of events in one table, every second and is isolated from the rest of the application data. The database is a +-10GB file and contains continuously the events of the last 7 days. So I think we can solve it (as a workaround) by adding a LoggerFactory to the DbContextOptionsBuilder. I will do some more testing, but because the exception is not occuring I can not capture the SQL statement. |
|
I am interested in trying to reproduce this problem. Can you share your code, or some piece of it? If not, how much can you tell me about how I would write a contained program that is similar? Thanks. |
|
As attachment below I added a little description of the application. Is this helpful? |
|
Yes, that's helpful. More questions: So once per second you insert one row? Each row in a separate transaction? Is this .NET Framework or .NET Core? Which version? If .NET Core, what's the OS underneath? Windows? (You're on recent EF, so I guess would mean you're on .NET Core...) How exactly does the character encoding issue fit in? It looks like you're passing down System.String, so by the time SQLitePCLRaw gets the string, it should be UTF-16, right? |
|
So once per second you insert one row? Each row in a separate transaction? Is this .NET Framework or .NET Core? Which version? (You're on recent EF, so I guess would mean you're on .NET Core...) How exactly does the character encoding issue fit in? It looks like you're passing down System.String, so by the time SQLitePCLRaw gets the string, it should be UTF-16, right? |
|
Yesterday I did a little test with the NullLoggerFactory and the exception also occurs. |
|
I'm still thinking it might be helpful to know what string was passed in. I'm wondering if the following info might help. Perhaps you could use this to modify your code to catch the exception and then dump out the diagnostic info? Just a thought... |
|
Thank you for the information Eric. Because I could not use the attributes on an async method I am now testing with the following approach which was also referenced in the document you mentionded. If you have any remarks, please let me know. |
|
Anything that can give us some insight into what's going on is worth exploring. I've also wondered if it would be possible to configure debugger settings to run the code under a debugger and inspect things at the moment of the crash. |
|
After adding The following is a zip file which contains a json file with all the MessageHistory rows that are being added when the crash (exception) occurs. I could not retrieve the SQL statement(s). I try to see if I can attach a debugger and maybe get a dump. |
|
I wrote the following test for the MessageHistory.json file which contains the messages on which the exception occurs in the application, but the exception is not occuring in the test. The database that is created during the test is the following <...> I forgot to mention that there is a small second table, but it contains only 2 fields of which 1 is a reference to the main table. |
|
To clarify the (test) code a little: The DbContext is encapsulated inside a wrapping entity repository (class) in which the DbContext is injected and on which some basic operations are available. The basic operation for AddRange is the following: |
|
We have also encountered this problem. Were you able to identify the root cause of the issue? We are currently using:
|
|
Hi Pressacco, No we have not yet identified the root cause. It is on our todo list. |
|
I don't know if this helps, but in our case:
Our next step will be to run the DICOM through a validator to ensure that the DICOM attributes are valid. Would you happen to know of any good (and free) DICOM validators ?? |
|
I've narrowed-down the root cause of this to the heap allocation from the "new byte[nlen]" in "to_utf8_with_z()" not being pinned later in "utf8_to_string()" in the "fixed (byte* q = sp)" block. Since it's not pinned, it may be moved by the GC and that appears to happen leading to the access violation. I confirmed this with the following code which briefly disables garbage collection during those calls in sqlite3_prepare_v2 as shown below. We were using EntityFramework v3.1.2 with SQLiteRaw v2.0.2.669. We have reverted to EntityFramework v2.2.6 which depends on SQLiteRaw v1.1.12 as a workaround for now. We use .Net Framework 4.8. The issue for the "fixed()" block may actually be in the CLR or .Net runtime. C:\SQLitePCL.raw\src\SQLitePCLRaw.core\raw.cs |
|
@levant Do you have more specific information about why you believe that specific thing array is the thing that is not pinned? I agree that if suspending the GC avoids the problem then that would seem to mean that something is being accessed without being pinned. And I agree that the byte array from I wouldn't question this, but the fact is that the array is pinned (with the fixed statement). Unless I am misusing fixed, but I can't find any indication in the docs that I'm doing this wrong. Unless, as you suggest, The only thing that strikes me as odd here is that I am referencing sp.Length inside the pin. sp is the Span for the byte array. I'd have to look at the CIL output to see if it's doing something weird here. But if so, the easy workaround is to get the Length into a local variable before the pin. |
|
Yes @ericsink that is exactly my conclusion as you said: “ Unless, as you suggest, fixed isn't doing what it is supposed to be doing. But if that were true, I would think it likely that such a bug would be runtime-specific, ie., it might show up in .NET Framework but not .NET Core, for example.” I don’t have time to dig deeper as we have already found a workable solution for us of going back to EntityFramework 2.2.6 whose dependency is your SQLiteRaw 1.1.12 which predates the Span enhancements. Keep in mind that Span was introduced by MS relatively recently so there may be poor support for it in some libraries like the older .Net 4.8 that we use vs .NetCore or the recently available consolidation into .Net 5.0 (available on preview). |
|
great work, guys. encountering same problem. thanks to this, got some starting points for investigation. cheers |
|
Hi @ericsink, i've got issues that sound very similar: application crashes with Access Violation while importing xml logs into a database. the program is(was): .net 4.7.2/win10/sqlite nuget packages after reading this thread, i converted my importer to .netcore 3.1 it used to crash, as per last crash log below, every few minutes. hope this triggers some ideas. Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. |
|
Hi @ericsink, I disabled a part of our logic to prevent this issue, but in another area where I can't disable this issue and where we intensively create records, the issue occurs continuously. Could I help in any way to solve this issue, or what could be done? At the moment I don't know what else I can do. |
|
@jay1891 Looks like the big thing I need here is a small sample app that reproduces the problem. If I have that, I can do a workaround or fix. Any chance you could provide that? |
|
@ericsink ok, I try to provide the sample app. |
|
@ericsink We the solution below I can reproduce it 1 in 5 times in the first 15 seconds after pressing "Create number of entities". It is very difficult (for me) to find a perfect reproduction example. If you have any suggestions, please let me know. Could you try to reproduce it with this sample?
|
|
In my case, the error happened on the eighth try. I was beginning to think it would not happen at all. :-) FWIW: If I run the sample app twice in a row without Clean in between, I get a different error, unable to load e_sqlite3 on some sort of a Migrate call. Do you see that too? In the 7 test runs before I saw the "memory is corrupt error", each time it failed at around 35-40 seconds with a SQLite error 5, db is locked. Do you see that too? I assume both of the above are unrelated to the memory corruption problem, but I thought I should mention them just in case. |
|
With a little luck it happened 3 times after each other. :-)
I didn't had if before, but running the application again that I added here, I had the issue too. I migrated from package.config to PackageReference and now it doesn't occur anymore.
Yes that occurs when I run it too
|
|
FYI, I've done some simplifications to your repro app and checked it in to the tree under My test runs do seem to suggest that the problem is specific to .NET Framework and does not exist with .NET Core. |
…, put sp.Length into a local so it doesn't get referenced inside the fixed block. I didn't really expect this to make a difference, and sure enough, it did not. the crash still happens.
…he issue, put sp.Length into a local so it doesn't get referenced inside the fixed block. I didn't really expect this to make a difference, and sure enough, it did not. the crash still happens." This reverts commit d980630.
|
FWIW, as another stab in the dark, I tried updating the repro app to System.Memory 4.5.4, and that didn't help either. |
|
If it could be of any value; I adjusted your repro.cs (see below) and with this adjustment I can reproduce it with every run of the console app (5/5) |
|
Hi, When using the For example, given a code like the following: static ReadOnlySpan<byte> GetSpan()
{
var myArray = new byte[50000000];
//// Option 1: Create a span from pointer
fixed (byte* ptr = myArray)
return new ReadOnlySpan<byte>(ptr, myArray.Length);
// Option 2: Create a span from array (ref)
//return myArray;
}
// Get a span and then force objects to be GCed.
ReadOnlySpan<byte> byteSpan = GetSpan();
GC.Collect();
// Try to access the whole span.
string str;
fixed (byte* ptr = byteSpan)
str = Encoding.UTF8.GetString(ptr, byteSpan.Length);That code creates a In contrast, when you run it on .NET Core 2.1 or higher, it will work without a problem as the However, from looking at the code in Thanks! |
|
I took a closer look on the call stack and the relevant code when the crash happens, and I think the situation described in my previous post can actually be the cause of this issue (and explain why it crashes on .NET Framework but works on .NET Core): In the call stack of the The called methods SQLitePCL.raw/src/SQLitePCLRaw.core/raw.cs Lines 730 to 742 in ee08e7e Notice that the method with a We can see that it uses a If I understand the SQLite documentation correctly, The method However, remember that the pointer passed to the native This works correctly on .NET Core because as mentioned above, while the byte array is pinned, the Note that there is a different overload of I think that one should work correctly, since instead of creating a new |
|
@kpreisser I'm still digesting what you've written above, but for now I want to say thanks for taking the time to investigate this and comment! |
…rence on the span explicitly instead of letting the C# 7.3 fixed statement call it implicitly. no change in behavior or effect. working on #321
…n investigation by @kpreisser I have changed the problematic overload of raw.sqlite3_prepare_v2() to call a different overload of same in the provider to avoid creating a span from a pointer. with this change, I can run the issue321 repro app at 20 iterations on net461 without a failure. so this seems to confirm with @kpreisser said, but now every case where I create a span from a pointer needs to be reviewed for this kind of lifetime issue.
|
WOW!
This is amazing detective work.
respect
From: Konstantin Preißer <[email protected]>
Sent: Thursday, 4 June 2020 9:36 PM
To: ericsink/SQLitePCL.raw <[email protected]>
Cc: wolfkumpitsch <[email protected]>; Comment <[email protected]>
Subject: Re: [ericsink/SQLitePCL.raw] System.AccessViolationException at System.Text.UTF8Encoding.GetCharCount(Byte*, Int32, System.Text.DecoderNLS) (#321)
I took a more close look on the call stack and the relevant code when the crash happens, and I think the situation described above can actually be the cause of this issue (and explain why it crashes on .NET Framework but works on .NET Core):
In the call stack of the AccessViolationException, we can see that sqlite3_prepare_v2 is called, which then calls utf8z.utf8_to_string:
at System.Text.UTF8Encoding.GetCharCount(Byte* bytes, Int32 count, DecoderNLS baseDecoder)
at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding)
at SQLitePCL.utf8z.utf8_to_string()
at SQLitePCL.raw.sqlite3_prepare_v2(sqlite3 db, String sql, sqlite3_stmt& stmt, String& tail)
The called methods raw.sqlite3_prepare_v2 overloads look like this:
https://github.com/ericsink/SQLitePCL.raw/blob/ee08e7e88f18ca107fa83da11392cdceea611e88/src/SQLitePCLRaw.core/raw.cs#L730-L742
Notice that the method with a string parameter creates a utf8z from a string, which creates a new byte[] and then returns a ReadOnlySpan<byte> for it. This utf8z is then passed to the provider implementation of sqlite3_prepare_v2 which looks like this (for cdecl):
https://github.com/ericsink/SQLitePCL.raw/blob/ee08e7e88f18ca107fa83da11392cdceea611e88/src/SQLitePCLRaw.provider.dynamic_cdecl/Generated/provider_dynamic_cdecl.cs#L261-L278
We can see that it uses a fixed block to pin the span referencing the byte array and then calls the native sqlite3_prepare_v2. While the array is still pinned, it creates a ReadOnlySpan<byte> from the returned p_tail value.
If I understand the SQLite documentation correctly, sqlite3_prepare_v2 sets the pzTail to point to a
different part of the string supplied in parameter p_sql.
The method raw.sqlite3_prepare_v2 then creates a new utf8z from the returned tail pointer and calls utf8_to_string on it, which needs to access the pointer.
However, remember that the pointer passed to the native sqlite3_prepare_v2 points to a byte array that was allocated in util.to_utf8_with_z. During the call of the native sqlite3_prepare_v2 function, this array was pinned, but once we call sp_tail.utf8_to_string(), the original byte array is no longer pinned (on .NET Framework) and also no longer referenced, so the Garbage Collector could already have collected it (or moved it), even though we still have the tail Span that points in the middle of the original byte array location. Then, when we want to access the tail by calling .utf8_to_string, the AccessViolationException occurs.
This works correctly on .NET Core because as mentioned above, while the byte array is pinned, the ReadOnlySpan<byte> is created from the p_tail pointer that points to a different part of the byte array, and therefore as long as the tail Span is on the stack, the runtime will prevent the byte array from being GCed (and probably from being moved by the GC).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#321 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTLZMUZZLJDEGQGTOEONOLRU6BJPANCNFSM4J33ZLIA> . <https://github.com/notifications/beacon/ABTLZMWPVC4IG4PDKQIW5CTRU6BJPA5CNFSM4J33ZLIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEYJTNBI.gif>
|
|
The 2.0.4-pre release I uploaded to nuget.org today contains the fix for this problem. |
|
Hi guys, Is there a way for you guys to push EFCore 3.1.* to use the newer Thanks |
Can you file an issue on dotnet/efcore so we can discuss? |
…. these overloads were not memory safe, as they returned a span made from a pointer obtained from a fixed block. technically, this is a breaking change, but it is likely that nothing was using those overloads. my own test suite was not, and Microsoft.Data.Sqlite was not. there was discussion of the problem with these overloads in #321. additional detective work on this happened while trying to figure out #430.
…re_v2/v3. these overloads were not memory safe, as they returned a span made from a pointer obtained from a fixed block. technically, this is a breaking change, but it is likely that nothing was using those overloads. my own test suite was not, and Microsoft.Data.Sqlite was not. there was discussion of the problem with these overloads in #321. additional detective work on this happened while trying to figure out #430." This reverts commit 264be3a.
|
Looks like this got fixed but never closed. |
Hi,
When adding records to the database with SQLitePCL 2.0.2 I'm getting sometimes the following exception. The entire .NET application crashes, so I can not catch the exception. What could be the reason for this?
Exception Info: System.AccessViolationException
at System.Text.UTF8Encoding.GetCharCount(Byte*, Int32, System.Text.DecoderNLS)
at System.String.CreateStringFromEncoding(Byte*, Int32, System.Text.Encoding)
at SQLitePCL.utf8z.utf8_to_string()
at SQLitePCL.raw.sqlite3_prepare_v2(SQLitePCL.sqlite3, System.String, SQLitePCL.sqlite3_stmt ByRef, System.String ByRef)
at Microsoft.Data.Sqlite.SqliteCommand+d__64.MoveNext()
at Microsoft.Data.Sqlite.SqliteCommand+d__54.MoveNext()
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(System.Data.CommandBehavior)
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReaderAsync(System.Data.CommandBehavior, System.Threading.CancellationToken)
at Microsoft.Data.Sqlite.SqliteCommand+d__60.MoveNext()
The Dlls are the following:
The text was updated successfully, but these errors were encountered: