LiteException during Saving huge file to FileStorage

[podkolzzzin]

I worked with a really big file using FileStorage and received this during saving.

I used method FileStorage.OpenWrite to get LiteStream and then wrapped it with StreamWriter.

It's not the first line I'm trying to write. In fact it is 1532935th line.

      foreach (var line in lines.Take(lines.Count - 1)) {
        if (version != lines.Version)
          throw new SourceChangedException();
        writer.WriteLine(line);
      }

Exception thrown: 'LiteDB.LiteException' in LiteDB.dll

Additional information: Cannot insert duplicate key in unique index '_id'. The duplicate value is '58a2ef199af4b1443051d445\00458'.



 	LiteDB.dll!LiteDB.IndexService.AddNode(LiteDB.CollectionIndex index = {LiteDB.CollectionIndex}, LiteDB.BsonValue key = {LiteDB.BsonValue}, byte level = 1, LiteDB.IndexNode last = null)	Unknown
 	LiteDB.dll!LiteDB.IndexService.AddNode(LiteDB.CollectionIndex index = {LiteDB.CollectionIndex}, LiteDB.BsonValue key = {LiteDB.BsonValue}, LiteDB.IndexNode last = null)	Unknown
 	LiteDB.dll!LiteDB.LiteEngine.InsertDocument(LiteDB.CollectionPage col = {LiteDB.CollectionPage}, LiteDB.BsonDocument doc = {LiteDB.BsonDocument})	Unknown
 	LiteDB.dll!LiteDB.LiteEngine.Insert.AnonymousMethod__0(LiteDB.CollectionPage col = {LiteDB.CollectionPage})	Unknown
 	LiteDB.dll!LiteDB.LiteEngine.Transaction<int>(string collection = "_chunks", bool addIfNotExists = true, System.Func<LiteDB.CollectionPage, int> action = {Method = {System.Reflection.RuntimeMethodInfo}})	Unknown
 	LiteDB.dll!LiteDB.LiteEngine.Insert(string collection = "_chunks", System.Collections.Generic.IEnumerable<LiteDB.BsonDocument> docs = {LiteDB.BsonDocument[1]})	Unknown
 	LiteDB.dll!LiteDB.LiteEngine.Insert(string collection = "_chunks", LiteDB.BsonDocument doc = {LiteDB.BsonDocument})	Unknown
 	LiteDB.dll!LiteDB.LiteFileStream.WriteChunks()	Unknown
 	LiteDB.dll!LiteDB.LiteFileStream.Write(byte[] buffer = {byte[3075]}, int offset = 0, int count = 1024)	Unknown
 	mscorlib.dll!System.IO.StreamWriter.Flush(bool flushStream, bool flushEncoder)	Unknown
 	mscorlib.dll!System.IO.StreamWriter.Write(char[] buffer = {char[526]}, int index = 170, int count = 356)	Unknown
 	mscorlib.dll!System.IO.TextWriter.WriteLine(string value)	Unknown
>	Devart.DbForge.Integration.Base.dll!Devart.DbForge.Integration.AddinEditor.SaveToWriter(System.IO.StreamWriter writer = {System.IO.StreamWriter}) Line 677	C#

[podkolzzzin]

Here is simple solution to reproduce error:
https://github.com/podkolzzzin/LiteExceptionDuplicateKey
On the 5-10th iteration it usually fails with "... duplicate key ..."

P.S. If you increase number of lines enough you'll get OOM from the related request.

[mbdavid]

Hi @podkolzzzin, transaction are not supported inside FileStorage operations (read/write/delete). But until last release 3.0.1 there is no validation about this. Now I add check to avoid too memory usage.

It´s because all modification during a transaction occurs in memory. Only when you commit, all dirty pages are write in this flushing memory.

But I will check you test to check if there is another error about duplicate key.

[podkolzzzin]

Oh, it's really unexpected behaviour. Is it written somewhere?
Thanks for fast responses and bugfixes.

P.S. I have written some new comments in #363, it's still reproducing. Please reopen it.

[mbdavid]

My documentation are very weak. I always has no time and patience to update :) (and it´s quite hard to write in english to me).

[podkolzzzin]

It's really not a problem to use LiteDB, it usually behaves very predictable and it's really cool. Some simple things can be easily detected investigating source code. But the issue with transactions was really unexpected. It would be enough to write just a small note in paragraph about FileStorage to inform everybody about this behaviour.

[mkosieradzki]

@mbdavid Just to make sure. Is it a supported scenario when one thread creates a transaction and does normal operations and simultaneously other thread writes to FileStorage without a transaction? I have tested such a scenario in 3.0.0 and I was receiving very strange errors (not tested yet in 3.0.1).

[mkosieradzki]

@mbdavid I have tested it in 3.0.1 and it seems not. I receive: "Transactions are not supported here: LiteFileStream" in one thread and "The read lock is being released without being held". I don't use transaction on the thread performing FileStorage operations.

[botne]

@mkosieradzki: I had a similar issue yesterday. I have multiple databases - some only used for filestorage. I found that a database with files only had active transactions (by looking at the count in the Engine).

When I put a lock around my file writing the problem dissapears.
@mbdavid: Could it be the transactions used internally while writing the chunks that causes this?

[lbnascimento]

Hi! With the objective of organizing our issues, we are closing old unsolved issues. Please check the latest version of LiteDB and open a new issue if your problem/question/suggestion still applies. Thanks!