Fatal Python error: deallocating None

[evandrocoan]

I have a complex application which parsers hundreds of files and for each line on each the file, create a dllistnode by subclassing it. The following code is what I do, but millions of times:

import llist

# bug fix for https://github.com/ajakubek/python-llist/issues/10
class FullRepresentation(llist.dllistnode):

    def __repr__(self):
        return "<representation>"

    def __str__(self):
        return self.__repr__()

class LogLine(FullRepresentation):

    def __init__(self, logline):
        super().__init__( self )
        self.logline = logline

loglist = llist.dllist()
mynode1 = LogLine( "Line 1" )
mynode2 = LogLine( "Line 2" )
mynode3 = LogLine( "Line 3" )

loglist.appendnode( mynode1 )
loglist.appendnode( mynode2 )
loglist.appendnode( mynode3 )

for node in loglist:
    print('')
    print('node', node)
    print('node.next', node.next)
    print('node.prev', node.prev)

print('')
print( 'loglist', loglist )

-->

node <representation>
node.next <representation>
node.prev None

node <representation>
node.next <representation>
node.prev <representation>

node <representation>
node.next None
node.prev <representation>

loglist dllist([<representation>, <representation>, <representation>])

After running my application, in a full database, I got the following error out of nowhere:

Fatal Python error: deallocating None

Thread 0x00007fdb05cb3700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 299 in wait
  File "/usr/lib/python3.6/threading.py", line 551 in wait
  File "/usr/local/lib/python3.6/dist-packages/tqdm/_monitor.py", line 69 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fdb09505740 (most recent call first):
  File "/__init__.py", line 254 in function
  File "/__init__.py", line 260 in function
  File "/__init__.py", line 260 in function
  File "/__init__.py", line 266 in another
  File "/__init__.py", line 204 in main
  File "/home/evandro/.local/bin/main", line 11 in <module>
Aborted (core dumped)

This is not related to the actual file being parsed because when running the application only with the lastest file running, everything works fine, i.e., no errors.

Searching on Google for this error Fatal Python error: deallocating None I found this threads:

1. https://bugs.python.org/issue6674
2. https://discuss.pytorch.org/t/fatal-python-error-deallocating-none/12073
3. https://stackoverflow.com/questions/15287590/why-should-py-increfpy-none-be-required-before-returning-py-none-in-c
4. https://docs.python.org/3.7/c-api/typeobj.html#c.PyTypeObject.tp_clear
5. https://stackoverflow.com/questions/24468667/whats-the-difference-between-tp-clear-tp-dealloc-and-tp-free

I ran gdb, but not stack trace was created. Then, I cloned this repository and removed all 4 lines which called Py_DECREF(Py_None). The following is a git diff of the changed files:

From 508f10c78d49edd904714202029a8b3411d7be1e Mon Sep 17 00:00:00 2001
Date: Tue, 20 Aug 2019 18:05:15 -0300
Subject: [PATCH] asdfasdf

---
 src/dllist.c | 4 ++--
 src/sllist.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/dllist.c b/src/dllist.c
index e765b6a..5c1bdbf 100644
--- a/src/dllist.c
+++ b/src/dllist.c
@@ -179,7 +179,7 @@ static int dllistnode_clear_refs(DLListNodeObject* self)
 {
     Py_CLEAR(self->value);
     Py_CLEAR(self->list_weakref);
-    Py_DECREF(Py_None);
+    // Py_DECREF(Py_None);
 
     return 0;
 }
@@ -562,7 +562,7 @@ static int dllist_clear_refs(DLListObject* self)
         }
     }
 
-    Py_DECREF(Py_None);
+    // Py_DECREF(Py_None);
 
     return 0;
 }
diff --git a/src/sllist.c b/src/sllist.c
index 1e27f71..02a9c47 100644
--- a/src/sllist.c
+++ b/src/sllist.c
@@ -115,7 +115,7 @@ static int sllistnode_clear_refs(SLListNodeObject* self)
 {
     Py_CLEAR(self->value);
     Py_CLEAR(self->list_weakref);
-    Py_DECREF(Py_None);
+    // Py_DECREF(Py_None);
 
     return 0;
 }
@@ -357,7 +357,7 @@ static int sllist_clear_refs(SLListObject* self)
         }
     }
 
-    Py_DECREF(Py_None);
+    // Py_DECREF(Py_None);
 
     return 0;
 }
-- 
2.17.1

Then, I reinstalled llist with pip3 install -v . and ran my application and the error Fatal Python error: deallocating None was fixed.

Now:

1. Why you where originally double freeing the Py_None objects?
2. Can all these 4 lines Py_DECREF(Py_None) be removed from the code?

Looking at the issue: #7 Memory leak when cycle reference to dllistnode

I could find this code: 26bb440 Support for cyclic garbage collection

Where you moved/played around changing the lines Py_DECREF(Py_None) from here to there.

[ajakubek]

This issue seems to be correlated with #10. Py_None is freed too many times because each node tries to Py_DECREF its own value, which happens to point to the node itself.

The Py_DECREF(Py_None) call in *_clear_refs functions is actually correct - it releases the Py_None reference which was taken in dllistnode_new.

Instead *_clear_refs should not release self->value if its equal to self.
I'll prepare a bugfix later today.

[ajakubek]

Fixed in master.