Experiencing a memory leak in pyramid.util.InstancePropertyMixin._set_properties

[pduval]

Using pyramid 1.4.3, and pyramid-webassets 0.7.1.

  File "/home/duvalp/.virtualenvs/ulysses/local/lib/python2.7/site-packages/pyramid-1.4.3-py2.7.egg/pyramid/router.py", line 251, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/home/duvalp/.virtualenvs/ulysses/local/lib/python2.7/site-packages/pyramid-1.4.3-py2.7.egg/pyramid/router.py", line 226, in invoke_subrequest
    request._set_extensions(extensions)
  File "/home/duvalp/.virtualenvs/ulysses/local/lib/python2.7/site-packages/pyramid-1.4.3-py2.7.egg/pyramid/util.py", line 93, in _set_extensions
    self._set_properties(extensions.descriptors)
  File "/home/duvalp/.virtualenvs/ulysses/local/lib/python2.7/site-packages/pyramid-1.4.3-py2.7.egg/pyramid/util.py", line 86, in _set_properties
    cls = type(parent.__name__, (parent, object), attrs)

As seen in above abridged stacktrace, the router object, in invoke_subrequest, calls request._set_extensions, which in turns calls _set_properties on InstancePropertyMixin , and this create a new Request class with the new propeties.
However this new Request class never goes out of scope, resulting with a new class created for every request staying around.

As far as I can tell, but my knowledge of pyramid internals isn't that great, the class is ultimately referenced by a zope.interface.adapter.AdapterRegistry and zope.interface.adapter.AdapterLookup object. (see the objgraph generated backref graph below).

I'm not sure whether this is an issue with my code or not, but any guidance appreciated.

[mmerickel]

ugh :-) contemplating just having pyramid.request.Request define its own __getattribute__ to avoid this malarky.

[mmerickel]

How can I reproduce this? Can I get some more information about what properties you're adding to the request? Whether you're using traversal or dispatch? Does your traversal tree hang onto the request? Could something else you've made be defining a circular reference with the request?

[pduval]

I probably should have started with a simple reproducible case. I can't really reproduce this outside my app (see gist I'm working on: https://gist.github.com/pduval/6229254), so I guess there is indeed some circular reference somewhere.
It looks now that I've been wasting people's time :-( sorry about that.

Apart from that, I'm using traversal, and the traversal tree as far as I know doesn't hang onto the request.
However, please note it's not the request instance that is not disposed of, it's the request class that's created by the titular method.

[mmerickel]

No problem. If you can reproduce this please reopen the ticket.

[mmerickel]

If you could fill in the appropriate objgraph code that'd be great. This is the first time I've used the library and I'm not sure how to reproduce your graph.

import tempfile
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
from pyramid.response import FileIter
from pyramid.request import Request

import objgraph
import gc

def hello_world(request):
    return Response('Hello %(name)s!' % request.matchdict)

def stop_in_debugger(request):
    gc.collect()

    fp = tempfile.NamedTemporaryFile(suffix='.png')
    objgraph.show_refs([request], filename=fp.name)
    return Response(app_iter=FileIter(fp), content_type='image/png')

def request_extension(request):
    print "extension"

class TestMetaRequest(type):
    pass

class TestRequest(Request):
    __metaclass__ = TestMetaRequest
    pass

class TestRequestFactory(object):
    def __call__(self, environ):
        return TestRequest(environ)

if __name__ == '__main__':
    config = Configurator(request_factory=TestRequestFactory())
    config.add_route('hello', '/hello/{name}')
    config.add_view(hello_world, route_name='hello')
    config.add_view(stop_in_debugger, name='debug')
    config.add_request_method(request_extension, name="print_extension", property=True)
    app = config.make_wsgi_app()
    server = make_server('0.0.0.0', 8080, app)
    server.serve_forever()

Note that on master property=True will be important because I fixed the thing to not reparent the request if there are no descriptors attached.

[pduval]

The better way to get a chain of references, back to a module, that keeps an object alive would be using:

import inspect
objgraph.show_chain(objgraph.find_backref_chain(request, inspect.ismodule), filename=fp.name)

However in my application, that took more time than my patience allowed, so I used show_backrefs instead, playing with the max_depth parameter:

objgraph.show_backrefs([request], filename=fp.name, max_depth=9)

The code I would actually have used to get the above graph would have been:

def stop_in_debugger(request):
    gc.collect()
    fp = tempfile.NamedTemporaryFile(suffix='.png')
    objgraph.show_backrefs(objgraph.by_type('TestMetaRequest')[-2], filename=fp.name, max_depth=9)
    return Response(app_iter=FileIter(fp), content_type='image/png')

(however in the current gist case, there are only ever 2 objects of type TestMetaRequest, since the classes are properly disposed)

[mcdonc]

Can you try the latest master? I added a fix via #1212 that I believe should squelch the leak.