Runtime Error: maximum recursion depth #114

gsteele13 · 2020-09-30T14:31:28Z

Bug description

I have a Jupyterhub configured to use Native Auth, with DockerSpawner.

After running for a while (months or weeks, a time which I suspect depends on how many users are active), the server experiences a runtime error RecursionError: maximum recursion depth exceeded while calling a Python object

Expected behaviour

I would expect that it would just run.

Actual behaviour

After some period of time, it triggers runtime errors:

Looking in the error message output, it seems that this might be related to the Native Auth authenticator:

    HTTPServerRequest(protocol='https', host='jupyterhub-steelelab.tnw.tudelft.nl', method='GET', uri='/hub/login?next=%2Fhub%2Fuser%2Flieuwelocht%2Fmetrics', version='HTTP/1.1', remote_ip='::ffff:1
45.94.145.117')
    Traceback (most recent call last):
      File "/root/miniconda3/lib/python3.7/site-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/login.py", line 135, in get
        self.finish(self._render(username=username))
      File "/root/native_auth/nativeauthenticator/nativeauthenticator/handlers.py", line 151, in _render
        {'next': self.get_argument('next', '')},
      File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 1044, in render_template
        return template.render(**template_ns)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "/root/native_auth/nativeauthenticator/nativeauthenticator/templates/native-login.html", line 1, in top-level template code
        {% extends "page.html" %}
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/environment.py", line 592, in _generate
        optimized=self.optimized,
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 87, in generate
        generator.visit(node)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 32, in visit
        return f(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 774, in visit_Template
        self.blockvisit(node.body, frame)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 381, in blockvisit
        self.visit(node, frame)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 32, in visit
        return f(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 1233, in visit_Macro
        macro_frame, macro_ref = self.macro_body(node, frame)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 580, in macro_body
        self.blockvisit(node.body, frame)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 381, in blockvisit
        self.visit(node, frame)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 32, in visit
        return f(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 1458, in visit_Assign
        self.visit(node.node, frame)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 32, in visit
        return f(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/compiler.py", line 70, in new_func
        new_node = self.optimizer.visit(node, frame.eval_ctx)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 33, in visit
        return self.generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/optimizer.py", line 27, in generic_visit
        node = super(Optimizer, self).generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 67, in generic_visit
        new_node = self.visit(old_value, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 33, in visit
        return self.generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/optimizer.py", line 27, in generic_visit
        node = super(Optimizer, self).generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 67, in generic_visit
        new_node = self.visit(old_value, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 33, in visit
        return self.generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/optimizer.py", line 27, in generic_visit
        node = super(Optimizer, self).generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 67, in generic_visit
        new_node = self.visit(old_value, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 33, in visit
        return self.generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/optimizer.py", line 27, in generic_visit
        node = super(Optimizer, self).generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 67, in generic_visit
        new_node = self.visit(old_value, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 33, in visit
        return self.generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/optimizer.py", line 27, in generic_visit
        node = super(Optimizer, self).generic_visit(node, *args, **kwargs)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/visitor.py", line 66, in generic_visit
        elif isinstance(old_value, Node):
    RecursionError: maximum recursion depth exceeded while calling a Python object

In particular, I am suspicious that this may be triggered by the line:

      File "/root/native_auth/nativeauthenticator/nativeauthenticator/templates/native-login.html", line 1, in top-level template code
        {% extends "page.html" %}

After this initial error message, the logfiles are filled with tracebacks like this one:

    Traceback (most recent call last):
      File "/root/miniconda3/lib/python3.7/site-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/login.py", line 135, in get
        self.finish(self._render(username=username))
      File "/root/native_auth/nativeauthenticator/nativeauthenticator/handlers.py", line 151, in _render
        {'next': self.get_argument('next', '')},
      File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 1043, in render_template
        template = self.get_template(name)
      File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 1037, in get_template
        return self.settings['jinja2_env'].get_template(name)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template
        return self._load_template(name, self.make_globals(globals))
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/environment.py", line 857, in _load_template
        template = self.loader.load(self, name, globals)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 426, in load
        return loader.load(environment, name, globals)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 426, in load
        return loader.load(environment, name, globals)
      File "/root/miniconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 426, in load
        return loader.load(environment, name, globals)
      [Previous line repeated 982 more times]
    RecursionError: maximum recursion depth exceeded

How to reproduce

Start jupyterhub server with NativeAuth
Allow ~15 users to connect and run things for a few months. Or possibly by having the browser of a user call an HTTP GET about 1900 times.
First recursion depth error occurs as described above, followed by the second recursion depth error as described above
Server can no longer be connected to by new connects (existing connections do persist, but if I open a link to server in an incogento tab, I cannot connect).

Your personal set up

OS:
Linux
Version:

jupyterhub version 1.0.0

Configuration:

Dockerspawner

The text was updated successfully, but these errors were encountered:

welcome · 2020-09-30T14:31:29Z

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

gsteele13 · 2020-09-30T14:32:53Z

Server can no longer be connected to by new connects (existing connections do persist, but if I open a link to server in an incogento tab, I cannot connect).

This also makes me suspect that the problem is related to the NativeAuth module

djangoliv · 2020-10-06T11:20:08Z

I also had this problem on two instances of jupyterhub with nativeauthentificator (Sometimes with few users and in less than a week).
My instance with ldap anthentificator has never encountered this problem.

fbessou · 2020-10-19T11:02:18Z

It seems to be caused by the jinja env.loader being wrapped in a new loader at each render.
It happens here

The env.loader was replaced by a new ChoiceLoader wrapping the previous loader on every render. It seems that the `_loaded` attribute was here to avoid re-registering the loader but it was set to `True` on the handler instance and not on its class making it useless since the a handler instance is created for each request. Closes jupyterhub#114.

fbessou · 2020-10-20T07:43:35Z

This bug can be easily reproduced by sending a lot of requests to the /hub/login page.

while true; do $JUPYTERHUB_BASE_URL/hub/login -s > /dev/null; done

It starts to crash after around 10 seconds on my machine.

The env.loader was replaced by a new ChoiceLoader wrapping the previous loader on every render. It seems that the `_loaded` attribute was here to avoid re-registering the loader but it was set to `True` on the handler instance and not on its class making it useless since the a handler instance is created for each request. Closes jupyterhub#114.

gsteele13 added the bug Something isn't working label Sep 30, 2020

fbessou mentioned this issue Oct 19, 2020

Ensure that jinja loader is not registered at each render #115

Merged

lambdaTotoro closed this as completed in #115 Oct 30, 2020

This was referenced Nov 12, 2020

login page stops working after a few days (error 500) jupyterhub/the-littlest-jupyterhub#631

Closed

Crash after 935 calls of login page #124

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime Error: maximum recursion depth #114

Runtime Error: maximum recursion depth #114

gsteele13 commented Sep 30, 2020 •

edited

Loading

welcome bot commented Sep 30, 2020

gsteele13 commented Sep 30, 2020

djangoliv commented Oct 6, 2020

fbessou commented Oct 19, 2020

fbessou commented Oct 20, 2020

Runtime Error: maximum recursion depth #114

Runtime Error: maximum recursion depth #114

Comments

gsteele13 commented Sep 30, 2020 • edited Loading

Bug description

Expected behaviour

Actual behaviour

How to reproduce

Your personal set up

welcome bot commented Sep 30, 2020

gsteele13 commented Sep 30, 2020

djangoliv commented Oct 6, 2020

fbessou commented Oct 19, 2020

fbessou commented Oct 20, 2020

gsteele13 commented Sep 30, 2020 •

edited

Loading