SystemError: excessive stack use when compiling a large function

[oscarbenjamin]

Bug report

This comes from a recently discovered SymPy issue: sympy/sympy#23690

The following code fails in CPython 3.10 or newer for large N apparently due to the changes in #25403 (CC @markshannon):

#!/usr/bin/env python3

template = """\
def %(name)s(A, B):
    (%(A)s,) = A
    (%(B)s,) = B
    return (%(AB)s,)
"""

def make_function(N):
    name = "monomial_mul"
    A = [f"a{i}" for i in range(N)]
    B = [f"b{i}" for i in range(N)]
    AB = [ "%s + %s" % (a, b) for a, b in zip(A, B) ]
    code = template % dict(name=name, A=", ".join(A), B=", ".join(B), AB=", ".join(AB))
    ns = {}
    exec(code, ns)
    return ns["monomial_mul"]

if __name__ == "__main__":
    from sys import argv
    make_function(int(argv[1]))

In CPython 3.9 or older this can run fine albeit slow for large values of N (I tested 100000).

With CPython 3.10 or current main (3.11b3) this will fail with SystemError for any N greater than 3000:

$ ./python test.py 3000
$ ./python test.py 3001
Traceback (most recent call last):
  File "/home/oscar/current/sympy/cpython/test.py", line 22, in <module>
    make_function(int(argv[1]))
  File "/home/oscar/current/sympy/cpython/test.py", line 17, in make_function
    exec(code, ns)
SystemError: excessive stack use: stack is 3001 deep

I wasn't sure that this was a deliberate effect of the changes or if it should definitely be considered a bug but the comment added in #25403 indicates that it should be:

cpython/Python/compile.c

Lines 49 to 71 in c0453a4

	/* A soft limit for stack use, to avoid excessive
	* memory use for large constants, etc.
	*
	* The value 30 is plucked out of thin air.
	* Code that could use more stack than this is
	* rare, so the exact value is unimportant.
	*/
	#define STACK_USE_GUIDELINE 30
	/* If we exceed this limit, it should
	* be considered a compiler bug.
	* Currently it should be impossible
	* to exceed STACK_USE_GUIDELINE * 100,
	* as 100 is the maximum parse depth.
	* For performance reasons we will
	* want to reduce this to a
	* few hundred in the future.
	*
	* NOTE: Whatever MAX_ALLOWED_STACK_USE is
	* set to, it should never restrict what Python
	* we can write, just how we compile it.
	*/
	#define MAX_ALLOWED_STACK_USE (STACK_USE_GUIDELINE * 100)

Your environment

Tested on Ubuntu with a range of Python versions.

[markshannon]

I wasn't sure that this was a deliberate effect of the changes or if it should definitely be considered a bug

This is a bug. Thanks for the report.

[DanielNoord]

@markshannon This seems to have been back ported to all relevant versions in #94448, #94446 and #94421

I think it can be closed!

[oscarbenjamin]

I guess since this is immediately fixed on relevant branches and #94445 is the continuation issue to look for a better long term fix then this can be closed.