Command killed due to excessive memory consumption; help with reducing memory

[yoelcortes]

Details

• Read the Docs project URL: https://readthedocs.org/projects/biosteam/
• Build URL (if applicable): https://readthedocs.org/projects/biosteam/builds/14492749/
• Read the Docs username (if applicable): yoelcortes

Problem

The project build stops when reading sources due to excessive memory consumption. This is similar to #6412, but maybe the cause is not the same as I do not use a conda installation and it stops only after everything is installed.

If there is anything I can do to reduce memory consumption, please let me know. Any help on this would greatly appreciated!

Thanks!

[humitos]

The error message is not accurate. It seems it reached the build time limit (900s). I increased it to be 1200s and triggered another build at: https://readthedocs.org/projects/biosteam/builds/14500657/

Let's see if it pass now. Otherwise, we can take another look.

[humitos]

1200s was not enough either :( --BTW, what's your Sphinx code doing that takes so much time?

[humitos]

1800s was no enough, either 🙃

https://readthedocs.org/projects/biosteam/builds/14500857/

[humitos]

Your build was taking 163s 1 week ago: https://readthedocs.org/projects/biosteam/builds/14438536/ and this started failing because of build time https://readthedocs.org/projects/biosteam/builds/14447089/.

So, there is a problem with the docs between this commit 57fb2f284e4518d59477f259f8354551e808f6ad and 751ced554874488d92bc156f9852442eaac23b9e I think.

[yoelcortes]

Thank you so much for looking into this.

I'm on Sphinx v3.5.4. It takes me less than a minute to run "make html" on my computer:
BioSTEAM_make_html.txt

I can switch to another version of Sphinx on my computer to check if that reproduces the error (just let me know).

I cannot find what could have made the error between the commits, but perhaps my dependencies made some changes that led to the problem... Let me know if you have any more ideas or things I could fix up.

Thanks!

[yoelcortes]

Update:

I tried building the docs with Sphinx v1.8.5 and it worked well on my computer, so maybe it's not a problem with the sphinx version...

BioSTEAM_make_html_v1_8_5.txt

Hope this helps,
THanks!

[yoelcortes]

Sorry, one more update:
Turns out that I had sphinx in my requirements.txt, which led to it installing the latest version of sphinx (after installing v1.8.5). Although the docs was building just fine two weeks ago even with sphinx in the requirements.txt, I removed sphinx from the requirements.txt anyway as I don't think I need it and, as expected, it didn't fix the issue.

[astrojuanlu]

Hi @yoelcortes , just one extra thing - I notice that you're pushing to master to debug Read the Docs. Notice that you can enable our pull request builder to use a branch instead: https://blog.readthedocs.com/pull-request-builder-general-availability/

[astrojuanlu]

I'm trying to reproduce the issue locally and it also gets stuck for me. In particular, the tutorial/Building_a_biorefinery document seems to be the culprit.

@yoelcortes I see that you're doing make clean before repeating the build, so in principle that should remove all the intermediate caching.

I can't point out at the moment what the problem is, but I don't think it's a Read the Docs issue. Will report back later if I find anything.

[astrojuanlu]

Well, something very weird goes on here.

1. I ran tutorial/Building_a_biorefinery.ipynb manually without problems from a Jupyter notebook. It took maybe 1 minute, much more than what Sphinx + nbsphinx took to render it.
2. I tried running tutorial/Creating_a_System but it contained some errors. Still, these were not told by nbsphinx.

Moreover, nbsphinx_execute = 'never' is set on conf.py! So the notebooks are not even being executed.

Profiling it with py-spy, it looks like all the time is spent here:

https://sourceforge.net/p/docutils/code/HEAD/tree/trunk/docutils/docutils/parsers/rst/states.py#l1036

    def implicit_inline(self, text, lineno):
        """
        Check each of the patterns in `self.implicit_dispatch` for a match,
        and dispatch to the stored method for the pattern.  Recursively check
        the text before and after the match.  Return a list of `nodes.Text`
        and inline element nodes.
        """
        if not text:
            return []
        for pattern, method in self.implicit_dispatch:
            match = pattern.search(text)

therefore, my guess is that the regex search is stuck in some of the notebook blobs.

[astrojuanlu]

Killed the process after 20 minutes, cleared all the outputs of all the cells, and the make html finished quickly.

Removed _build, restored all the outputs of all the cells, and make html got stuck again.

At least on my computer this is 100 % reproducible and I confirm it's not a RTD problem. Can't point exactly at the root cause though.

[yoelcortes]

Thanks @astrojuanlu! With this, I think I can pinpoint which cell is the problem, report back, and just not run the cell for the meantime until the issue is solved.

[yoelcortes]

@astrojuanlu, @humitos I found the solution to the bug!

It turns out that in nbsphinx v0.8.7, you should not have 'ipynb' in the source_suffix variable of conf.py as it messes up the parser. This wasn't a problem in v0.8.6:
spatialaudio/nbsphinx#595

Closing this issue now,
Thanks for your help!