Incremental docbuild broken on 7.3

[infinity0]

Whilst trying to package sagemath for Debian: ​https://anonscm.debian.org/cgit/debian-science/packages/sagemath.git/

During the docbuild, if I run:

$ find ./src/doc/en/reference/ -type f -exec ls -l --time-style=long-iso '{}' \; | sort -k6,7

I can see that the .rst files get touched during the docbuild. This breaks incremental build and makes it very time-consuming to debug segfaults.

I have already tried applying build/pkgs/sphinx/patches/environment.patch to my system sphinx. The patch applies, but the above behaviour remains.

[mantepse]

Is this the same as #21611?

[infinity0]

No, it's definitely different. This is about incremental build, errors are not relevant.

More information: the doctrees and inventory is currently getting built in ~/.local/share/sagemath/doc, but the rst files are being built in sage.git/src/doc/en/reference and they're being deleted and re-written every time.

I'm testing out a patch for it. It's a massive hack, but it seems to be working and it'll take just a bit more time for me to make sure. Basically, fixing up the logic in write_auto_rest_file and commenting out some cleaning logic, both in src/sage_setup/docbuild/__init__.py.

[infinity0]

Ok, here is the patch I'm using. Incremental build works again, but as you can see, it's quite a hack and I don't know how this affects the rest of sage. ​https://xkcd.com/1739/ is relevant, sigh.

[infinity0]

Another bonus is that the docbuild (with the patch applied) seems to take less memory AND disk space.

For example, now my build actually finishes instead of sometimes getting killed due to OOM. And my doc/share/doctrees/en/reference/* build dir is now only 400MB instead of 14GB. I am fairly sure the 14GB included lots of duplicate entries - each subdir in there used to be 200MB, suspiciously all similar in size - whereas now each subdir is more proportional in size to the corresponding directory in the source repo.

[infinity0]

OK, it looks like memory usage is not really improved directly. If I try the docbuild from scratch, my system will still fail it with an OOM.

However, this patch allows me to re-run the build - now it will skip the already-built stuff, proceed directly to the memory-intensive "Merging environment/index files" step, and this time it will succeed. This suggests that there is a rather bad memory leak in the docbuilder - the skipped stuff was unnecessary to perform the merging, yet it still used up enough memory to crash the docbuilder the first time around.