Reimplement parallel docbuild using make

[mkoeppe]

Based on #31353 (sage --docbuild: Add options to list all documents), we can remove the issues with parallel (#31344) and serial (#31936) docbuild by restructuring the docbuild using make.

Blocker because it solves blocker issue #31936.

[jhpalmieri]

I'm not quite sure how to go about this. I tried this:

build/make/Makefile.in

@@ -335 +335 @@
        $(inst_ipykernel) $(inst_jupyter_client) $(inst_conway_polynomials) \
        $(inst_tachyon) $(inst_jmol) $(inst_thebe) $(inst_ipywidgets)
 
-doc: doc-html
+doc: doc-new
+
+doc-new: $(DOC_DEPENDENCIES)
+       for doc in $(shell cd $(SAGE_ROOT) && ./sage --docbuild --all-documents all); do \
+           $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links $$doc html $(SAGE_DOCBUILD_OPTS)" logs/dochtml.log; \
+       done
 
 doc-html: $(DOC_DEPENDENCIES)
        $(AM_V_at)cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html $(SAGE_DOCBUILD_OPTS)' logs/dochtml.log

and it seems to build the documentation, but it doesn't do it in parallel, I think because we've turned off parallel processing for recursive calls to make earlier in the file.

If it did work, this approach would need a little tinkering: we should build the reference manual first, then everything else.

A separate question: why do we use cd ../.. in the recipe for doc-html (and other doc targets), rather than cd $(SAGE_ROOT)?

[jhpalmieri]

And if we really want to pull all of the parallel building out of the current system, then we should build the pieces of the reference manual separately in the Makefile. I think this is what you intended.

[jhpalmieri]

Here is my current attempt, which doesn't quite work:

build/make/Makefile.in

@@ -335 +335 @@
        $(inst_ipykernel) $(inst_jupyter_client) $(inst_conway_polynomials) \
        $(inst_tachyon) $(inst_jmol) $(inst_thebe) $(inst_ipywidgets)
 
-doc: doc-html
+doc: doc-references doc-other
+
+doc-references: $(DOC_DEPENDENCIES)
+       $(eval DOCS = $(shell cd $(SAGE_ROOT) && ./sage --docbuild --all-documents reference))
+       $(eval biblio = $(firstword $(DOCS)))
+       $(eval other_docs = $(wordlist 2, 100, $(DOCS)))
+       $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links $(biblio) inventory $(SAGE_DOCBUILD_OPTS)" logs/docs-reference-html.log; \
+       for doc in $(other_docs); do \
+           $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links $$doc inventory $(SAGE_DOCBUILD_OPTS)" logs/docs-reference-html.log; \
+       done
+       $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links reference inventory $(SAGE_DOCBUILD_OPTS)" logs/docs-reference-html.log; \
+       $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links $(biblio) html $(SAGE_DOCBUILD_OPTS)" logs/docs-reference-html.log; \
+       for doc in $(other_docs); do \
+           $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links $$doc html $(SAGE_DOCBUILD_OPTS)" logs/docs-reference-html.log; \
+       $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links reference html $(SAGE_DOCBUILD_OPTS)" logs/docs-reference-html.log; \
+       done
+
+doc-other: $(DOC_DEPENDENCIES) doc-references
+       for doc in $(wordlist 2, 100, $(shell cd $(SAGE_ROOT) && ./sage --docbuild --all-documents all)); do \
+           $(AM_V_at)cd $(SAGE_ROOT) && sage-logger -p "./sage --docbuild --no-pdf-links $$doc html $(SAGE_DOCBUILD_OPTS)" logs/docs-other-html.log; \
+       done
 
 doc-html: $(DOC_DEPENDENCIES)
        $(AM_V_at)cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html $(SAGE_DOCBUILD_OPTS)' logs/dochtml.log

src/sage_docbuild/__init__.py

@@ -116 +116 @@
             # WEBSITESPHINXOPTS is either empty or " -A hide_pdf_links=1 "
             options += WEBSITESPHINXOPTS
 
-        if kwds.get('use_multidoc_inventory', True):
+        if kwds.get('use_multidoc_inventory', True) and type != 'inventory':
             options += ' -D multidoc_first_pass=0'
         else:
             options += ' -D multidoc_first_pass=1'

Several problems:

• lack of parallel building
• line 348 should be replaced by something that just builds the top-level reference manual stuff, assembling the parts into one set of indices, etc. Probably the same for line 352.
• I'm getting some warning messages that I don't know what to make of: "[schemes ] Handler <function on_build_finished at 0x33872ef70> for event 'build-finished' threw an exception", for example.

[mkoeppe]

To get make to parallelize it, the individual documents need to become make targets. A combination of $(eval...) and a macro call can do this -- similar to ​https://www.gnu.org/software/make/manual/html_node/Eval-Function.html

[jhpalmieri]

I don't see how to do anything like that unless we hardcode the list of documents: we can't run sage --docbuild ... without building some other targets, so it seems to me that we have to construct the list of documents inside a recipe. But I'm very far from a Makefile expert, so I could be missing something.

[jhpalmieri]

Or write a standalone script to construct the document list.

[mkoeppe]

Could you push your current version to the ticket please?

[jhpalmieri]

Here it is.

---

New commits:

​c2c9b3c

trac 31948: very rough draft

[mkoeppe]

Here's a solution using recursive make invocation

---

New commits:

​393ab65

build/make/Makefile.in (doc-reference): Invoke make recursively to build documents in parallel

[jhpalmieri]

This doesn't work for me, unfortunately. If I do make doc-clean && make doc, I get this error, which I don't understand.

[reference] building [inventory]: targets for 1 source files that are out of date
[reference] updating environment: [new config] 1 added, 0 changed, 0 removed
[reference] The inventory files are in local/share/doc/sage/inventory/en/reference/references.
Build finished. The built documents can be found in /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/share/doc/sage/inventory/en/reference/references
Deleting empty directory /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/src/doc/en/reference/manifolds/sage/manifolds
Deleting empty directory /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/src/doc/en/reference/manifolds/sage
Error building the documentation.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1762, in main
    builder()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 769, in _wrapper
    self.write_auto_rest_file(module_name)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1070, in write_auto_rest_file
    with open(filename, 'w') as outfile:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/src/doc/en/reference/manifolds/sage/manifolds/utilities.rst'

With the following change, the documentation builds:

build/make/Makefile.in

@@ -349 +349 @@
        $(eval DOCS = $(shell cd $(SAGE_ROOT) && ./sage --docbuild --all-documents reference))
        $(eval biblio = $(firstword $(DOCS)))
        $(eval other_docs = $(wordlist 2, 100, $(DOCS)))
-       +$(MAKE_REC) doc-inventory-$(subst /,-,$(biblio))
-       +$(MAKE_REC) $(foreach doc, $(other_docs), doc-inventory-$(subst /,-,$(doc)))
        +$(MAKE_REC) doc-inventory-reference
-       +$(MAKE_REC) doc-html-$(subst /,-,$(biblio))
-       +$(MAKE_REC) $(foreach doc, $(other_docs), doc-html-$(subst /,-,$(doc)))
        +$(MAKE_REC) doc-html-reference
 
 doc-other: $(DOC_DEPENDENCIES) doc-references

This is a hybrid model, using the old parallel building methods for the reference manual. The inventory build is fine, but I get some strange warnings during the html build:

[repl     ] building [html]: targets for 35 source files that are out of date
[polynomia] building [html]: targets for 62 source files that are out of date
[spkg     ] building [html]: targets for 316 source files that are out of date
[tensor_fr] building [html]: targets for 19 source files that are out of date
[algebras ] building [html]: targets for 97 source files that are out of date
[manifolds] building [html]: targets for 81 source files that are out of date
[repl     ] writing additional pages...  searchdone
[repl     ] dumping search index in English (code: en)... done
[repl     ] The HTML pages are in local/share/doc/sage/html/en/reference/repl.
[repl     ] Extension error:
[repl     ] Handler <function on_build_finished at 0x33c247af0> for event 'build-finished' threw an exception
Build finished. The built documents can be found in /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/share/doc/sage/html/en/reference/repl
[tensor_fr] writing additional pages...  searchdone
[tensor_fr] dumping search index in English (code: en)... done
[tensor_fr] The HTML pages are in local/share/doc/sage/html/en/reference/tensor_free_modules.
[tensor_fr] Extension error:
[tensor_fr] Handler <function on_build_finished at 0x33d1c3040> for event 'build-finished' threw an exception

I don't know what Handler <function on_build_finished at 0x33d1c3040> for event 'build-finished' threw an exception means or where it comes from.

[mkoeppe]

Replying to jhpalmieri:

This doesn't work for me, unfortunately. If I do make doc-clean && make doc, I get this error, which I don't understand.

[reference] building [inventory]: targets for 1 source files that are out of date
[reference] updating environment: [new config] 1 added, 0 changed, 0 removed
[reference] The inventory files are in local/share/doc/sage/inventory/en/reference/references.
Build finished. The built documents can be found in /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/share/doc/sage/inventory/en/reference/references
Deleting empty directory /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/src/doc/en/reference/manifolds/sage/manifolds
Deleting empty directory /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/src/doc/en/reference/manifolds/sage
Error building the documentation.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1762, in main
    builder()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 769, in _wrapper
    self.write_auto_rest_file(module_name)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1070, in write_auto_rest_file
    with open(filename, 'w') as outfile:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.4.beta1/src/doc/en/reference/manifolds/sage/manifolds/utilities.rst'

Looks like something forgot to create the directory in which this .rst file is to be created

[mkoeppe]

I was able to reproduce an error of this form as well.

[mkoeppe]

If you do make doc-clean && make SAGE_DOCBUILD_OPTS="--verbose=3" doc, you can see that this is a race between creating directories and "deleting empty directories"

[mkoeppe]

I'll add an option to turn it off

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d08633f

build/make/Makefile.in, src/sage_docbuild/__init__.py: New option --no-prune-empty-dirs, use it to remove races

[jhpalmieri]

I still see

[valuation] Extension error:
[valuation] Handler <function on_build_finished at 0x3375808b0> for event 'build-finished' threw an exception

for (I think) every piece of the reference manual. Any ideas about this?

Edit: the manual seems to build correctly, regardless.

[mkoeppe]

No idea about this error. I don't see it here. I suppose you could try to run the docbuild for a document in pdb to debug this.

[jhpalmieri]

I haven't tried pdb, but I get it when doing make doc-clean && make — running make again at that point doesn't produce the error. I also get it on two different machines, one Big Sur, one Catalina.

One more task: the reference manual is built twice: the line

+$(MAKE_REC) SAGE_DOCBUILD_OPTS="$(SAGE_DOCBUILD_OPTS) --no-prune-empty-dirs" doc-inventory--reference

needs to be replaced with something that just builds the top-level ref manual indices, and similarly for the html build. I may be able to work on this later, but please go ahead if you want to take a stab at it.

[mkoeppe]

If it shows up again, perhaps #31005 (Add traceback information to exceptions during docbuild) can help to diagnose it

[mkoeppe]

OK, I see Handler <function on_build_finished at 0x3b9411310> for event 'build-finished' threw an exception now too

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​06c6365

build/make/Makefile.in: Build doc-pdf in parallel using make

[mkoeppe]

Here's an attempt to do the same for PDF.

[jhpalmieri]

When I merged with #31005, I didn't see any extra information.

[dimpase]

I tried the html build here, and the Japanese is gone. I see that Handler <function on_build_finished... messages too. Something to configure for sphinx, I gather. The default on_build_finished() needs to be replaced by something we should do (maybe just pass)

[dimpase]

src/docs/conf.py:              'sphinx.ext.graphviz'

and in Sphinx in sphinx/ext/graphviz.py there is

def on_build_finished(app: Sphinx, exc: Exception) -> None:
    if exc is None:
        src = path.join(sphinx.package_dir, 'templates', 'graphviz', 'graphviz.css')
        dst = path.join(app.outdir, '_static')
        copy_asset(src, dst)

[jhpalmieri]

Okay, I'm annoyed. First, we've ​had problems with the Sphinx graphviz extension before. Why do we even need it? As far as I can tell, it only ​provides Sphinx directives .. graphviz::, .. graph, and .. digraph::, and I don't see any of those in our code. Second, it's labeled an optional Sage package, so why load the extension all of the time? Third, what sort of ridiculous package is it? Its spkg-install.in script just says

#! /usr/bin/env bash
echo Error: graphviz is not installed as a system package.
echo Please install the system package recommended by ./configure
exit 1

Can we delete it?

The documentation builds without error if we remove sphinx.ext.graphviz and the related sphinx.ext.inheritance_diagram from the list of Sphinx extensions. Can we remove these extensions?

[mkoeppe]

This type of packages is there to make system package information available -- this is what will show at the end of ./configure.

[mkoeppe]

I'm pretty sure that this sphinx extension is not used anywhere in our documentation. It's probably a leftover from antiquity

[jhpalmieri]

Replying to mkoeppe:

This type of packages is there to make system package information available -- this is what will show at the end of ./configure.

I think it would make more sense for graphviz to be part of _recommended rather than its own package.

[mkoeppe]

No, that would not be an improvement. Our configure script detects the presence of system features on the granularity of packages. A catch-all "_recommended" package can't do that. See #30930.

In any case, all of this is unrelated to the errors in the present ticket.

[jhpalmieri]

Do we want to deprecate the use of ./sage --docbuild all ..., while keeping ./sage --docbuild DOCUMENT ... for specific documents?

[dimpase]

graphviz sphinx extension came in #7549 (in 2009)

[jhpalmieri]

Replying to mkoeppe:

No, that would not be an improvement. Our configure script detects the presence of system features on the granularity of packages. A catch-all "_recommended" package can't do that. See #30930.

In any case, all of this is unrelated to the errors in the present ticket.

I agree with all of this, but I don't understand why graphviz is treated differently than ffmpeg, for example.

[mkoeppe]

Replying to jhpalmieri:

Do we want to deprecate the use of ./sage --docbuild all ..., while keeping ./sage --docbuild DOCUMENT ... for specific documents?

I think that's a good idea.

[jhpalmieri]

Replying to dimpase:

graphviz sphinx extension came in #7549 (in 2009)

Thank you, Dima. Let's delete those extensions.

[mkoeppe]

Replying to jhpalmieri:

I don't understand why graphviz is treated differently than ffmpeg, for example.

ffmpeg should also be broken out of _recommended and transformed to its separate script package. When I created _recommended, I did not anticipate that users would be confused when the configure script recommends to install things that are already installed.

[jhpalmieri]

I think that we could also delete the cleanup method in src/sage_docbuild/ext/inventory_builder.py: its docstring says

        Remove the '_static' directory.

        This directory is unnecessary for the inventory build, but it
        may be created by the graphviz extension. Its presence will
        break the docbuild later on, so remove it.

But let's keep it for now. It doesn't cause any harm.

---

New commits:

​501d3fa

trac 31948: no longer use sphinx.ext.graphviz or

[dimpase]

Replying to jhpalmieri:

Replying to dimpase:

graphviz sphinx extension came in #7549 (in 2009)

Thank you, Dima. Let's delete those extensions.

Sure. Reading #7549 tells that it was meant for ​https://www.sphinx-doc.org/en/master/usage/extensions/inheritance.html

[jhpalmieri]

Replying to mkoeppe:

Replying to jhpalmieri:

Do we want to deprecate the use of ./sage --docbuild all ..., while keeping ./sage --docbuild DOCUMENT ... for specific documents?

I think that's a good idea.

Actually, can we do this? I don't know enough about distro use: can we really recommend "make doc" and "make doc-pdf" as replacements?

[mkoeppe]

Replying to jhpalmieri:

Replying to mkoeppe:

Replying to jhpalmieri:

Do we want to deprecate the use of ./sage --docbuild all ..., while keeping ./sage --docbuild DOCUMENT ... for specific documents?

I think that's a good idea.

Actually, can we do this? I don't know enough about distro use: can we really recommend "make doc" and "make doc-pdf" as replacements?

I'd say just adding a deprecation warning won't break anything.

Having these make recipes in our main Makefile probably shouldn't be the final form of this. In #31356, I hope that we can restructure the docbuild as script packages (sagemath_doc_html, sagemath_doc_pdf), and in the next step as pip-installable packages (#29868).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6f60fd9	trac 31948: deprecate "sage --docbuild all ..."
​4057d34	trac 31948: implement "sage --docbuild reference_top FORMAT"

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0fee939

trac 31948: implement "sage --docbuild reference_top FORMAT"

[jhpalmieri]

Okay, here is a deprecation warning, plus my attempt to add a builder for just the top-level reference manual document(s). It looks like it works, but it needs more testing.

[jhpalmieri]

(In general I feel like the docbuilding code should be torn down and rebuilt from scratch; I find it very confusing and hard to work with.)

[mkoeppe]

The HTML build seems to work well for me, even with high parallelization. PDF stops somewhere with LaTeX complaining about an undefined reference and some unicode characters, but that may be coming from some tickets that I have merged into my development branch.

I'd say let's get this in and see how this works for other people

[mkoeppe]

[docpdf] Latexmk: Log file says output to 'combinat.pdf'
[docpdf] Latexmk: List of undefined refs and citations:
[docpdf]   Reference `index:module-sage.combinat' on page 3639 undefined on input line 348243
[docpdf] Latexmk: Summary of warnings from last run of *latex:
[docpdf]   Latex failed to resolve 1 reference(s)
[docpdf]   =====Latex reported missing or unavailable character(s).
[docpdf] =====See log file for details.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​1c41284

trac 31948: implement "sage --docbuild reference_top FORMAT"

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​b17b674

trac 31948: implement "sage --docbuild reference_top FORMAT"

[jhpalmieri]

I'm struggling to get this to work. The current branch works for me when building the html documentation with make doc, but ./sage --docbuild all html fails with

Error building the documentation.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/palmieri/Desktop/Sage/git/sage/local/lib/python3.9/site-packages/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/Users/palmieri/Desktop/Sage/git/sage/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1811, in main
    builder()
  File "/Users/palmieri/Desktop/Sage/git/sage/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 345, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/Users/palmieri/Desktop/Sage/git/sage/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 589, in _wrapper
    self._build_top_level(format, *args, **kwds)
  File "/Users/palmieri/Desktop/Sage/git/sage/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 571, in _build_top_level
    getattr(ReferenceTopBuilder(), format)(*args, **kwds)
  File "/Users/palmieri/Desktop/Sage/git/sage/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 627, in __init__
    DocBuilder.__init__(self, *args, **kwds)
TypeError: __init__() missing 1 required positional argument: 'name'

If I add an argument, for example "reference", to the call

        DocBuilder.__init__(self, *args, **kwds)

in ReferenceTopBuilder.__init__, then make doc fails.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​06dfb90

trac 31948: fix "help_documents"

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8930551

trac 31948: repair "sage --docbuild all html"

[jhpalmieri]

I think that fixes my problem. I've seen the combinat.pdf failure once, but I can't reproduce it. I'll keep trying.

[dimpase]

on Lunix I am seeing

[dochtml] [combinat ] updating environment: [new config] 382 added, 0 changed, 0 removed
[dochtml] [manifolds] building [inventory]: targets for 81 source files that are out of date
[dochtml] [manifolds] updating environment: [new config] 81 added, 0 changed, 0 removed
[dochtml] Error, reached the pre-set memory limit
[dochtml] (change it with the -o command line option)
[dochtml] Error, was not in any namespace

not sure what limit is involved here and why.

[dimpase]

otherwise, all works on Linux, both html and pdf builds.

[jhpalmieri]

Replying to dimpase:

on Lunix I am seeing

[dochtml] [combinat ] updating environment: [new config] 382 added, 0 changed, 0 removed
[dochtml] [manifolds] building [inventory]: targets for 81 source files that are out of date
[dochtml] [manifolds] updating environment: [new config] 81 added, 0 changed, 0 removed
[dochtml] Error, reached the pre-set memory limit
[dochtml] (change it with the -o command line option)
[dochtml] Error, was not in any namespace

not sure what limit is involved here and why.

Is that error new? Is it repeatable?

[dimpase]

it might be due to the new GAP 4.11.1 I have in my branch (this message comes from GAP, that's certain). Let's ignore it now. I'll try to reproduce it on another system.

[jhpalmieri]

This also works for me with the Sphinx upgrade at #31696.

[dimpase]

I wonder if it is easy to get rid of build_many(), and replace it by a make-based parallelisation.

[jhpalmieri]

Replying to dimpase:

I wonder if it is easy to get rid of build_many(), and replace it by a make-based parallelisation.

Probably, but on another ticket. Should we start using src/doc/Makefile for more than just cleaning? As I said earlier, I am unclear on the distro situation: I don't know what packages are going to be available in general, and in particular in this case: if we have the documentation source files, can we be guaranteed to have sage_docbuild?

In any case, this may all fit somehow into #31356 and related tickets.

[mkoeppe]

Replying to jhpalmieri:

Replying to dimpase:

I wonder if it is easy to get rid of build_many(), and replace it by a make-based parallelisation.

Probably, but on another ticket.

I agree.

Should we start using src/doc/Makefile for more than just cleaning?

I think that's a good idea. The things that we added in build/make/Makefile.in could just be moved there.

[jhpalmieri]

Replying to mkoeppe:

Replying to jhpalmieri:

Should we start using src/doc/Makefile for more than just cleaning?

I think that's a good idea. The things that we added in build/make/Makefile.in could just be moved there.

I'm experimenting with this and it seems to be working. A few questions about our Makefiles:

• Where is SAGE_ROOT defined? It appears to work if I just use it in src/doc/Makefile, but I don't know why.

• Do I need to use +$(MAKE_REC) or can I just use $(MAKE)? The latter seems to work.

• Do I need to define AM_V_at in this Makefile, or does it inherit the definition from build/make/Makefile?

[mkoeppe]

Replying to jhpalmieri:

Replying to mkoeppe:

Replying to jhpalmieri:

Should we start using src/doc/Makefile for more than just cleaning?

I think that's a good idea. The things that we added in build/make/Makefile.in could just be moved there.

I'm experimenting with this and it seems to be working. A few questions about our Makefiles:

• Where is SAGE_ROOT defined? It appears to work if I just use it in src/doc/Makefile, but I don't know why.

It comes from the environment and is set in build/make/install.

• Do I need to use +$(MAKE_REC) or can I just use $(MAKE)? The latter seems to work.

$(MAKE) is fine here - ​https://www.gnu.org/software/make/manual/make.html#MAKE-Variable

• Do I need to define AM_V_at in this Makefile, or does it inherit the definition from build/make/Makefile?

You'd need to copy this definition from Makefile.in; it's not exported

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​522b450

trac 31948: use src/doc/Makefile for docbuilding commands

[jhpalmieri]

That's helpful, thank you. Here's an attempt. I'm not sure how to handle SAGE_ROOT in src/doc/Makefile: if it's not set, exit with an error? That's my current approach.

The new branch keeps the logging in build/make/Makefile: if someone wants to use src/doc/Makefile to build the documentation, maybe they don't want it logged. This seems consistent with how we do logging in general. It does mean that make doc-pdf logs all docbuilding, including inventory building, in docpdf.log, whereas it used to put the inventory logs into dochtml.log.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​3d90c59

trac 31948: use src/doc/Makefile for docbuilding commands

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2c3a9ae

trac 31948: move some "$(AM_V_at)"s around

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​54e68e0

trac 31948: move some "$(AM_V_at)"s around

[jhpalmieri]

Okay, I think I'm done tinkering now.

[mkoeppe]

Looks good and seems to work well. Positive review from my side

[jhpalmieri]

Dima, any comments? I'm happy with Matthias' contributions.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​53f260c

trac 31948: fix doctests

[dimpase]

looks good, thanks!

[jhpalmieri]

Great, thank you!

[klee]

Previously sage --docbuild all pdf built a main website listing all documentations with icons linking to pdf files. Now it seems that make doc-pdf does not build such an website. Is there a way to build the website?

[mkoeppe]

I've opened #32043 for this

[dimpase]

I am seeing a race condition, sphinx may race FileExistsError, cf. e.g.

[dochtml] [reference] preparing documents... skipping loading of indexes... done
[dochtml] [reference] Merging js index files...
[dochtml] [reference]     algebras: 4424 js index entries
[dochtml] [reference]     arithgroup: 1204 js index entries
...

[dochtml] [reference]     topology: 1937 js index entries
[dochtml] [reference]     valuations: 973 js index entries
[dochtml] [reference] ... done (64387 js index entries)
[dochtml] [reference] copying static files... failed
[dochtml] [reference] WARNING: cannot copy static file
FileExistsError(17, 'File exists')
[dochtml] [reference] dumping search index in English (code: en)... done
[dochtml] [reference] The HTML pages are in
local/share/doc/sage/html/en/reference.
[dochtml] Error building the documentation.

make -j1 allows the docs to build.

See e.g. ​https://github.com/sagemath/sagetrac-mirror/runs/3104375540

It's of course not totally surprising that several processes may try to, say, create the same directory. Perhaps the relevant part of the builder should be launched with this specific error being ignored?

[jhpalmieri]

It's strange because that doesn't look like part of the docbuild process that's done in parallel with anything else: it looks like the main reference page, built after the other parts of the reference manual. How reliably can you reproduce this?

I think this part of the process (copying static files) is controlled by Sphinx, not by Sage or its modifications to Sphinx, for what that's worth.

[mkoeppe]

Replying to dimpase:

I am seeing a race condition, sphinx may race FileExistsError

new ticket please?

[klee]

Replying to mkoeppe:

Replying to dimpase:

I am seeing a race condition, sphinx may race FileExistsError

new ticket please?

This is now #32262