Opened 17 months ago

Closed 8 months ago

#31356 closed enhancement (fixed)

Restructure docbuild as script packages sagemath_doc_html, sagemath_doc_pdf

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.5
Component: documentation Keywords:
Cc: jhpalmieri Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: fd298c4 (Commits, GitHub, GitLab) Commit: fd298c4b7a6f845335b862f6dc2f8e18753937e7
Dependencies: #30383 Stopgaps:

Status badges

Description (last modified by mkoeppe)

The specific code in Makefile.in for the documentation and its dependencies is moved to the new script packages.

In preparation for #32759:

  • We introduce another installation tree (in addition to SAGE_LOCAL and SAGE_VENV), called SAGE_DOCS. When the installation tree variable is empty, we suppress building the package.
  • We introduce a mechanism to select an installation tree, the file trees.txt. Using this mechanism, we associate the new script packages with the installation tree SAGE_DOCS.

Next step:

  • #32759 ./configure --disable-doc
  • #29868 pip-installable packages sagemath-doc-src, sagemath-doc-inventory, sagemath-doc-html, sagemath-doc-pdf
  • rename src/doc/bootstrap to build/pkgs/sagemath_doc_html/bootstrap.

Change History (52)

comment:1 Changed 17 months ago by mkoeppe

  • Branch set to u/mkoeppe/restructure_docbuild_as_optional_script_packages_sagemath_doc_html__sagemath_doc_pdf

comment:2 Changed 17 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:3 Changed 16 months ago by mkoeppe

  • Commit set to 42b771403d0aebf6c8ef9e97fda48eee4c543f53
  • Dependencies set to #30010

New commits:

42b7714WIP sagemath_doc_html sagemath_doc_pdf

comment:4 Changed 16 months ago by git

  • Commit changed from 42b771403d0aebf6c8ef9e97fda48eee4c543f53 to f68feab886142727b63f2419637e4b3f119b48b8

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

c262e44build/pkgs/sage_docbuild/src: Update metadata
52778c3build/pkgs/sage_docbuild/spkg-install: Use sdh_setup_bdist_wheel
b6abe5esrc/sage/docs/conf.py: Use absolute module names for sage_docbuild extensions
b42920btrac 30010: (typo) change sage_doctest -> sage_docbuild
0a472f2build/pkgs/sage_docbuild/src/setup.py: Restrict find_namespace_packages to sage_docbuild
1dfeb84build/make/Makefile.in (sage_docbuild-clean): New, run it from build-clean
881c4afsrc/sage_setup/__init__.py: Restore
4eca937build/pkgs/sage_docbuild/src/setup.{cfg,py}: Use an explicit list of packages, add install_requires
aaab1d3build/pkgs/sage_docbuild/dependencies: Add dependencies on .py files
f68feabbuild/pkgs/sagemath_doc_{html,pdf}: New, delegate 'make doc-{html,pdf}' to here

comment:5 Changed 16 months ago by git

  • Commit changed from f68feab886142727b63f2419637e4b3f119b48b8 to 0e05512988599a82ec65ee704297f544776c0cd6

Branch pushed to git repo; I updated commit sha1. New commits:

0e05512build/pkgs/sagemath_doc_html/dependencies: Add sage_docbuild

comment:6 Changed 16 months ago by git

  • Commit changed from 0e05512988599a82ec65ee704297f544776c0cd6 to 5c19f46afa2241c5f237486f7aad16a4dc6ff8d4

Branch pushed to git repo; I updated commit sha1. New commits:

d39fd70build/pkgs/sagemath_doc_html: Add SPKG.rst, type
5c19f46build/pkgs/sagemath_doc_html: Add package-version.txt

comment:7 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 16 months ago by mkoeppe

  • Dependencies changed from #30010 to #30383
  • Description modified (diff)
  • Summary changed from Restructure docbuild as optional script packages sagemath_doc_html, sagemath_doc_pdf to Restructure docbuild as script packages sagemath_doc_html, sagemath_doc_pdf

comment:9 Changed 16 months ago by git

  • Commit changed from 5c19f46afa2241c5f237486f7aad16a4dc6ff8d4 to 3cd70ffcc0336d7f7ea4c9fe1c8b78c93c0fb1c1

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

6dd12bdbuild/pkgs/sagemath_doc_{html,pdf}: New, delegate 'make doc-{html,pdf}' to here
ec43b4cbuild/pkgs/sagemath_doc_html/dependencies: Add sage_docbuild
862c6acbuild/pkgs/sagemath_doc_html: Add SPKG.rst, type
3cd70ffbuild/pkgs/sagemath_doc_html: Add package-version.txt

comment:10 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 12 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:13 Changed 8 months ago by git

  • Commit changed from 3cd70ffcc0336d7f7ea4c9fe1c8b78c93c0fb1c1 to d62fc4824724d9b51a2d20e3c5cc52d39d1525c2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d62fc48WIP sagemath_doc_html sagemath_doc_pdf

comment:14 Changed 8 months ago by git

  • Commit changed from d62fc4824724d9b51a2d20e3c5cc52d39d1525c2 to 0968c1cb47a210bd637f0c03adb78edd1d8e64ed

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0968c1cWIP sagemath_doc_html sagemath_doc_pdf

comment:15 Changed 8 months ago by git

  • Commit changed from 0968c1cb47a210bd637f0c03adb78edd1d8e64ed to 37718fbb281cec60148eae339fb9b73494e963a8

Branch pushed to git repo; I updated commit sha1. New commits:

6ab935dm4/sage_spkg_collect.m4: Determine install tree for python3 via new trees.txt mechanism
37718fbbuild/make/Makefile.in: Install doc via trees.txt = SAGE_DOC_PREFIX

comment:16 Changed 8 months ago by git

  • Commit changed from 37718fbb281cec60148eae339fb9b73494e963a8 to 8a6db35d39cea7faf3bd38617dea0247f73b7926

Branch pushed to git repo; I updated commit sha1. New commits:

8a6db35build/make/Makefile.in: WIP Disablable trees

comment:17 Changed 8 months ago by git

  • Commit changed from 8a6db35d39cea7faf3bd38617dea0247f73b7926 to eeb8777b0c31bbf15f0d97171c3eccce119cffac

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8e26780build/pkgs/sagemath_doc_{html,pdf}: New; delegate to them from targets doc-html etc.
eeb8777m4/sage_spkg_collect.m4: Determine install tree for python3 via new trees.txt mechanism

comment:18 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:19 Changed 8 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Status changed from new to needs_review

comment:20 Changed 8 months ago by git

  • Commit changed from eeb8777b0c31bbf15f0d97171c3eccce119cffac to c8e19a07dd6039be28f17e67e9fd2e693f14a52b

Branch pushed to git repo; I updated commit sha1. New commits:

c8e19a0fixup

comment:21 Changed 8 months ago by git

  • Commit changed from c8e19a07dd6039be28f17e67e9fd2e693f14a52b to 5a4424b3fa0c546bc05aef5320429749f1b143e9

Branch pushed to git repo; I updated commit sha1. New commits:

5a4424bbuild/make/Makefile.in (all-sage): Do not include packages installed in SAGE_DOCS

comment:22 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:23 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:24 Changed 8 months ago by jhpalmieri

I feel like trees.txt should be tree.txt, unless there is some way to specify more than one tree there. Obviously not a big deal. I would like this file to be documented, though, and maybe a comment in m4/sage_spkg_collect.m4 is the right place.

comment:25 Changed 8 months ago by mkoeppe

Yes, actually several trees are allowed to be specified there. We don't need this quite yet, but a separate environment for the Jupyter notebook could make sense; and then some packages such as jupyter_core would have to be installed in both SAGE_VENV and the new notebook environment.

comment:26 Changed 8 months ago by mkoeppe

I'll add a bit to the developer's guide

comment:27 Changed 8 months ago by git

  • Commit changed from 5a4424b3fa0c546bc05aef5320429749f1b143e9 to cb7874121095397431952b039fb10a03c538b734

Branch pushed to git repo; I updated commit sha1. New commits:

cb78741src/doc/en/developer/packaging.rst: Exlpain trees.txt

comment:28 Changed 8 months ago by jhpalmieri

It's not clear to me what ./configure --enable-sagemath_doc_pdf is supposed to do just by reading ./configure --help. If I run make, will it also run make doc-pdf, or does it just do some prep work for the PDF build? I see this:

  --enable-sagemath_doc_pdf={no|if_installed (default)|yes}
                          enable build and use of the optional package sagemath_doc_pdf
                          * package info: ./sage -info sagemath_doc_pdf

and then unfortunately ./sage -info sagemath_doc_pdf is not very useful.

comment:29 Changed 8 months ago by mkoeppe

I'll add SPKG.rst files to make the info more useful

comment:30 Changed 8 months ago by jhpalmieri

I think that sage_docbuild should be a dependency of each of the new packages: I just got this error (sagemath_doc_html-none.log):

/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/bin/python3: No module named sage_docbuild
make doc-inventory--
cd /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5 && ./sage --docbuild --no-pdf-links .o inventory 
/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/bin/python3: No module named sage_docbuild
make[6]: *** [doc-inventory--.o] Error 1
make[5]: *** [doc-inventory-reference] Error 2

comment:31 Changed 8 months ago by git

  • Commit changed from cb7874121095397431952b039fb10a03c538b734 to 7ad6cd9a460a42a0d8a5f4f1397539bbc87d9c89

Branch pushed to git repo; I updated commit sha1. New commits:

7ad6cd9build/pkgs/sagemath_doc_*/SPKG.rst: New

comment:32 Changed 8 months ago by git

  • Commit changed from 7ad6cd9a460a42a0d8a5f4f1397539bbc87d9c89 to eb65e034bad3ec82b19c90211ca5bf70533e8fe5

Branch pushed to git repo; I updated commit sha1. New commits:

eb65e03build/pkgs/sagemath_doc_*/dependencies: Add sage_docbuild

comment:33 Changed 8 months ago by git

  • Commit changed from eb65e034bad3ec82b19c90211ca5bf70533e8fe5 to 7989d8767056d68def98ee22116fbd24d7187d2f

Branch pushed to git repo; I updated commit sha1. New commits:

7989d87src/doc/Makefile: Handle errors from './sage --docbuild --all-documents'

comment:34 Changed 8 months ago by jhpalmieri

I'm getting an error with the pdf build, one that I don't get with the develop branch: en/reference/plotting is failing, and the file local/share/doc/sage/latex/en/reference/plotting/plotting.log ends with

Package pdftex.def Info: plot-23.pdf  used on input line 864.
(pdftex.def)             Requested size: 462.52684pt x 346.89516pt.
<plot-24.pdf, id=486, 462.52798pt x 346.89601pt>
File: plot-24.pdf Graphic file (type pdf)
<use plot-24.pdf>
Package pdftex.def Info: plot-24.pdf  used on input line 884.
(pdftex.def)             Requested size: 462.52684pt x 346.89516pt.

[23 <./plot-22.pdf>] [24 <./plot-23.pdf>] [25 <./plot-24.pdf
!pdfTeX error: pdflatex (file ./plot-24.pdf): PDF inclusion: type <cmd> cannot 
be copied
 ==> Fatal error occurred, no output PDF file produced!

Any ideas?

comment:35 Changed 8 months ago by mkoeppe

No idea, I haven't seen this failure

comment:36 Changed 8 months ago by git

  • Commit changed from 7989d8767056d68def98ee22116fbd24d7187d2f to 6afc0d5535530f9f71538821ec07a7cb95d44d19

Branch pushed to git repo; I updated commit sha1. New commits:

d64fb76build/make/Makefile.in: Undo reintroduction of SAGE_SKIP_PLOT_DIRECTIVE, SAGE_DOC_MATHJAX settings
6afc0d5build/pkgs/sagemath_doc_html/spkg-install: Move handling of SAGE_DOC_JSMATH environment variable here from src/bin/sage-env

comment:37 Changed 8 months ago by git

  • Commit changed from 6afc0d5535530f9f71538821ec07a7cb95d44d19 to b2cf40c2a9b023c2449dbf4186aeb369f604390a

Branch pushed to git repo; I updated commit sha1. New commits:

b2cf40cbuild/make/Makefile.in: Undo reintroduction of SAGE_SKIP_PLOT_DIRECTIVE, SAGE_DOC_MATHJAX settings (fixup)

comment:38 Changed 8 months ago by jhpalmieri

I'm also getting an error with the html build:

[developer] The HTML pages are in local/share/doc/sage/html/en/developer.
Error building the documentation.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1814, in main
    builder()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 133, in f
    runsphinx()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/sphinxbuild.py", line 323, in runsphinx
    sys.stderr.raise_errors()
  File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/sphinxbuild.py", line 258, in raise_errors
    raise OSError(self._error)
OSError: /Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta5/src/doc/en/developer/packaging.rst:721: WARNING: Duplicate explicit target name: "section-dependencies".

This is after make distclean && ./configure && make.

comment:39 Changed 8 months ago by mkoeppe

Thanks, I see this one now too, after make distclean

comment:40 Changed 8 months ago by git

  • Commit changed from b2cf40c2a9b023c2449dbf4186aeb369f604390a to 37a03bd94e1716af2926c30eddbaf6d38303f869

Branch pushed to git repo; I updated commit sha1. New commits:

37a03bdsrc/doc/en/developer/packaging.rst: Fix up section label

comment:41 Changed 8 months ago by git

  • Commit changed from 37a03bd94e1716af2926c30eddbaf6d38303f869 to fc89a6657450c0e10c8322615f1fced0f8ebb984

Branch pushed to git repo; I updated commit sha1. New commits:

fc89a66build/pkgs/sagemath_doc_pdf/spkg-install: Set SAGE_DOC_MATHJAX as a workaround

comment:42 Changed 8 months ago by mkoeppe

PDF builds for me without error

comment:43 Changed 8 months ago by jhpalmieri

I think that sagemath_doc_html should be a dependency for sagemath_doc_pdf. Otherwise both may try to build at the same time, and they may both try to build the inventory files at the same time, which could easily lead to race conditions.

comment:44 Changed 8 months ago by mkoeppe

Yes, I agree

comment:45 Changed 8 months ago by git

  • Commit changed from fc89a6657450c0e10c8322615f1fced0f8ebb984 to fd298c4b7a6f845335b862f6dc2f8e18753937e7

Branch pushed to git repo; I updated commit sha1. New commits:

fd298c4build/pkgs/sagemath_doc_pdf/dependencies: Depend on sagemath_doc_html

comment:46 Changed 8 months ago by jhpalmieri

Everything now builds for me.

comment:47 Changed 8 months ago by mkoeppe

Thanks for testing!

comment:48 Changed 8 months ago by jhpalmieri

  • Reviewers set to John Palmieri

I think this is ready to go. Merge now or wait until 9.6? I can imagine it breaking something for unforeseen reasons.

comment:49 Changed 8 months ago by mkoeppe

I think we can merge it now

comment:50 Changed 8 months ago by jhpalmieri

  • Status changed from needs_review to positive_review

comment:51 Changed 8 months ago by mkoeppe

Thank you!

comment:52 Changed 8 months ago by vbraun

  • Branch changed from u/mkoeppe/restructure_docbuild_as_optional_script_packages_sagemath_doc_html__sagemath_doc_pdf to fd298c4b7a6f845335b862f6dc2f8e18753937e7
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.