Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#28856 closed enhancement (fixed)

Update to sphinx 3

Reported by: Timo Kaufmann Owned by:
Priority: major Milestone: sage-9.2
Component: packages: standard Keywords: sphinx
Cc: Antonio Rojas, François Bissey, Julian Rüth, fchapoton, John Palmieri, Jeroen Demeyer, Mitesh Patel, Tobias Hansen, Steven Trogdon Merged in:
Authors: Timo Kaufmann, John Palmieri, François Bissey, Antonio Rojas Reviewers: Timo Kaufmann, John Palmieri, François Bissey, Antonio Rojas
Report Upstream: N/A Work issues:
Branch: bd99462 (Commits, GitHub, GitLab) Commit:
Dependencies: #28000, #29547 Stopgaps:

Status badges

Description (last modified by François Bissey)

Migrate to sphinx 3.

The attributes todo_all_todos and citations were previously implicitly initialized by sphinx. That changed in sphinx 2.1 due to some refactoring:

https://github.com/sphinx-doc/sphinx/commit/9abb4820b18201f63d77fdccb4ef394a21442f7a https://github.com/sphinx-doc/sphinx/commit/885d35e374b56d1d17f5036fe07b74229cdbec7f

So now we need to check for the case where they are not initialized yet. We can't actually update to sphinx 2.1 yet, since it's python3 only. This is mostly intended to make things easier for distros and allow us to update in the future.

Another issue occurs with intersphinx. Since a refactoring (https://github.com/sphinx-doc/sphinx/pull/5826) the build fails with

[manifolds] Exception occurred:
[manifolds]   File "/nix/store/ldl3rb92yvl070c7q3nnjgwl5mdx2wvb-python3.7-sphinx-2.0.0/lib/python3.7/site-packages/sphinx/ext/intersphinx.py", line 207, in load_mappings
[manifolds]     for key, (name, (uri, invs)) in app.config.intersphinx_mapping.items():
[manifolds] ValueError: too many values to unpack (expected 2)
[manifolds] The full traceback has been saved in /build/sphinx-err-e6h8o39l.log, if you want to report the issue to the developers.
[manifolds] Please also report this if it was a user error, so that a better error message can be provided next time.
[manifolds] A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Traceback (most recent call last):
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/sphinxbuild.py", line 328, in <module>
    runsphinx()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/sphinxbuild.py", line 317, in runsphinx
    sys.stderr.raise_errors()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/sphinxbuild.py", line 252, in raise_errors
    raise OSError(self._error)
OSError: Exception occurred:
Error building the documentation.
Traceback (most recent call last):
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
    main()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 1671, in main
    builder()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 310, in _wrapper
    getattr(get_builder(document), name)(*args, **kwds)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 504, in _wrapper
    build_many(build_ref_doc, L)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 258, in build_many
    _build_many(target, args, processes=NUM_THREADS)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 283, in build_many
    raise worker_exc.original_exception

which seems to suggest that the normalization hook added in the refactoring PR is not called.

Tarballs:

Change History (150)

comment:1 Changed 3 years ago by Timo Kaufmann

Branch: u/gh-timokau/sphinx-2.1-support
Commit: af695b21426f49cc6d35c116dcc413a8d5648d25
Status: newneeds_review

New commits:

af695b2Make docbuild merging compatible with spinx >=2.1

comment:2 Changed 3 years ago by Timo Kaufmann

Summary: Support sphinx >= 2.1Make docbuild merging compatible with spinx >=2.1

comment:3 Changed 3 years ago by Timo Kaufmann

Summary: Make docbuild merging compatible with spinx >=2.1Make docbuild merging compatible with spinx 2.1

comment:4 Changed 3 years ago by git

Commit: af695b21426f49cc6d35c116dcc413a8d5648d25a3382d8fdab55abe383d71252f3b551d4f37245f

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a3382d8Make docbuild merging compatible with spinx 2.1

comment:5 Changed 3 years ago by git

Commit: a3382d8fdab55abe383d71252f3b551d4f37245f1c3099488e4d11999fe1c3337bab21553e52da6e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

1c30994Make docbuild merging compatible with spinx 2.1

comment:6 Changed 3 years ago by git

Commit: 1c3099488e4d11999fe1c3337bab21553e52da6e4c3780c2368b9a840535a9bbb3c593b8cb709c2b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4c3780cMake docbuild merging compatible with spinx 2.1

comment:7 Changed 3 years ago by git

Commit: 4c3780c2368b9a840535a9bbb3c593b8cb709c2b447b6b071243f410df701679376c908394f759fd

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

447b6b0Make docbuild merging compatible with spinx 2.1

comment:8 Changed 3 years ago by Timo Kaufmann

Status: needs_reviewneeds_work

Remaining problem:

[manifolds] Exception occurred:
[manifolds]   File "/nix/store/ldl3rb92yvl070c7q3nnjgwl5mdx2wvb-python3.7-sphinx-2.0.0/lib/python3.7/site-packages/sphinx/ext/intersphinx.py", line 207, in load_mappings
[manifolds]     for key, (name, (uri, invs)) in app.config.intersphinx_mapping.items():
[manifolds] ValueError: too many values to unpack (expected 2)
[manifolds] The full traceback has been saved in /build/sphinx-err-erras5d4.log, if you want to report the issue to the developers.
[manifolds] Please also report this if it was a user error, so that a better error message can be provided next time.
[manifolds] A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Traceback (most recent call last):
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/sphinxbuild.py", line 328, in <module>
    runsphinx()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/sphinxbuild.py", line 317, in runsphinx
    sys.stderr.raise_errors()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/sphinxbuild.py", line 252, in raise_errors
    raise OSError(self._error)
OSError: Exception occurred:
Error building the documentation.
Traceback (most recent call last):
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/9894fxjpg8v99j14kip6b13rdfgf4m5k-python3-3.7.5/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
    main()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 1671, in main
    builder()
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 310, in _wrapper
    getattr(get_builder(document), name)(*args, **kwds)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 504, in _wrapper
    build_many(build_ref_doc, L)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 258, in build_many
    _build_many(target, args, processes=NUM_THREADS)
  File "/nix/store/i0gjbmskaxkmj2qif96ayvb803l6hm4k-python3.7-sagelib-9.0.beta6/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 283, in build_many
    raise worker_exc.original_exception
subprocess.CalledProcessError: Command '['python', '-um', 'sage_setup.docbuild.sphinxbuild', '-N', '-b', 'html', '-d', '/build/share/doc/sage/doctrees/en/reference/manifolds', '-D', 'multidoc_first_pass=0', '/bui>
builder for '/nix/store/rbwjmqg1dp0padhni45xkhj4vqmxc4x9-sagedoc-9.0.beta6.drv' failed with exit code 1

Related to https://github.com/sphinx-doc/sphinx/pull/5826 and the way we use intersphinx.

comment:9 Changed 3 years ago by Samuel Lelièvre

Description: modified (diff)
Keywords: sphinx added
Summary: Make docbuild merging compatible with spinx 2.1Make docbuild merging compatible with sphinx 2.1

comment:10 Changed 3 years ago by Timo Kaufmann

Description: modified (diff)
Summary: Make docbuild merging compatible with sphinx 2.1Make sage compatible with sphinx 2.1

comment:11 Changed 3 years ago by Timo Kaufmann

Cc: fchapoton John Palmieri Jeroen Demeyer Mitesh Patel added

Since I have little experience with sphinx and don't know how and why our intersphinx usage differs from normal usage, I'm having issues fixing this issue.

I've CC'ed some people who have touched the sphinx config in the past. Can anybody help with the second issue described in the top post (regarding intersphinx)?

comment:12 Changed 3 years ago by Timo Kaufmann

Does anyone have experience with sphinx hooks? Understanding why the hook introduced in https://github.com/sphinx-doc/sphinx/pull/5826 isn't called would help a lot.

comment:13 Changed 3 years ago by Markus Wageringel

I do not have much experience with sphinx, but, at the bottom of src/sage/docs/conf.py, it appears that intersphinx is not loaded the usual way by adding it to the extensions list, but instead it is manually initialized in order to inject a custom function for missing_reference.

        app.add_config_value('intersphinx_mapping', {}, False)
        app.add_config_value('intersphinx_cache_limit', 5, False)
        # We do *not* fully initialize intersphinx since we call it by hand
        # in find_sage_dangling_links.
        #   app.connect('missing-reference', missing_reference)
        app.connect('missing-reference', find_sage_dangling_links)
        import sphinx.ext.intersphinx
        app.connect('builder-inited', set_intersphinx_mappings)
        app.connect('builder-inited', sphinx.ext.intersphinx.load_mappings)

Probably this means that normalize_intersphinx_mapping is never called, but adjusting the code above as in the PR might fix it.

comment:14 Changed 3 years ago by Erik Bray

Milestone: sage-9.0sage-9.1

Ticket retargeted after milestone closed

comment:15 Changed 3 years ago by Timo Kaufmann

For what it's worth, I have not been able to get this working and don't have time to work on it any further right now. My current attempt (trying to get the normalization in there at the right point):

diff --git a/src/sage/docs/conf.py b/src/sage/docs/conf.py
index d86fc9c6d1..013f1794e4 100644
--- a/src/sage/docs/conf.py
+++ b/src/sage/docs/conf.py
@@ -8,6 +8,7 @@ from docutils import nodes
 from docutils.transforms import Transform
 from sphinx.ext.doctest import blankline_re
 from sphinx import highlighting
+import sphinx.ext.intersphinx as intersphinx
 from IPython.lib.lexers import IPythonConsoleLexer, IPyLexer
 
 # If your extensions are in another directory, add it here.
@@ -175,7 +176,7 @@ intersphinx_mapping = {
                              "python{}.inv".format(python_version))),
     'pplpy': (PPLPY_DOCS, None)}
 
-def set_intersphinx_mappings(app, _config):
+def set_intersphinx_mappings(app, config):
     """
     Add precompiled inventory (the objects.inv)
     """
@@ -201,6 +202,8 @@ def set_intersphinx_mappings(app, _config):
             dst = os.path.join(invpath, directory, 'objects.inv')
             app.config.intersphinx_mapping[src] = dst
 
+    intersphinx.normalize_intersphinx_mapping(app, config)
+
 
 # By default document are not master.
 multidocs_is_master = True
@@ -669,7 +672,7 @@ def call_intersphinx(app, env, node, contnode):
     """
     debug_inf(app, "???? Trying intersphinx for %s" % node['reftarget'])
     builder = app.builder
-    res =  sphinx.ext.intersphinx.missing_reference(
+    res =  intersphinx.missing_reference(
         app, env, node, contnode)
     if res:
         # Replace absolute links to $SAGE_DOC by relative links: this
@@ -852,12 +855,11 @@ def setup(app):
     if app.srcdir.startswith(SAGE_DOC_SRC):
         app.add_config_value('intersphinx_mapping', {}, False)
         app.add_config_value('intersphinx_cache_limit', 5, False)
+        app.connect('config-inited', set_intersphinx_mappings)
+        # app.connect('config-inited', intersphinx.normalize_intersphinx_mapping)
+        app.connect('builder-inited', intersphinx.load_mappings)
         # We do *not* fully initialize intersphinx since we call it by hand
         # in find_sage_dangling_links.
         #   app.connect('missing-reference', missing_reference)
         app.connect('missing-reference', find_sage_dangling_links)
-        import sphinx.ext.intersphinx
-        app.connect('config-inited', set_intersphinx_mappings)
-        app.connect('config-inited', sphinx.ext.intersphinx.normalize_intersphinx_mapping)
-        app.connect('builder-inited', sphinx.ext.intersphinx.load_mappings)
         app.connect('builder-inited', nitpick_patch_config)

comment:16 Changed 3 years ago by John Palmieri

I have been attempting to initialize intersphinx as an ordinary Sphinx extension. The reference manual builds this way, but other pieces of the documentation fail because of missing citations, and I don't know why. But succeeding with the reference manual seems like progress. Anyway, here is my current attempt; this is using the version of Sphinx which is included in Sage — I thought I would try get that to work first with this approach, and then move on to a more recent version of Sphinx.

  • src/sage/docs/conf.py

    diff --git a/src/sage/docs/conf.py b/src/sage/docs/conf.py
    index 23cf5d6792..226e68a31b 100644
    a b extensions = ['inventory_builder', 
    2525              'sphinx.ext.inheritance_diagram',
    2626              'sphinx.ext.todo',
    2727              'sphinx.ext.extlinks',
     28              'sphinx.ext.intersphinx',
    2829              'IPython.sphinxext.ipython_directive',
    2930              'matplotlib.sphinxext.plot_directive']
    3031
    from sage.all_cmdline import * 
    8889plot_html_show_formats = False
    8990plot_formats = ['svg', 'pdf', 'png']
    9091
    91 # We do *not* fully initialize intersphinx since we call it by hand
    92 # in find_sage_dangling_links.
    93 #, 'sphinx.ext.intersphinx']
    94 
    95 
    9692# Add any paths that contain templates here, relative to this directory.
    9793templates_path = [os.path.join(SAGE_DOC_SRC, 'common', 'templates'), 'templates']
    9894
    def setup(app): 
    850846    # set to a temporary directory.  We don't want to use intersphinx,
    851847    # etc., when doing introspection.
    852848    if app.srcdir.startswith(SAGE_DOC_SRC):
    853         app.add_config_value('intersphinx_mapping', {}, False)
    854         app.add_config_value('intersphinx_cache_limit', 5, False)
    855         # We do *not* fully initialize intersphinx since we call it by hand
    856         # in find_sage_dangling_links.
    857849        #   app.connect('missing-reference', missing_reference)
    858850        app.connect('missing-reference', find_sage_dangling_links)
    859851        import sphinx.ext.intersphinx

comment:17 Changed 3 years ago by Timo Kaufmann

Sounds like good progress indeed! Throwing away our hacks and becoming a "normal" intersphinx consumer would be the best solution.

comment:18 Changed 3 years ago by Antonio Rojas

Opened #29095 for the runtime issues with latest sphinx (which are much easier to fix)

comment:19 Changed 3 years ago by Timo Kaufmann

Just as a heads-up: The combination of an outdated sphinx version and a newer (>=0.15) docutils version leads to (harmless) doctest failures:

File "/nix/store/kyzn2mbdpjgdh1n8ldqwah3s1k4b6mza-sage-src-8.9/src/sage/interfaces/tachyon.py", line 191, in sage.interfaces.tachyon.TachyonRT.help
Failed example:
    t.help(use_pager=False)
Expected:
    This help, which was written by John Stone, describes ...
Got:
    doctest:warning
      File "/nix/store/kra72yycg2ixhkmx944qgrlgaqhr6ia4-sage-with-env-8.9/bin/sage-runtests", line 179, in <module>
        err = DC.run()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/control.py", line 1227, in run
        self.run_doctests()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/control.py", line 928, in run_doctests
        self.dispatcher.dispatch()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 2041, in dispatch
        self.parallel_dispatch()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 1933, in parallel_dispatch
        w.start()  # This might take some time
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 2208, in start
        super(DocTestWorker, self).start()
      File "/nix/store/k5iyhnvy9cdlq3mpavw6s8blyr0px522-python-2.7.17/lib/python2.7/multiprocessing/process.py", line 130, in start
        self._popen = Popen(self)
      File "/nix/store/k5iyhnvy9cdlq3mpavw6s8blyr0px522-python-2.7.17/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
        code = process_obj._bootstrap()
      File "/nix/store/k5iyhnvy9cdlq3mpavw6s8blyr0px522-python-2.7.17/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
        self.run()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 2180, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 2512, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 2561, in _run
        result = runner.run(test)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 897, in run
        return self._run(test, compileflags, out)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 1131, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.tachyon.TachyonRT.help[2]>", line 1, in <module>
        t.help(use_pager=False)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/interfaces/tachyon.py", line 784, in help
        f = format(s)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 720, in format
        s = detex(s, embedded=embedded)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 228, in detex
        s = sphinxify(s, format='text')
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sage/misc/sphinxify.py", line 123, in sphinxify
        sphinx_app.build(None, [rst_name])
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/application.py", line 338, in build
        self.builder.build_specific(filenames)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 335, in build_specific
        summary=__('%d source files given on command line') % len(to_write))
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 360, in build
        updated_docnames = set(self.read())
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 468, in read
        self._read_serial(docnames)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 490, in _read_serial
        self.read_doc(docname)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 534, in read_doc
        doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/io.py", line 318, in read_doc
        pub.publish()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/docutils/core.py", line 219, in publish
        self.apply_transforms()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/docutils/core.py", line 200, in apply_transforms
        self.document.transformer.apply_transforms()
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/transforms/__init__.py", line 90, in apply_transforms
        Transformer.apply_transforms(self)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/docutils/transforms/__init__.py", line 171, in apply_transforms
        transform.apply(**kwargs)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/transforms/__init__.py", line 245, in apply
        apply_source_workaround(n)
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/sphinx/util/nodes.py", line 94, in apply_source_workaround
        for classifier in reversed(node.parent.traverse(nodes.classifier)):
      File "/nix/store/3j74nni97qk6pj1slbbgn67q5qw4r51s-python-2.7.17-env/lib/python2.7/site-packages/docutils/nodes.py", line 46, in wrapper
        warnings.warn(msg, FutureWarning, stacklevel=2)
    :
    FutureWarning: 
       The iterable returned by Node.traverse()
       will become an iterator instead of a list in Docutils > 0.16.
    This help, which was written by John Stone, describes how to create
    scene files.
    <BLANKLINE>
    At the present time, scene description files are very simple. The
    parser can't handle multiple file scene descriptions, although they
    may be added in the future.  Most of the objects and their scene
    description are closely related to the RAY API *(See the API docs for
    additional info.)*

These will likely turn into errors for docutils 0.16, but are already fixed in sphinx upstream: https://github.com/sphinx-doc/sphinx/commit/faedcc48ccb942b9a7b758b699b30f0d026c0771

comment:20 Changed 3 years ago by John Palmieri

In the meantime, a beta for Sphinx 3.0.0 is available: see https://github.com/sphinx-doc/sphinx/blob/3.0.x/CHANGES

comment:21 Changed 3 years ago by François Bissey

A user of sage-on-gentoo did go and talk to upstream about our difficulties with intersphinx in https://github.com/sphinx-doc/sphinx/issues/7409 and one of the dev took it to heart. Apparently we have to switch from builder-inetd to config-inetd and make sure we have sphinx including https://github.com/sphinx-doc/sphinx/pull/7415

I haven't got around figuring out how to use all that information.

comment:22 Changed 3 years ago by François Bissey

And now that I have looked more closely, it is all included in the newly released sphinx-3.0.

comment:23 Changed 3 years ago by François Bissey

I am trying to get to support sphinx 3.0 in gentoo in time for sage-9.1. So far I have combined the branch attached to this ticket plus comment:15.

  • sage/docs/conf.py

    a b from docutils import nodes 
    88from docutils.transforms import Transform
    99from sphinx.ext.doctest import blankline_re
    1010from sphinx import highlighting
     11import sphinx.ext.intersphinx as intersphinx
    1112from IPython.lib.lexers import IPythonConsoleLexer, IPyLexer
    1213
    1314# If your extensions are in another directory, add it here.
    intersphinx_mapping = { 
    175176                             "python{}.inv".format(python_version))),
    176177    'pplpy': (PPLPY_DOCS, None)}
    177178
    178 def set_intersphinx_mappings(app):
     179def set_intersphinx_mappings(app, config):
    179180    """
    180181    Add precompiled inventory (the objects.inv)
    181182    """
    def set_intersphinx_mappings(app): 
    201202            dst = os.path.join(invpath, directory, 'objects.inv')
    202203            app.config.intersphinx_mapping[src] = dst
    203204
     205    intersphinx.normalize_intersphinx_mapping(app, config)
     206
    204207
    205208# By default document are not master.
    206209multidocs_is_master = True
    def call_intersphinx(app, env, node, contnode): 
    669672    """
    670673    debug_inf(app, "???? Trying intersphinx for %s" % node['reftarget'])
    671674    builder = app.builder
    672     res =  sphinx.ext.intersphinx.missing_reference(
     675    res =  intersphinx.missing_reference(
    673676        app, env, node, contnode)
    674677    if res:
    675678        # Replace absolute links to $SAGE_DOC by relative links: this
    def setup(app): 
    852855    if app.srcdir.startswith(SAGE_DOC_SRC):
    853856        app.add_config_value('intersphinx_mapping', {}, False)
    854857        app.add_config_value('intersphinx_cache_limit', 5, False)
     858        app.connect('config-inited', set_intersphinx_mappings)
     859        # app.connect('config-inited', intersphinx.normalize_intersphinx_mapping)
     860        app.connect('builder-inited', intersphinx.load_mappings)
    855861        # We do *not* fully initialize intersphinx since we call it by hand
    856862        # in find_sage_dangling_links.
    857863        #   app.connect('missing-reference', missing_reference)
    858864        app.connect('missing-reference', find_sage_dangling_links)
    859         import sphinx.ext.intersphinx
    860         app.connect('builder-inited', set_intersphinx_mappings)
    861         app.connect('builder-inited', sphinx.ext.intersphinx.load_mappings)
    862865        app.connect('builder-inited', nitpick_patch_config)
  • sage_setup/docbuild/ext/multidocs.py

    diff --git a/sage_setup/docbuild/ext/multidocs.py b/sage_setup/docbuild/ext/multidocs.py
    index 71a08cd..3a3d09a 100644
    a b def merge_environment(app, env): 
    4747    - domaindata['py']['modules'] # list of python modules
    4848    """
    4949    logger.info(bold('Merging environment/index files...'))
     50    if not hasattr(env, "todo_all_todos"):
     51        env.todo_all_todos = []
     52    if not hasattr(env.domaindata["std"], "citations"):
     53        env.domaindata["std"]["citations"] = dict()
    5054    for curdoc in app.env.config.multidocs_subdoc_list:
    5155        logger.info("    %s:"%curdoc, nonl=1)
    5256        docenv = get_env(app, curdoc)
    5357        if docenv is not None:
    5458            fixpath = lambda path: os.path.join(curdoc, path)
     59            todos = docenv.todo_all_todos if hasattr(docenv, "todo_all_todos") else []
     60            citations = docenv.domaindata["std"].get("citations", dict())
    5561            logger.info(" %s todos, %s index, %s citations"%(
    56                     len(docenv.todo_all_todos),
     62                    len(todos),
    5763                    len(docenv.indexentries),
    58                     len(docenv.domaindata["std"]["citations"])
     64                    len(citations)
    5965                    ), nonl=1)
    6066
    6167            # merge titles
    6268            for t in docenv.titles:
    6369                env.titles[fixpath(t)] = docenv.titles[t]
    6470            # merge the todo links
    65             for dct in docenv.todo_all_todos:
     71            for dct in todos:
    6672                dct['docname'] = fixpath(dct['docname'])
    67             env.todo_all_todos += docenv.todo_all_todos
     73            env.todo_all_todos += todos
    6874            # merge the html index links
    6975            newindex = {}
    7076            for ind in docenv.indexentries:
    def merge_environment(app, env): 
    8692                env.metadata[ind] = md
    8793            # merge the citations
    8894            newcite = {}
    89             citations = docenv.domaindata["std"]["citations"]
    90             for ind, (path, tag, lineno) in six.iteritems(docenv.domaindata["std"]["citations"]):
     95            for ind, (path, tag, lineno) in six.iteritems(citations):
    9196                # TODO: Warn on conflicts
    9297                newcite[ind] = (fixpath(path), tag, lineno)
    9398            env.domaindata["std"]["citations"].update(newcite)
    def fetch_citation(app, env): 
    253258    with open(filename, 'rb') as f:
    254259        cache = cPickle.load(f)
    255260    logger.info("done (%s citations)."%len(cache))
    256     cite = env.domaindata["std"]["citations"]
     261    cite = env.domaindata["std"].get("citations", dict())
    257262    for ind, (path, tag, lineno) in six.iteritems(cache):
    258263        if ind not in cite: # don't override local citation
    259264            cite[ind] = (os.path.join("..", path), tag, lineno)

And it goes some way with multiple warnings

[plot3d   ] /usr/lib/python3.7/site-packages/sphinx/pycode/__init__.py:186: RemovedInSphinx40Warning: ModuleAnalyzer.encoding is deprecated.
[plot3d   ]   RemovedInSphinx40Warning)

which is probably coming from something else (like docutils - I have to track it). But ultimately it breaks down at

[combinat ] /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/build/lib/sage/combinat/permutation.py:docstring of sage.combinat.permutation.Permutation.left_action_product:1: WARNING: duplicate object description of sage.combinat.permutation.Permutation.left_action_product, other instance in sage/combinat/permutation, use :noindex: for one of them
[combinat ] /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/build/lib/sage/combinat/permutation.py:docstring of sage.combinat.permutation.Permutation.right_action_product:1: WARNING: duplicate object description of sage.combinat.permutation.Permutation.right_action_product, other instance in sage/combinat/permutation, use :noindex: for one of them

which leads to the build of documentation stopping

Error building the documentation.
Traceback (most recent call last):
  File "sage_setup/docbuild/__main__.py", line 2, in <module>
    main()
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 1720, in main
    builder()
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 327, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 552, in _wrapper
    self._build_everything_except_bibliography(lang, format, *args, **kwds)
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 538, in _build_everything_except_bibliography
    build_many(build_ref_doc, non_references)
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 280, in build_many
    _build_many(target, args, processes=NUM_THREADS)
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/utils.py", line 283, in build_many
    raise worker_exc.original_exception
OSError: /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/build/lib/`sage/combinat/permutation.py:docstring of sage.combinat.permutation.Permutation.left_action_product:1: WARNING: duplicate object description of sage.combinat.permutation.Permutation.left_action_product, other instance in sage/combinat/permutation, use :noindex: for one of them

I have looked at sage/combinat/permutation.py but I cannot figure where I should add :noindex:. Some assistance at this stage would be welcome.

comment:24 Changed 3 years ago by Antonio Rojas

The deprecation warnings come from sage itself, specifically from https://git.sagemath.org/sage.git/tree/src/sage_setup/docbuild/ext/sage_autodoc.py#n539 - fixing it gives a similar amount of warnings, this time about safe_getmembers. Haven't looked into the replacement for that one yet.

Building with -k makes it go past the duplicate object error, but then one gets again the ValueError: too many values to unpack error when it reaches the second pass.

comment:25 Changed 3 years ago by François Bissey

I see the safe_getmembers warnings as well - without fixing sage_autodoc. It floods a little less :) I thought sphinx-3.0 was supposed to help with the "too many value to unpack". I guess that was too optimistic.

comment:26 Changed 3 years ago by Antonio Rojas

I added some debug, seems that normalize_intersphinx_mapping is never being run

comment:27 Changed 3 years ago by Antonio Rojas

Alright, turns out that was my fault... conf.py was moved to sagelib a while ago so I was using the (unpatched) version from the installed sagemath instead of the patched one in the source code when building sagemath-doc.

Now, after sorting it out, the new errors (on the second pass) are:

[tutorial ]   File "/usr/lib/python3.8/urllib/parse.py", line 108, in <genexpr>
[tutorial ]     return tuple(x.decode(encoding, errors) if x else '' for x in args)
[tutorial ] AttributeError: 'tuple' object has no attribute 'decode'

These seem to be cause by a bad intersphinx mapping normalization... looks like normalize_intersphinx_mapping is not idempotent, and when running it on an already normalized item it will try to renormalize it and break its format.

comment:28 in reply to:  23 Changed 3 years ago by Antonio Rojas

Getting closer: with the attached patch the double normalization issue is worked around and the compilation finishes (with -k). It also fixes the most annoying deprecation warnings. However, there are many files missing which were present when compiling with sphinx 1.

Not sure if it's related, but at the end of the first pass one gets the error:

[reference] Extension error:
[reference] Domain 'index' is not registered

so it seems that the index attribute also needs to be explicitely initialized now, similar to citations and todo_all_todos

obsolete patch removed

Last edited 3 years ago by Antonio Rojas (previous) (diff)

comment:29 Changed 3 years ago by Antonio Rojas

Updated patch that fixes the Domain 'index' is not registered error and all sphinx deprecation warnings. Still, html for the reference manual subdirs is not being generated.

obsolete patch removed

Last edited 3 years ago by Antonio Rojas (previous) (diff)

comment:30 Changed 3 years ago by François Bissey

Well, that's some progress I guess. I think we should move this ticket milestone to 9.2 and make it "upgrade to sphinx 3", that would be more honest because I don't think we can keep it compatible with sphinx-1.8 at the same time.

comment:31 Changed 3 years ago by Tobias Hansen

Cc: Tobias Hansen added

comment:32 Changed 3 years ago by Antonio Rojas

Found the problem, some used sphinx API has been removed. With the attached version the docs compile (still with -k). Next issue: citation links are not working.

  • src/sage_setup/docbuild/__init__.py

    a b class ReferenceSubBuilder(DocBuilder): 
    817817
    818818        env_pickle = os.path.join(self._doctrees_dir(), 'environment.pickle')
    819819        try:
    820             env = BuildEnvironment.frompickle(env_pickle, FakeApp(self.dir))
    821             logger.debug("Opened Sphinx environment: %s", env_pickle)
    822             return env
     820            with open(env_pickle, 'rb') as f:
     821                import pickle
     822                env = pickle.load(f)
     823                env.app = FakeApp(self.dir)
     824                env.config.values = env.app.config.values
     825                logger.debug("Opened Sphinx environment: %s", env_pickle)
     826                return env
    823827        except IOError as err:
    824828            logger.debug("Failed to open Sphinx environment: %s", err)
    825829
  • src/sage/docs/conf.py

    a b from docutils import nodes 
    88from docutils.transforms import Transform
    99from sphinx.ext.doctest import blankline_re
    1010from sphinx import highlighting
     11import sphinx.ext.intersphinx as intersphinx
    1112from IPython.lib.lexers import IPythonConsoleLexer, IPyLexer
    1213
    1314# If your extensions are in another directory, add it here.
    todo_include_todos = True 
    169169
    170170# Cross-links to other project's online documentation.
    171171python_version = sys.version_info.major
    172 intersphinx_mapping = {
    173     'python': ('https://docs.python.org/',
    174                 os.path.join(SAGE_DOC_SRC, "common",
    175                              "python{}.inv".format(python_version))),
    176     'pplpy': (PPLPY_DOCS, None)}
    177172
    178 def set_intersphinx_mappings(app):
     173def set_intersphinx_mappings(app, config):
    179174    """
    180175    Add precompiled inventory (the objects.inv)
    181176    """
    def set_intersphinx_mappings(app): 
    186182        app.config.intersphinx_mapping = {}
    187183        return
    188184
    189     app.config.intersphinx_mapping = intersphinx_mapping
     185    app.config.intersphinx_mapping =  {
     186    'python': ('https://docs.python.org/',
     187                os.path.join(SAGE_DOC_SRC, "common",
     188                             "python{}.inv".format(python_version))),
     189    'pplpy': (PPLPY_DOCS, None)}
    190190
    191191    # Add master intersphinx mapping
    192192    dst = os.path.join(invpath, 'objects.inv')
    def set_intersphinx_mappings(app): 
    201201            dst = os.path.join(invpath, directory, 'objects.inv')
    202202            app.config.intersphinx_mapping[src] = dst
    203203
     204    intersphinx.normalize_intersphinx_mapping(app, config)
    204205
    205206# By default document are not master.
    206207multidocs_is_master = True
    def call_intersphinx(app, env, node, contnode): 
    669672    """
    670673    debug_inf(app, "???? Trying intersphinx for %s" % node['reftarget'])
    671674    builder = app.builder
    672     res =  sphinx.ext.intersphinx.missing_reference(
     675    res =  intersphinx.missing_reference(
    673676        app, env, node, contnode)
    674677    if res:
    675678        # Replace absolute links to $SAGE_DOC by relative links: this
    def setup(app): 
    852855    if app.srcdir.startswith(SAGE_DOC_SRC):
    853856        app.add_config_value('intersphinx_mapping', {}, False)
    854857        app.add_config_value('intersphinx_cache_limit', 5, False)
     858        app.connect('config-inited', set_intersphinx_mappings)
     859        app.connect('builder-inited', intersphinx.load_mappings)
    855860        # We do *not* fully initialize intersphinx since we call it by hand
    856861        # in find_sage_dangling_links.
    857862        #   app.connect('missing-reference', missing_reference)
    858863        app.connect('missing-reference', find_sage_dangling_links)
    859         import sphinx.ext.intersphinx
    860         app.connect('builder-inited', set_intersphinx_mappings)
    861         app.connect('builder-inited', sphinx.ext.intersphinx.load_mappings)
    862864        app.connect('builder-inited', nitpick_patch_config)
  • src/sage_setup/docbuild/ext/multidocs.py

    a b def merge_environment(app, env): 
    4747    - domaindata['py']['modules'] # list of python modules
    4848    """
    4949    logger.info(bold('Merging environment/index files...'))
     50    if not hasattr(env, "todo_all_todos"):
     51        env.todo_all_todos = []
     52    if not env.domaindata["std"].get("citations"):
     53        env.domaindata["std"]["citations"] = dict()
    5054    for curdoc in app.env.config.multidocs_subdoc_list:
    5155        logger.info("    %s:"%curdoc, nonl=1)
    5256        docenv = get_env(app, curdoc)
    5357        if docenv is not None:
    5458            fixpath = lambda path: os.path.join(curdoc, path)
     59            todos = docenv.todo_all_todos if hasattr(docenv, "todo_all_todos") else []
     60            citations = docenv.domaindata["std"].get("citations", dict())
     61            indexentries = docenv.domaindata["index"].get("entries", dict())
    5562            logger.info(" %s todos, %s index, %s citations"%(
    56                     len(docenv.todo_all_todos),
    57                     len(docenv.indexentries),
    58                     len(docenv.domaindata["std"]["citations"])
     63                    len(todos),
     64                    len(indexentries),
     65                    len(citations)
    5966                    ), nonl=1)
    6067
    6168            # merge titles
    6269            for t in docenv.titles:
    6370                env.titles[fixpath(t)] = docenv.titles[t]
    6471            # merge the todo links
    65             for dct in docenv.todo_all_todos:
     72            for dct in todos:
    6673                dct['docname'] = fixpath(dct['docname'])
    67             env.todo_all_todos += docenv.todo_all_todos
     74            env.todo_all_todos += todos
    6875            # merge the html index links
    6976            newindex = {}
    70             for ind in docenv.indexentries:
     77            for ind in indexentries:
    7178                if ind.startswith('sage/'):
    72                     newindex[fixpath(ind)] = docenv.indexentries[ind]
     79                    newindex[fixpath(ind)] = indexentries[ind]
    7380                else:
    74                     newindex[ind] = docenv.indexentries[ind]
    75             env.indexentries.update(newindex)
     81                    newindex[ind] = indexentries[ind]
     82            env.domaindata['index']['entries'].update(newindex)
    7683            # merge the all_docs links, needed by the js index
    7784            newalldoc = {}
    7885            for ind in docenv.all_docs:
    def merge_environment(app, env): 
    8693                env.metadata[ind] = md
    8794            # merge the citations
    8895            newcite = {}
    89             citations = docenv.domaindata["std"]["citations"]
    90             for ind, (path, tag, lineno) in six.iteritems(docenv.domaindata["std"]["citations"]):
     96            for ind, (path, tag, lineno) in six.iteritems(citations):
    9197                # TODO: Warn on conflicts
    9298                newcite[ind] = (fixpath(path), tag, lineno)
    93             env.domaindata["std"]["citations"].update(newcite)
     99            env.domaindata['std']['citations'].update(newcite)
    94100            # merge the py:module indexes
    95101            newmodules = {}
    96             for ind,(modpath,v1,v2,v3) in (
     102            for ind,(modpath,v1,v2,v3,v4) in (
    97103                six.iteritems(docenv.domaindata['py']['modules'])):
    98                 newmodules[ind] = (fixpath(modpath),v1,v2,v3)
     104                newmodules[ind] = (fixpath(modpath),v1,v2,v3,v4)
    99105            env.domaindata['py']['modules'].update(newmodules)
    100106            logger.info(", %s modules"%(len(newmodules)))
    101107    logger.info('... done (%s todos, %s index, %s citations, %s modules)'%(
    102108            len(env.todo_all_todos),
    103             len(env.indexentries),
    104             len(env.domaindata["std"]["citations"]),
     109            len(env.domaindata['index']['entries']),
     110            len(env.domaindata['std']['citations']),
    105111            len(env.domaindata['py']['modules'])))
    106     write_citations(app, env.domaindata["std"]["citations"])
     112    write_citations(app, env.domaindata['std']['citations'])
    107113
    108114
    109115def get_env(app, curdoc):
    def fetch_citation(app, env): 
    253259    with open(filename, 'rb') as f:
    254260        cache = cPickle.load(f)
    255261    logger.info("done (%s citations)."%len(cache))
    256     cite = env.domaindata["std"]["citations"]
     262    cite = env.domaindata["std"].get("citations", dict())
    257263    for ind, (path, tag, lineno) in six.iteritems(cache):
    258264        if ind not in cite: # don't override local citation
    259265            cite[ind] = (os.path.join("..", path), tag, lineno)
  • src/sage_setup/docbuild/ext/sage_autodoc.py

    a b import sys 
    3535from docutils.statemachine import ViewList
    3636
    3737import sphinx
    38 from sphinx.ext.autodoc.importer import mock, import_object, get_object_members
     38from sphinx.ext.autodoc import mock
     39from sphinx.ext.autodoc.importer import import_object, get_object_members, get_module_members
    3940from sphinx.locale import _, __
    4041from sphinx.pycode import ModuleAnalyzer
    4142from sphinx.errors import PycodeError
    4243from sphinx.util import logging
    4344from sphinx.util import rpartition, force_decode
    4445from sphinx.util.docstrings import prepare_docstring
    45 from sphinx.util.inspect import isdescriptor, safe_getmembers, \
     46from sphinx.util.inspect import isdescriptor, \
    4647    safe_getattr, object_description, is_builtin_class_method, \
    4748    isenumattribute, isclassmethod, isstaticmethod, getdoc
    4849
    class Documenter(object): 
    536537
    537538        # add content from docstrings
    538539        if not no_docstring:
    539             encoding = self.analyzer and self.analyzer.encoding
     540            encoding = self.analyzer and self.analyzer._encoding
    540541            docstrings = self.get_doc(encoding)
    541542            if not docstrings:
    542543                # append at least a dummy docstring, so that the event
    class ModuleDocumenter(Documenter): 
    882883            if not hasattr(self.object, '__all__'):
    883884                # for implicit module members, check __module__ to avoid
    884885                # documenting imported objects
    885                 return True, safe_getmembers(self.object)
     886                return True, get_module_members(self.object)
    886887            else:
    887888                memberlist = self.object.__all__
    888889                # Sometimes __all__ is broken...
    class ModuleDocumenter(Documenter): 
    893894                        '(in module %s) -- ignoring __all__' %
    894895                        (memberlist, self.fullname))
    895896                    # fall back to all members
    896                     return True, safe_getmembers(self.object)
     897                    return True, get_module_members(self.object)
    897898        else:
    898899            memberlist = self.options.members or []
    899900        ret = []

comment:33 in reply to:  30 Changed 3 years ago by Antonio Rojas

Replying to fbissey:

Well, that's some progress I guess. I think we should move this ticket milestone to 9.2 and make it "upgrade to sphinx 3", that would be more honest because I don't think we can keep it compatible with sphinx-1.8 at the same time.

That seems totally reasonable to me - it's too late for 9.1, and I hope nobody will object to move on to py3-only for 9.2

comment:34 Changed 3 years ago by Antonio Rojas

Branch: u/gh-timokau/sphinx-2.1-supportu/arojas/sphinx-2.1-support

comment:35 Changed 3 years ago by Antonio Rojas

Authors: Timo KaufmannTimo Kaufmann, John Palmieri, François Bissey, Antonio Rojas
Commit: 447b6b071243f410df701679376c908394f759fd7041a6bff42f1f03d832eeed506701c6209cd897
Summary: Make sage compatible with sphinx 2.1Update to sphinx 3

I've pushed the current status to the branch. This now builds fine with -k, including references and citations.

It remains to figure out the WARNING: duplicate object description so it builds without -k (or decide that it's not important and whitelist the warning)


New commits:

2ac886aMerge remote-tracking branch 'origin/develop' into t/28856/sphinx-2.1-support
9a98405Normalize intersphinx mappings
f2087ccUpdate sphinx to 3.0.1
1fe8969Fix deprecation warnings with sphinx 3
34294f3Port away from BuildEnvironment.frompickle, removed in sphinx 3
8cc052dUse correct citation domain
0854100docenv.domaindata['py']['modules'] has one more entry in sphinx 3
7041a6bPort to index domain

comment:36 in reply to:  35 Changed 3 years ago by Markus Wageringel

Replying to arojas:

It remains to figure out the WARNING: duplicate object description so it builds without -k (or decide that it's not important and whitelist the warning)

The methods left_action_product and right_action_product actually do appear twice in the documentation, which is caused by this block:

    .. automethod:: Permutation.left_action_product
    .. automethod:: Permutation.right_action_product

In the past, this block was referencing private functions _left_to_right_multiply_on_right/_left_to_right_multiply_on_left, so the automethod directive was used to make those underscore functions appear in the documentation. Since these functions are not private anymore and appear in the documentation by default, this block is not needed anymore and removing it should solve the problem I hope.

comment:37 Changed 3 years ago by Antonio Rojas

Removing those lines does indeed fix this warning. But then compilation stops a while later with another warning:

OSError: /build/sagemath-doc/src/sage-9.0/src/doc/en/thematic_tutorials/structures_in_coding_theory.rst:722: WARNING: Could not lex literal_block as "python". Highlighting skipped.

comment:38 Changed 3 years ago by git

Commit: 7041a6bff42f1f03d832eeed506701c6209cd8970705982d22cbdd71e5343dbbdeaa6c3865a9b560

Branch pushed to git repo; I updated commit sha1. New commits:

0705982fix WARNING: duplicate object description

comment:39 Changed 3 years ago by Tobias Hansen

Thanks for the patch. I tried it with sphinx 2.4.3 which is what we have in Debian now. It failed with the following error:

[dochtml] [reference] Merging environment/index files...
[dochtml] [reference] Exception occurred:
[dochtml] [reference]   File "/home/thansen/src/sage/sagemath/sagemath/sage/src/sage_setup/docbuild/ext/multidocs.py", line 105, in merge_environment
[dochtml] [reference]     for ind,(modpath,v1,v2,v3,v4) in (
[dochtml] [reference] ValueError: not enough values to unpack (expected 5, got 4)

Do I have to change anything for sphinx 2.4? Which command are you running with -k?

comment:40 in reply to:  39 Changed 3 years ago by Antonio Rojas

Replying to thansen:

Thanks for the patch. I tried it with sphinx 2.4.3 which is what we have in Debian now. It failed with the following error:

[dochtml] [reference] Merging environment/index files...
[dochtml] [reference] Exception occurred:
[dochtml] [reference]   File "/home/thansen/src/sage/sagemath/sagemath/sage/src/sage_setup/docbuild/ext/multidocs.py", line 105, in merge_environment
[dochtml] [reference]     for ind,(modpath,v1,v2,v3,v4) in (
[dochtml] [reference] ValueError: not enough values to unpack (expected 5, got 4)

Do I have to change anything for sphinx 2.4? Which command are you running with -k?

Yes, the relevant sphinx change [1] is only in 3.x. For 2.4, remove the two line changes adding the v4 parameter from the patch.

The command I'm using to build the docs is python sage_setup/docbuild --no-pdf-links --mathjax all html -k

[1] https://github.com/sphinx-doc/sphinx/commit/5ff3b9dc4d07b5c324680e17b2aeaa298e619a9a

comment:41 Changed 3 years ago by git

Commit: 0705982d22cbdd71e5343dbbdeaa6c3865a9b56010c7ffd6e1b1f10ec3db39c17c4356741b5fc243

Branch pushed to git repo; I updated commit sha1. New commits:

10c7ffdPort to todo domain

comment:42 Changed 3 years ago by Antonio Rojas

I see the WARNING: Could not lex literal_block as "python" issue already came up in https://groups.google.com/forum/#!msg/sage-release/xcGXMuHuEOw/eqISWOiDAAAJ but, in that case, it affected python2 only.

comment:43 Changed 3 years ago by John Palmieri

With Sphinx 3.0.2 I see

Building reference manual, first pass.

[reference] Exception occurred:
[reference]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/sphinxext/plot_directive.py", line 268, in setup
[reference]     app.add_directive('plot', plot_directive, True, (0, 2, False), **options)
[reference] TypeError: add_directive() got an unexpected keyword argument 'alt'

This is coming from matplotlib 2.2.4. Do I need to upgrade matplotlib?

By the way, to install the new version of Sphinx in Sage, we need to add some other packages: sphinxcontrib-applehelp and about 5 others.

comment:44 Changed 3 years ago by John Palmieri

To answer my own question: upgrading matplotlib helps.

comment:45 Changed 3 years ago by John Palmieri

I'm now seeing another issue:

[plotting ] /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/sage/plot/contour_plot.py:docstring of sage.plot.contour_plot.contour_plot:438: WARNING: Exception occurred in plotting contour_plot-24
[plotting ]  from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/src/doc/en/reference/plotting/sage/plot/contour_plot.rst:
[plotting ] Traceback (most recent call last):
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/sphinxext/plot_directive.py", line 484, in run_code
[plotting ]     exec(code, ns)
[plotting ]   File "<string>", line 4, in <module>
[plotting ]   File "<string>", line 28, in sphinx_plot
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/sage/plot/graphics.py", line 2778, in matplotlib
[plotting ]     g._render_on_subplot(subplot)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/sage/plot/contour_plot.py", line 222, in _render_on_subplot
[plotting ]     cb = colorbar.Colorbar(cax, CSF, **kwds)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 1220, in __init__
[plotting ]     ColorbarBase.__init__(self, ax, **kw)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 459, in __init__
[plotting ]     ['uniform', 'proportional'], spacing=spacing)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/cbook/__init__.py", line 2145, in _check_in_list
[plotting ]     .format(v, k, ', '.join(map(repr, values))))
[plotting ] ValueError: None is not a valid value for spacing; supported values are 'uniform', 'proportional'

comment:46 Changed 3 years ago by François Bissey

Which version of matplotlib did you go for? I think arch has stuff for 3.2+ but 3.1 should be OK.

comment:47 Changed 3 years ago by Matthias Köppe

#29444 upgraded matplotlib to 2.2.5, which is the last release supporting py2. Am I right that this ticket is for 9.2, in which we drop python 3 support?

comment:48 in reply to:  46 Changed 3 years ago by John Palmieri

Replying to fbissey:

Which version of matplotlib did you go for? I think arch has stuff for 3.2+ but 3.1 should be OK.

I upgraded to 3.2.1 (within Sage).

comment:49 in reply to:  47 Changed 3 years ago by John Palmieri

Replying to mkoeppe:

#29444 upgraded matplotlib to 2.2.5, which is the last release supporting py2. Am I right that this ticket is for 9.2, in which we drop python 3 support?

Yes, I think it has to be.

comment:50 Changed 3 years ago by Matthias Köppe

Milestone: sage-9.1sage-9.2

comment:51 in reply to:  45 ; Changed 3 years ago by Antonio Rojas

Replying to jhpalmieri:

I'm now seeing another issue:

[plotting ] /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/sage/plot/contour_plot.py:docstring of sage.plot.contour_plot.contour_plot:438: WARNING: Exception occurred in plotting contour_plot-24
[plotting ]  from /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/src/doc/en/reference/plotting/sage/plot/contour_plot.rst:
[plotting ] Traceback (most recent call last):
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/sphinxext/plot_directive.py", line 484, in run_code
[plotting ]     exec(code, ns)
[plotting ]   File "<string>", line 4, in <module>
[plotting ]   File "<string>", line 28, in sphinx_plot
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/sage/plot/graphics.py", line 2778, in matplotlib
[plotting ]     g._render_on_subplot(subplot)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/sage/plot/contour_plot.py", line 222, in _render_on_subplot
[plotting ]     cb = colorbar.Colorbar(cax, CSF, **kwds)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 1220, in __init__
[plotting ]     ColorbarBase.__init__(self, ax, **kw)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 459, in __init__
[plotting ]     ['uniform', 'proportional'], spacing=spacing)
[plotting ]   File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.1.rc0/local/lib/python3.7/site-packages/matplotlib/cbook/__init__.py", line 2145, in _check_in_list
[plotting ]     .format(v, k, ', '.join(map(repr, values))))
[plotting ] ValueError: None is not a valid value for spacing; supported values are 'uniform', 'proportional'

You need another patch for matplotlib 3.2: https://git.archlinux.org/svntogit/community.git/tree/trunk/sagemath-matplotlib-3.2.patch?h=packages/sagemath

Last edited 3 years ago by Antonio Rojas (previous) (diff)

comment:52 Changed 3 years ago by Antonio Rojas

The remaining warning comes from the lines

.. CODE-BLOCK:: python

    :class:`sage.coding.repetition_code.BinaryRepetitionCode <sage.coding.repetition_code.BinaryRepetitionCode>`
    #the line above creates a link to the class in the html documentation of coding theory library
    from sage.coding.repetition_code import BinaryRepetitionCode

in structures_in_coding_theory.rst. This is because the backtick is no longer valid syntax in python3.

I don't understand what this code block is meant to do. In the python2 version, it displayed a code block with those three lines, which doesn't make much sense to me (the first line is not python code). Perhaps the link to the class is meant to be outside the code block? (which still doesn't make sense to me in the overall context of the page)

comment:53 Changed 3 years ago by git

Commit: 10c7ffd6e1b1f10ec3db39c17c4356741b5fc243aa41a308eb3bc215cf0390d2e6fe70d58b83fdcf

Branch pushed to git repo; I updated commit sha1. New commits:

aa41a30Fix incomplete port to citation domain

comment:54 Changed 3 years ago by François Bissey

Nice little commit. So far building without -k and I passed the point where it would have bailed out because of missing citations in [tensor_fr]. But citation and citations, we have to be careful not to get too confused.

comment:55 Changed 3 years ago by François Bissey

But bail out at structures_in_coding_theory.rst as should have been expected. Until that one is sorted we still need the -k.

comment:56 Changed 3 years ago by Antonio Rojas

For now I'm removing the first two lines of the affected code block downstream until someone clears that up. Doesn't seem much of a loss. After that it builds fine without -k

comment:57 Changed 3 years ago by Antonio Rojas

FTR, this was marked as a python code block in #27549

comment:58 Changed 3 years ago by John Palmieri

I think it's fine to delete the first two lines of that code block.

comment:59 in reply to:  51 ; Changed 3 years ago by John Palmieri

Replying to arojas:

You need another patch for matplotlib 3.2: https://git.archlinux.org/svntogit/community.git/tree/trunk/sagemath-matplotlib-3.2.patch?h=packages/sagemath

Thanks, that works.

Now I'm seeing a ton of messages like

WARNING: citation not found: BM2012

which causes the build to fail unless I use -k.

comment:60 in reply to:  59 ; Changed 3 years ago by Antonio Rojas

Replying to jhpalmieri:

Replying to arojas:

You need another patch for matplotlib 3.2: https://git.archlinux.org/svntogit/community.git/tree/trunk/sagemath-matplotlib-3.2.patch?h=packages/sagemath

Thanks, that works.

Now I'm seeing a ton of messages like

WARNING: citation not found: BM2012

which causes the build to fail unless I use -k.

Did you pull the latest commit?

comment:61 in reply to:  60 Changed 3 years ago by John Palmieri

Replying to arojas:

Replying to jhpalmieri:

Replying to arojas:

You need another patch for matplotlib 3.2: https://git.archlinux.org/svntogit/community.git/tree/trunk/sagemath-matplotlib-3.2.patch?h=packages/sagemath

Thanks, that works.

Now I'm seeing a ton of messages like

WARNING: citation not found: BM2012

which causes the build to fail unless I use -k.

Did you pull the latest commit?

I'm sorry, no. I meant to and then started having computer problems, which distracted me. With the latest commit, everything builds!

comment:62 Changed 3 years ago by git

Commit: aa41a308eb3bc215cf0390d2e6fe70d58b83fdcf830a5d5c6dbceaf7505947726a6b36f58b67e6cc

Branch pushed to git repo; I updated commit sha1. New commits:

830a5d5Remove non-python code from python code block

comment:63 Changed 3 years ago by Antonio Rojas

It remains to sort out the Sage packaging side: add the missing sphinx dependencies, open a ticket to upgrade matplotlib and make it a dependency of this one. I'll leave that to someone else, I've had enough of this ticket for a while.

comment:64 in reply to:  63 Changed 3 years ago by François Bissey

Replying to arojas:

It remains to sort out the Sage packaging side: add the missing sphinx dependencies, open a ticket to upgrade matplotlib and make it a dependency of this one. I'll leave that to someone else, I've had enough of this ticket for a while.

That's fine, you did a good job here and we in distros are now ready to tackle 9.1 in python3 only. Thank you very much for your work here.

I should be able to find time to shepherd the rest in early 9.2.

comment:65 Changed 3 years ago by François Bissey

@jhpalmieri do you have something ready for matplotlib or should I go ahead with a new ticket and branch?

comment:66 Changed 3 years ago by François Bissey

I am going to add the following sphinxcontrib packages following the dependencies of the Gentoo ebuild

  • sphinxcontrib-applehelp
  • sphinxcontrib-devhelp
  • sphinxcontrib-jsmath
  • sphinxcontrib-htmlhelp
  • sphinxcontrib-serializinghtml
  • sphinxcontrib-qthelp

which should cover @jhpalmieri's remark about needing 6 new sphinxcontrib packages. Did anyone identify something else missing from the current list of sage packages?

comment:67 Changed 3 years ago by John Palmieri

Those are the ones. With those, Sphinx 3.0.2 builds for me with Sage 9.1.rc0 (py3) on OS X.

comment:68 in reply to:  65 Changed 3 years ago by John Palmieri

Replying to fbissey:

@jhpalmieri do you have something ready for matplotlib or should I go ahead with a new ticket and branch?

I can open up a ticket. I have a version which builds, but I haven't run doctests yet.

Edit: #29547

Last edited 3 years ago by John Palmieri (previous) (diff)

comment:69 Changed 3 years ago by François Bissey

Darn, we are still on 3.0.1. I'll bump to 3.0.2 while I am at it. Do open the ticket for the new matplotlib. We do know that some doctests needs fixing from 3.1 onwards. I'll fill the details on #29547 if it is not there already.

comment:70 Changed 3 years ago by François Bissey

Branch: u/arojas/sphinx-2.1-supportu/fbissey/sphinx3
Commit: 830a5d5c6dbceaf7505947726a6b36f58b67e6cc88928933ac72a1d2bf131a223e065359bcc62a3e

Branch with sphinx 3.0.2 and all new packages.


New commits:

c62c1a4update sphinx to 3.0.2
66f544dupdate sphinx_contribwebsupport
8892893Add new dependencies of sphinx

comment:71 Changed 3 years ago by John Palmieri

Ready for review?

comment:72 Changed 3 years ago by François Bissey

I'd say. I haven't tested everything in vanilla sage but this builds on sage-on-gentoo and presumably sage-on-arch [with sphinx 3.0.1 in gentoo so we could find an issue]. My packaging needs to be checked.

comment:73 Changed 3 years ago by François Bissey

Description: modified (diff)

comment:74 Changed 3 years ago by François Bissey

Description: modified (diff)
Status: needs_workneeds_review

comment:75 Changed 3 years ago by John Palmieri

It builds for me on OS X (with some homebrew packages).

comment:76 Changed 3 years ago by John Palmieri

I get two doctest failures (this is combined with #29547):

sage -t --long --warn-long 80.3 src/sage/misc/sagedoc.py
**********************************************************************
File "src/sage/misc/sagedoc.py", line 23, in sage.misc.sagedoc
Failed example:
    with open(docfilename) as fobj:  # optional - dochtml
        for line in fobj:
            if "#sage.symbolic.expression.Expression.numerical_approx" in line:
                print(line)
Expected:
    <code class="descname">numerical_approx</code><span class="sig-paren">(</span><em>prec=None</em>, <em>digits=None</em>, <em>algorithm=None</em><span class="sig-paren">)</span>...
Got:
    <code class="sig-name descname">numerical_approx</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">prec</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="n">digits</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="n">algorithm</span><span class="o">=</span><span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#sage.symbolic.expression.Expression.numerical_approx" title="Permalink to this definition">¶</a></dt>
    <BLANKLINE>

and

sage -t --long --warn-long 80.3 src/sage/docs/conf.py
**********************************************************************
File "src/sage/docs/conf.py", line 666, in sage.docs.conf.call_intersphinx
Failed example:
    for line in open(thematic_index).readlines():  # optional - dochtml
        if "padics" in line:
            _ = sys.stdout.write(line)
Expected:
    <li><a class="reference external" href="../reference/padics/sage/rings/padics/tutorial.html#sage-rings-padics-tutorial" title="(in Sage... Reference Manual: p-Adics v...)"><span>Introduction to the p-adics</span></a></li>
Got:
    <li><p><a class="reference external" href="../reference/padics/sage/rings/padics/tutorial.html#sage-rings-padics-tutorial" title="(in Sage 9.1.rc0 Reference Manual: p-Adics v9.1.rc0)"><span>Introduction to the p-adics</span></a></p></li>

They don't look hard to fix.

comment:77 Changed 3 years ago by François Bissey

Unexpected. I didn't see that in my doctesting in Gentoo, may be one of the package sphinx/sphinxcontrib bumps caused this. But as you say, it is trivial to fix.

comment:78 Changed 3 years ago by John Palmieri

I'm trying again, building from scratch, just to make sure.

comment:79 Changed 3 years ago by Matthias Köppe

Needs rebasing on top of 9.1.rc1.

Also please add an upstream_url field (see https://wiki.sagemath.org/ReleaseTours/sage-9.1)

comment:80 Changed 3 years ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:81 Changed 3 years ago by François Bissey

That's a cool new feature considering all the instances of "please the package is not on the mirror" on sage-release and sage-devel.

comment:82 Changed 3 years ago by git

Commit: 88928933ac72a1d2bf131a223e065359bcc62a3e0e44ccc36bd7acbf32ddab87a74359ec61509a26

Branch pushed to git repo; I updated commit sha1. New commits:

ba2f503Merge branch 'develop' into sphinx3
0e44cccadd upstream_url to all sphinx packages.

comment:83 Changed 3 years ago by François Bissey

Status: needs_workneeds_review

comment:84 Changed 3 years ago by John Palmieri

I see the same doctest failures on two different machines, so I think they're real.

comment:85 Changed 3 years ago by François Bissey

I'll fix them then. Having trouble building sage-on-gentoo right now. The head from Volker (past 9.1.rc1) doesn't build for me. I will have to test on the plain develop branch which is unusual for me these days.

comment:86 Changed 3 years ago by git

Commit: 0e44ccc36bd7acbf32ddab87a74359ec61509a260a64b94508bfe0b28750681d463c2aedc25379b3

Branch pushed to git repo; I updated commit sha1. New commits:

0a64b94Fix the last two doctests

comment:87 Changed 3 years ago by François Bissey

All fixed now.

comment:88 Changed 3 years ago by Antonio Rojas

Only one of them is fixed AFAICS

comment:89 Changed 3 years ago by Matthias Köppe

For upstream_url, it's better to use pypi.io URLs because they will be stable for the next updates (see matplotlib, for example)

comment:90 in reply to:  81 Changed 3 years ago by Matthias Köppe

Replying to fbissey:

That's a cool new feature considering all the instances of "please the package is not on the mirror" on sage-release and sage-devel.

Glad you like it. Note that it's not enabled by default, so that developers still complain when the release manager forgets to upload it to the mirror.

comment:91 in reply to:  88 ; Changed 3 years ago by François Bissey

Replying to arojas:

Only one of them is fixed AFAICS

Weird only one of my changes made it up. I must have fumbled a git command last night.

comment:92 Changed 3 years ago by git

Commit: 0a64b94508bfe0b28750681d463c2aedc25379b3454f2a88102b346c16139382c4af34d0db2ccd8f

Branch pushed to git repo; I updated commit sha1. New commits:

454f2a8add forgotten doctest fix.

comment:93 in reply to:  89 ; Changed 3 years ago by François Bissey

Replying to mkoeppe:

For upstream_url, it's better to use pypi.io URLs because they will be stable for the next updates (see matplotlib, for example)

OK, is this some sort of secret API or something? Because it is not visible (anymore) from the pypi website. I do vaguely recall it being done that way in the past. Will it go away completely sometime in the future?

comment:94 in reply to:  91 Changed 3 years ago by François Bissey

Replying to fbissey:

Replying to arojas:

Only one of them is fixed AFAICS

Weird only one of my changes made it up. I must have fumbled a git command last night.

Fixed now.

comment:95 in reply to:  93 ; Changed 3 years ago by Matthias Köppe

Replying to fbissey:

Replying to mkoeppe:

For upstream_url, it's better to use pypi.io URLs because they will be stable for the next updates (see matplotlib, for example)

OK, is this some sort of secret API or something? Because it is not visible (anymore) from the pypi website. I do vaguely recall it being done that way in the past. Will it go away completely sometime in the future?

@isuruf showed me this trick, I don't know where this is documented. It's also mentioned here: https://stackoverflow.com/questions/47781035/does-pypi-have-simple-urls-for-package-downloads

comment:96 Changed 3 years ago by git

Commit: 454f2a88102b346c16139382c4af34d0db2ccd8f6f0e8b36d840c1559bbec88e5cadaeaa5e1bb058

Branch pushed to git repo; I updated commit sha1. New commits:

6f0e8b3change upstream_url to a (hopefully) more stable and easy to decipher address for sphinx packages

comment:97 in reply to:  95 Changed 3 years ago by François Bissey

Replying to mkoeppe:

Replying to fbissey:

Replying to mkoeppe:

For upstream_url, it's better to use pypi.io URLs because they will be stable for the next updates (see matplotlib, for example)

OK, is this some sort of secret API or something? Because it is not visible (anymore) from the pypi website. I do vaguely recall it being done that way in the past. Will it go away completely sometime in the future?

@isuruf showed me this trick, I don't know where this is documented. It's also mentioned here: https://stackoverflow.com/questions/47781035/does-pypi-have-simple-urls-for-package-downloads

OK, I have gone ahead and made the change. I think we are all done here.

comment:98 Changed 3 years ago by Matthias Köppe

Best to replace pip in dependencies by $(PYTHON_TOOLCHAIN), see #26018

comment:99 Changed 3 years ago by François Bissey

Oh, right. I had seen that. Doing after my (ongoing) morning meeting.

comment:100 Changed 3 years ago by git

Commit: 6f0e8b36d840c1559bbec88e5cadaeaa5e1bb05828ccaa39ac5e6f3196bc8918ed7c3da6c70dee5d

Branch pushed to git repo; I updated commit sha1. New commits:

28ccaa3move sphinx packages from pip to PYTHON_TOOLCHAIN

comment:101 Changed 3 years ago by François Bissey

Was bored. Done. Any other things that should be migrated to new toys?

comment:102 in reply to:  101 ; Changed 3 years ago by Matthias Köppe

Replying to fbissey:

Any other things that should be migrated to new toys?

Migrate, I don't know, but you could test this branch with the new CI system by pushing the branch to a github repo. For extra fun, use a pull request against the branch of #29530.

comment:103 in reply to:  102 Changed 3 years ago by François Bissey

Replying to mkoeppe:

Replying to fbissey:

Any other things that should be migrated to new toys?

Migrate, I don't know, but you could test this branch with the new CI system by pushing the branch to a github repo. For extra fun, use a pull request against the branch of #29530.

I really should give that a try. Not sure about #29530 though.

comment:104 Changed 3 years ago by François Bissey

Anyone tried to build the pdf doc? I got difficulties with it in sage-on-gentoo. Volker usually doesn't think it is critical though, more a nice to have.

comment:105 Changed 3 years ago by François Bissey

That's what I get at the end of a serial build, it is a bit scary

[1528]

LaTeX Warning: Hyper reference `sage/combinat/tableau:sage.combinat.tableau.Row
StandardTableaux' on page 1529 undefined on input line 132175.

[1529]
! TeX capacity exceeded, sorry [input stack size=5000].
\font@name ->
             \TS1/ptm/m/n/10 
l.132260 ...ikipedia article Zolotarev’s\_lemma}
                                                  ^^M
If you really absolutely need more capacity,
you can ask a wizard to enlarge me.

 
Here is how much of TeX's memory you used:
 21911 strings out of 481640
 403969 string characters out of 5928785
 1146413 words of memory out of 5000000
 31270 multiletter control sequences out of 15000+600000
 610234 words of font info for 159 fonts, out of 8000000 for 9000
 817 hyphenation exceptions out of 8191
 5000i,29n,9989p,5239b,810s stack positions out of 5000i,500n,10000p,200000b,80000s
!  ==> Fatal error occurred, no output PDF file produced!

comment:106 Changed 3 years ago by John Palmieri

I explored some of this "by hand": I did ./sage --docbuild all latex and then ran pdflatex on a bunch of the LaTeX files. I've found three errors like this, and they all arise from this sort of line:

See \sphinxhref{https://en.wikipedia.org/wiki/Dilworth\textquotesingle{}s\_theorem}{Wikipedia article Dilworth’s\_theorem}.

Removing \textquotesingle{} allows the file to compile.

Can we rewrite our :wikipedia: markup converter to fix this? Or is it an issue with a change in Sphinx?

comment:107 Changed 3 years ago by John Palmieri

The file sphinx/util/texescape.py seems to contain the change causing the issue, so the question is how we handle this within the Sage library.

(In Sphinx, commenting out the line

("'", r'\textquotesingle{}'),  # else ' renders curly, and '' is a ligature

in sphinx/util/texescape.py fixes it, but we want to avoid patching Sphinx. I'm guessing that our introduction of the :wikipedia: markup is not done the "right way".)

comment:108 Changed 3 years ago by Markus Wageringel

You could try to escape single quotes in external links by %27 in process_extlinks in sagedoc.py, e.g. https://en.wikipedia.org/wiki/Dilworth%27s_theorem.

Edit: Or use urllib.parse.quote (actually, this might escape too much).

Last edited 3 years ago by Markus Wageringel (previous) (diff)

comment:109 Changed 3 years ago by François Bissey

I am running a sage-on-gentoo github issue in parallel at https://github.com/cschwan/sage-on-gentoo/issues/582 I have more details on some of my failures. Both you in here and Steven Trogdon on github have been busy while I was snoozing.

Have to assemble the info from both threads sometimes today.

comment:110 Changed 3 years ago by Steven Trogdon

Cc: Steven Trogdon added

comment:111 Changed 3 years ago by François Bissey

Indication seem to be that using %27 after :wikipedia: is effective, but there are quite a few place that replacement can be made. I would aim for consistency

src/sage/categories/semigroups.py:                - :wikipedia:`Green's_relations`
src/sage/categories/semigroups.py:                - :wikipedia:`Green's_relations`
src/sage/categories/semigroups.py:                - :wikipedia:`Green's_relations`
src/sage/categories/semigroups.py:                - :wikipedia:`Green's_relations`
src/sage/combinat/partition.py:        - :wikipedia:`Zolotarev's_lemma`
src/sage/combinat/posets/posets.py:        :wikipedia:`Dilworth's_theorem` for more information. The width is
src/sage/combinat/posets/posets.py:        See :wikipedia:`Dilworth's_theorem`.
src/sage/combinat/root_system/plot.py::wikipedia:`Wikipedia's E8 3D picture <File:E8_3D.png>`::
src/sage/functions/jacobi.py:- :wikipedia:`Jacobi's_elliptic_functions`
src/sage/geometry/riemannian_manifolds/surface3d_generators.py:        For more information, see :wikipedia:`Dini's_surface`.
src/sage/graphs/generators/smallgraphs.py:    :wikipedia:`Tietze's_graph`.
src/sage/graphs/path_enumeration.pyx:    See [Yen1970]_ and the :wikipedia:`Yen's_algorithm` for more details on the
src/sage/graphs/base/static_sparse_graph.pyx:    :wikipedia:`Tarjan's_strongly_connected_components_algorithm`.
src/sage/graphs/bipartite_graph.py:            algorithm (:wikipedia:`Kőnig's_theorem_(graph_theory)`)
src/sage/graphs/spanning_tree.pyx:        - :wikipedia:`Kruskal's_algorithm`
src/sage/graphs/spanning_tree.pyx:        - :wikipedia:`Kruskal's_algorithm`
src/sage/matroids/linear_matroid.pyx:        See also :wikipedia:`Kirchhoff's_theorem`.

Plus

src/sage/plot/plot3d/parametric_plot3d.py:    Boy's surface (:wikipedia:`Boy%27s_surface` and https://mathcurve.com/surfaces/boy/boy.shtml)::

which I just changed.

comment:112 Changed 3 years ago by Steven Trogdon

sage: from sage.misc.sagedoc import process_extlinks
sage: process_extlinks(":wikipedia:`Boy's`")
"https://en.wikipedia.org/wiki/Boy's"

So, can process_extlinks be modified to replace ' with %27? Or is process_extlinks the wrong place?

comment:113 Changed 3 years ago by John Palmieri

process_extlinks won't help: as far as I can tell, the extlinks dictionary is passes as is to Sphinx for processing. (process_extlinks is for printing docstrings for introspection at the command line.)

comment:114 Changed 3 years ago by François Bissey

I am currently replacing all those pesky ' that I have quoted earlier. I will let you know how that goes.

comment:115 Changed 3 years ago by John Palmieri

Branch: u/fbissey/sphinx3u/jhpalmieri/sphinx3

comment:116 in reply to:  114 Changed 3 years ago by John Palmieri

Commit: 28ccaa39ac5e6f3196bc8918ed7c3da6c70dee5d94b4570cbb42a0f11adf2a2d4f1fd8af2cc695b4

Replying to fbissey:

I am currently replacing all those pesky ' that I have quoted earlier. I will let you know how that goes.

Independently I was doing the same thing. Here is a branch. It works for me; please test it.

(The html and pdf are uglier, since the links now include "%27" instead of an apostrophe, but I think we have to settle for that.)


New commits:

94b4570trac 28856: fix apostrophes in wikipedia links

comment:117 Changed 3 years ago by François Bissey

I had two left to go :) oh well, you beat me to it.

comment:118 Changed 3 years ago by John Palmieri

There was also a missing space in matrix_gps/finitely_generated.py that caused problems.

comment:119 Changed 3 years ago by François Bissey

Works for me. I guess modulo a rebase when 9.1 is out, we are ready.

comment:120 Changed 3 years ago by Tobias Hansen

Did you try searching in the html documentation? For me this does not work anymore with Sphinx 2.4 and the patches from here. It never returns any results.

The reason I tested it is that the Debian packaging tool dh_sphinxdoc performs some sanity checks on sphinx documentation and complained that (at least) one of the search.html does not load searchindex.js. The checks it performs can be seen here: https://salsa.debian.org/python-team/modules/sphinx/-/blob/debian/master/debian/dh-sphinxdoc/dh_sphinxdoc#L284

comment:121 in reply to:  120 Changed 3 years ago by Antonio Rojas

Replying to thansen:

Did you try searching in the html documentation? For me this does not work anymore with Sphinx 2.4 and the patches from here. It never returns any results.

Works for me. I do get some missing labels and icons on the search page, but I see the exact same thing when using sphinx 1 and on the online version

Last edited 3 years ago by Antonio Rojas (previous) (diff)

comment:122 in reply to:  120 Changed 3 years ago by François Bissey

Replying to thansen:

Did you try searching in the html documentation? For me this does not work anymore with Sphinx 2.4 and the patches from here. It never returns any results.

The reason I tested it is that the Debian packaging tool dh_sphinxdoc performs some sanity checks on sphinx documentation and complained that (at least) one of the search.html does not load searchindex.js. The checks it performs can be seen here: https://salsa.debian.org/python-team/modules/sphinx/-/blob/debian/master/debian/dh-sphinxdoc/dh_sphinxdoc#L284

Most of my development is remote and I must say it must have been years since I tried that particular functionality.

comment:123 Changed 3 years ago by Antonio Rojas

I've opened #29576 for the broken search page

comment:124 Changed 3 years ago by François Bissey

@jhpalmieri you own the branch on this ticket and #29547. May be we should rebase them. I think we should go ahead with #29547, although we could try upgrading freetype to 2.10.2 for the people who experience the crash.

comment:125 Changed 3 years ago by Antonio Rojas

I'm getting a new error now, not sure when it started (with 3.0.3)

OSError: /build/sagemath-doc/src/sage-9.1/src/doc/en/installation/source.rst:237: WARNING: Include file '/build/sagemath-doc/src/sage-9.1/src/doc/en/installation/debian.txt' not found or reading it failed

comment:126 Changed 3 years ago by François Bissey

During the development of 9.1 a bootstrap step has been added for the documentation. I think I mentioned it somewhere else, did you run it? https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/sage/sage-9.1.ebuild#L328 in sage-on-gentoo (${S} is SAGE_ROOT/src in vanilla sage terms).

comment:127 in reply to:  125 ; Changed 3 years ago by François Bissey

Replying to arojas:

I'm getting a new error now, not sure when it started (with 3.0.3)

OSError: /build/sagemath-doc/src/sage-9.1/src/doc/en/installation/source.rst:237: WARNING: Include file '/build/sagemath-doc/src/sage-9.1/src/doc/en/installation/debian.txt' not found or reading it failed

Yes this is totally what happened to you. Look at my comment https://trac.sagemath.org/ticket/29053#comment:78 on the ticket that introduced the change.

comment:128 in reply to:  127 Changed 3 years ago by Antonio Rojas

Replying to fbissey:

Yes this is totally what happened to you. Look at my comment https://trac.sagemath.org/ticket/29053#comment:78 on the ticket that introduced the change.

Thanks, clearly I wasn't paying much attention to that ticket.

comment:129 Changed 3 years ago by git

Commit: 94b4570cbb42a0f11adf2a2d4f1fd8af2cc695b4e5415da0fbdb0f40411b977d947760cf88767ea5

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

3d1403aRemove non-python code from python code block
ed74467update sphinx to 3.0.2
8db286bupdate sphinx_contribwebsupport
d09e965Add new dependencies of sphinx
2061379add upstream_url to all sphinx packages.
2b03045Fix the last two doctests
8387b7badd forgotten doctest fix.
2ddcd31change upstream_url to a (hopefully) more stable and easy to decipher address for sphinx packages
649289emove sphinx packages from pip to PYTHON_TOOLCHAIN
e5415datrac 28856: fix apostrophes in wikipedia links

comment:130 in reply to:  124 Changed 3 years ago by John Palmieri

Replying to fbissey:

@jhpalmieri you own the branch on this ticket and #29547. May be we should rebase them.

I just rebased this one.

I think we should go ahead with #29547, although we could try upgrading freetype to 2.10.2 for the people who experience the crash.

With #29547, git rebase develop works with no intervention, but also upgrading freetype doesn't seem to fix the doctest failures, so I'm not sure what to do with that ticket.

comment:131 Changed 3 years ago by François Bissey

We need at least Matplotlib 3.1 but I suspect even that could have an issue with freetype. Note that the problem may not be freetype itself but the interface to it in matplotlib.

comment:132 Changed 3 years ago by François Bissey

We have a merge conflict with #28000 (removal of py2 support from sagelib). Not very surprising. I will try to merge our branch with it and fix all the conflicts (removal of all calls to six). Volker is currently merging #28000 in his dev branch where he merges tickets incrementally before doing releases.

comment:133 Changed 3 years ago by François Bissey

I should mention that #29547 is not affected by #28000 which is already something.

comment:134 Changed 3 years ago by François Bissey

Branch: u/jhpalmieri/sphinx3u/fbissey/sphinx3
Commit: e5415da0fbdb0f40411b977d947760cf88767ea5b9d44e392d13b8db7c405e7e74e18edc105ee4d7
Dependencies: 28000

Merged with #28000 so no more merge conflicts there.


New commits:

f368e32trac 28000: rebase commits from Erik M. Bray <erik.bray@lri.fr>:
b9d44e3Merge branch 'ticket-28000' into sphinx3

comment:135 Changed 3 years ago by git

Commit: b9d44e392d13b8db7c405e7e74e18edc105ee4d78f66113223b4925deea44bcdfc2f3bb6c58fa697

Branch pushed to git repo; I updated commit sha1. New commits:

8f66113Minor upgdrade to Sphinx 3.0.3

comment:136 Changed 3 years ago by François Bissey

Ticket to keep a lookout on: #29633 it changes SPKG.txt to SPKG.rst. We may want to align the format if it is merged first.

comment:137 Changed 2 years ago by François Bissey

I will make a last bump to 3.0.4 which I know is safe. I don't know yet about 3.1.0 released in the last 24 hours. But I think it is time we move this ticket to positive review. Any volunteer reviewers?

comment:138 Changed 2 years ago by git

Commit: 8f66113223b4925deea44bcdfc2f3bb6c58fa697321e2bf0cb7ab489ac7362eae607698c7ec37347

Branch pushed to git repo; I updated commit sha1. New commits:

321e2bfBump to 3.0.4

comment:139 Changed 2 years ago by Antonio Rojas

Unsurprisingly, 3.1 doesn't work

[reference] dumping object inventory... failed
[reference] Exception occurred:
[reference]   File "/usr/lib/python3.8/site-packages/sphinx/domains/python.py", line 1325, in get_objects
[reference]     yield (modname, modname, 'module', mod.docname, mod.node_id, 0)
[reference] AttributeError: 'tuple' object has no attribute 'docname'

comment:140 Changed 2 years ago by François Bissey

I would prefer if we could merge this one already as is and open a new one for 3.1.x.

comment:141 Changed 2 years ago by John Palmieri

Dependencies: 28000#28000, #29547

comment:142 Changed 2 years ago by John Palmieri

This doesn't merge cleanly with 9.2.beta0.

comment:143 Changed 2 years ago by John Palmieri

Reviewers: Timo Kaufmann, John Palmieri, François Bissey, Antonio Rojas
Status: needs_reviewpositive_review

Let's move forward with this, though. (Maybe the merge problems can be fixed by squashing some commits? I don't really see what needs doing: the rebasing fails for multidocs.py, but after I clean up the rebasing, my diff looks just like the diff on this ticket.)

comment:144 Changed 2 years ago by François Bissey

I actually have a branch that merges 9.2.beta0 and #29547. I should double check it and push it. It should merge cleanly.

comment:145 Changed 2 years ago by git

Commit: 321e2bf0cb7ab489ac7362eae607698c7ec37347bd99462f6f89757b9b1674fbeb55743ce7f6fab6
Status: positive_reviewneeds_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

9a4b89btrac 29541: matplotlib 3.2.1
cad0196Easy to fix doctests
8640711trac 29547: fix doctest in root_lattice_realizations.py
b436dbctrac 29547: rebase and restore upstream_url line in checksums.ini
bb7e259Add matplotlib upstream patch to deal with memory leaks on OS X. The patch should probably removed in the next matplotlib upgrade.
eee7843trac 29547: use sage-python23 instead of sage-system-python to install
ee7195cMerge branch 'matplotlib3.2.1' into sphinx3
bd99462Merge branch 'develop' into sphinx3

comment:146 Changed 2 years ago by François Bissey

It should merge cleanly now.

comment:147 Changed 2 years ago by John Palmieri

Status: needs_reviewpositive_review

Now git merge develop works for me. Thank you for rebasing!

comment:148 Changed 2 years ago by Volker Braun

Branch: u/fbissey/sphinx3bd99462f6f89757b9b1674fbeb55743ce7f6fab6
Resolution: fixed
Status: positive_reviewclosed

comment:149 Changed 2 years ago by Antonio Rojas

Commit: bd99462f6f89757b9b1674fbeb55743ce7f6fab6

3.1 upgrade up at #30001

comment:150 Changed 2 years ago by gh-kliem

It seems that this ticket breaks building sage on centos 7.

I opened #30008, please fix.

Note: See TracTickets for help on using tickets.