Remove the highlighting patch for Sphinx

[jhpalmieri]

The goal of this ticket is to clean up highlighting in the documentation. There are several issues. Right now, we are patching pygments and Sphinx so that they know about the "sage:" prompt. Also, we are telling Sphinx to highlight all code blocks as if they were Python console blocks. There are various other types of code blocks in the Sage docs: Python, Cython, reST, LaTeX, etc., and right now many are not highlighted at all: Sphinx/pygments tries to parse them as the Python console and fails, so does nothing. Even some of the Sage console blocks aren't highlighted correctly, because pygments refuses to parse input like sage: a?: valid IPython but not valid Python.

To improve the situation: Sage's console is based more on IPython than on Python, and IPython comes with a lexer to use when building its own documentation. Furthermore, that lexer explicitly allows for customization of the input prompt. So we can get rid of the prompt patches and use IPython's lexer, and it succeeds at highlighting blocks containing sage: a? and other IPython-type text. The only disadvantage to this is that IPython's lexer doesn't convert <BLANKLINE> in doctest output to a blank line, so we add a Sphinx transform to handle this.

We also mark some non-Python blocks as what they are, to enable their highlighting.

[jhpalmieri]

We want Sage's doctest blocks (and most other literal blocks) to be typeset as Python console code. Setting the default highlighting language to pycon does this, and it means that we can then not use the highlighting patch for Sphinx.

Currently in Sage, there are some blocks which are not highlighted correctly. See ​this page: in the first block (containing CombinatorialFreeModule?), the prompt "sage:" is not highlighted. With the proposed change, that block will still not be highlighted the way we want, and in fact it will produce a warning:

WARNING: Could not lex literal_block as "pycon". Highlighting skipped.

We have set up the docbuilding process so that warnings stop it, so we also need to silence this warning.

There are no doubt some blocks in the Sage code which should not be highlighted using pycon, but as things stand in current Sage, Sphinx already tries to highlight them using the default lexer, which basically means python3 with a graceful failure if it can't parse it as Python 3: see ​the Sphinx docs. Once we set the highlighting language to something other than default, we lose the graceful failure, and so we have to silence the warnings.

In my spot-checking, the current non-Python console blocks fail their highlighting and so have no highlighting at all, and that behavior continues with this branch. At some point we could try to modify those blocks: something like

Here is a block::

.. highlight:: rest

    stuff

But that can be on another ticket.

[jhpalmieri]

A few blocks in Sage look like

Traceback:
..   <-- should be exactly three dots
more stuff

Having 2 or 4 dots means that Sphinx can't handle the highlighting: you can see this ​here, where there are 4 dots instead of 3.

Along the same lines, having extra spaces at the end of NotImplementedError causes highlighting to fail ​here.

[jhpalmieri]

New commits:

​1debde6

trac 27528: remove highlighting.patch for Sphinx

[jhpalmieri]

By the way, I've tried this with #26451 (Sphinx 1.8.4), #25680 (Python 3.7.2), and both together. No troubles.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​48b7acd	trac 27528: remove some patches for Sphinx and pygments.
​b23b8cc	trac 27528: use a custom Sphinx transform to handle <BLANKLINE>
​687b338	trac 27528: clean up some code-blocks in the reST documentation.

[jhpalmieri]

Here is a new approach: I just discovered the IPython lexer, and we should definitely use it rather than Python lexer.

[jhpalmieri]

A few things to check:

• See ​http://doc.sagemath.org/html/en/constructions/interface_issues.html for some code blocks for other languages (LaTeX, BibTeX, shell session). These are not currently highlighted, but they will be with this branch. Same goes for various shell session blocks scattered through the developer's guide.
• The first block in the ​tour-help section of the tutorial is not currently highlighted, but it will be with this branch.
• double check that <BLANKLINE> does not appear in the reference manual. One spot to check: from the main reference manual page, click "Symbolic Calculus" and then "Riemann Mapping". The method "complex_to_spiderweb" (toward the end) has some <BLANKLINE>s in the doctest output, and they should be rendered as blank lines in the reference manual.

[jhpalmieri]

To do on another ticket: go through more of the reST docs and perhaps the reference manual, marking the non-sage-doctest blocks using .. CODE-BLOCK:: language. I'll open up a followup ticket eventually.

[dimpase]

works for me nicely. I looked at few random places in various docs, all good.

[jhpalmieri]

Thank you for the review.

See #27549 for a followup: more code-block cleanup.

[jdemeyer]

Remember to change the package version if you change patches.

[dimpase]

How about we just mark this ticket to be merged together with #26451 - which does a version bump?

[jdemeyer]

Replying to dimpase:

How about we just mark this ticket to be merged together with #26451 - which does a version bump?

If it can be merged separately, I see no reason to bundle them.

[dimpase]

What's the point of doing this? To do more work than needed? Let's just merge them together.

[jdemeyer]

New commits:

​278fa3b

Bump package versions

[jdemeyer]

Sometimes it takes less effort to do the right thing rather than arguing about it...

[dimpase]

OK

[jdemeyer]

Looking at this diff, it seems that indentation is not correct in a few places, for example

+  that the user should be aware of.
+
+   .. CODE-BLOCK:: rest

[jdemeyer]

I also have doubts about

There are 2 tests in doc/en/developer/coding_in_python.rst that are not being run

[jdemeyer]

And shouldn't the in2_regex for IPythonConsoleLexer be customized too to match ....: ?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a323c09

Fix a few CODE-BLOCK indentations

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4cabfa7

Fix _test_enough_doctests()

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​83436ea

Fix usage of IPythonConsoleLexer

[jdemeyer]

I didn't understand the rationale for making a difference depending on the Python version running the docbuilder: the sources are the same on Python 2 and Python 3, so the documentation should also be the same. So I'm using the default version (which is currently Python 2) always.

[jdemeyer]

Please review the last 3 commits.

[dimpase]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​a323c09

Fix a few CODE-BLOCK indentations

I thought wrong indentations result in sphinx errors...

[jhpalmieri]

My one objection is that the version for the pygments package should not say .p0 since there are no patches any more. Is there anything to be done about this? (Aside from waiting for the next release, that is.)

[jhpalmieri]

Replying to dimpase:

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​a323c09

Fix a few CODE-BLOCK indentations

I thought wrong indentations result in sphinx errors...

Sometimes it just results in Sphinx silently ignoring the directive.

[jhpalmieri]

Replying to jhpalmieri:

Replying to dimpase:

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​a323c09

Fix a few CODE-BLOCK indentations

I thought wrong indentations result in sphinx errors...

Sometimes it just results in Sphinx silently ignoring the directive.

Actually this case looked like

text

 .. CODE-BLOCK:: rest

    more text

and Sphinx interpreted the extra space in front of .. CODE-BLOCK... just as indentation, so the code block was highlighted correctly, just indented more than it should have been.

[jhpalmieri]

Jeroen, thanks for the fixes. There is probably nothing to be done about the pygments version number.