Upgrade to Sphinx 1.5.x

[jdemeyer]

The newer version of Sphinx needs requests.

Tarballs:

• ​https://pypi.python.org/packages/a7/df/4487783152b14f2b7cd0b0c9afb119b262c584bf972b90ab544b61b74c62/Sphinx-1.5.3.tar.gz
• ​https://pypi.python.org/packages/16/09/37b69de7c924d318e51ece1c4ceb679bf93be9d05973bb30c35babd596e2/requests-2.13.0.tar.gz

We add a patch to Sphinx to allow running without SSL: ​https://github.com/sphinx-doc/sphinx/pull/3554

[fbissey]

Should we update docutils to 0.13 at the same time? sphinx 1.5.2 as all the fix to work with it as far as I know, can be in a separate ticket but it is better if they are done at the same time given the comments in ​https://github.com/cschwan/sage-on-gentoo/issues/451

[jdemeyer]

I would say: let's do whatever is easiest. So the question is: is it easier to make Sage and Sphinx-1.5.2 work with docutils-0.12 or with docutils-0.13?

[fbissey]

Thinking more about it, sphinx 1.5.2 should work with both, so we can do sphinx first then docutils afterwards.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6a61a49

Doc fixes for Sphinx-1.5

[jhpalmieri]

The line

    import ssl

from sphinx/util/requests.py is going to cause problems on systems without SSL (like my Mac, until I ran make openssl and sage -f python2).

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​04fe397	Upgrade to Sphinx 1.5.3
​33e4007	Doc fixes for Sphinx 1.5.x

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9333e9a

Docbuild fixes for Sphinx 1.5.x

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d8cdc76	Upgrade to Sphinx 1.5.3
​6c37777	Docbuild fixes for Sphinx 1.5.x

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​879c401	Upgrade to Sphinx 1.5.3
​78e6718	Docbuild fixes for Sphinx 1.5.x

[jdemeyer]

This last version actually manages to compile the documentation.

[jhpalmieri]

What about the issue in 8? Is ssl going to become a dependency for building the Sage docs?

[fbissey]

Modulo 8 it works for me.

[jpflori]

I think SSl is going to be a Sage dependency soon anyway so that we can upgrade R.

[jpflori]

In fact, SSL is already a Sage dependency: #22189.

[fbissey]

That's a broad interpretation but I am ready to give this a positive review anyway.

[jdemeyer]

I'm not convinced by the openSSL story.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ced4920

Allow running Sphinx without SSL

[jhpalmieri]

This builds (and the docs build) for me even when import ssl fails in Sage's Python.

[jhpalmieri]

By the way, if I build the docs and then switch to this branch and build again, it fails, with the message

TypeError: ('__init__() takes exactly 4 arguments (1 given)', <class 'sphinx.environment.BuildEnvironment'>, ())

This suggests that #19882 might still be useful.

[hivert]

It seems that you removed the patch giving extra memory to LaTeX to compile tp PDF file the huge Sage doc. We should keep this somewhere. I'm waiting for the patchbot which apparently check the compiling of the PDF doc.

As far as I understand the ssl workaround is sound. And I confirm that the patch_domain_init function in src/sage_setup/docbuild/__init__.py should indeed be removed. So this looks good to me except:

• the huge tex problem above
• the patch in build/pkgs/sphinx/patches/environment.patch which deals with changing some config value.

[jhpalmieri]

I can't comment on the TeX memory problem because I can't get the PDF docs to build with this branch. The first one attempted is reference/manifolds, which very early produces an error

[docpdf] 
[docpdf] Package hyperref Message: Driver (autodetected): hpdftex.
[docpdf] 
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/hyperref/hpdftex.def
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/rerunfilecheck.sty))
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/hypcap.sty)
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/jknapltx/mathrsfs.sty)
[docpdf] ! Undefined control sequence.
[docpdf] l.48 \ifPDFTeX
[docpdf]

Trying again with the patch

src/doc/common/conf.py

@@ -308 +308 @@
 \usepackage{amssymb}
 \usepackage{textcomp}
 \usepackage{mathrsfs}
+\usepackage{iftex}
 
 % Only declare unicode characters when compiling with pdftex; E.g. japanese
 % tutorial does not use pdftex

[jhpalmieri]

Based on ​https://github.com/sphinx-doc/sphinx/issues/2727, for example, maybe we should consider using latex_engine to determine the preamble. Maybe for another ticket, if we can get the PDF docs to build without much hard work.

[jhpalmieri]

Okay, the PDF docs build for me after adding \usepackage{iftex} to the preamble, without any other changes to the current branch, and in particular without the "huge tex" changes from previous versions.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​67f3b42

Add missing patches back

[jdemeyer]

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found

So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

[jhpalmieri]

On my OS X machine, extractbb seems to be part of TeXLive 2016, and is just a symlink to xdvipdfmx:

$ ls -l /usr/local/texlive/2016/bin/universal-darwin/extractbb
lrwxr-xr-x  1 palmieri  wheel  9 Sep  6  2016 /usr/local/texlive/2016/bin/universal-darwin/extractbb@ -> xdvipdfmx

Maybe if you install some package of TeX extras, it will be there?

[jhpalmieri]

In any case, everything works with the current branch on OS X if

• I also apply the patch in 28.
• and I run LC_ALL=C make doc-pdf, as suggested by Volker ​on sage-release.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f20f00a

Use TeX package iftex

[jdemeyer]

I think the LC_ALL issue is orthogonal to this ticket, so I didn't do anything for that. I did apply the patch from 28

[jhpalmieri]

Replying to jdemeyer:

I think the LC_ALL issue is orthogonal to this ticket, so I didn't do anything for that.

I agree.

[jhpalmieri]

I am happy with this, but it needs testing on platforms other than OS X.

[fbissey]

Replying to jdemeyer:

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found

So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

I should check that I can build it now. I changed system so I am starting from a cleaner slate. I used to have problem with russian documentation because of tex fonts problems. According to my ebuild I need the following from tex

L10N="ca de en fr hu it ja pt ru tr" USE=extra emerge -1Nv texlive

I usually do

VARTEX=$somelocations make doc-pdf

Gentoo doesn't install "unfolded" russian (and japanese?) tex fonts. If you don't specify VARTEX, it may try to do it in a system location.

[fbissey]

Japanese pdf doc build OK here with this ticket as is (a vanilla build rather than a sage-on-gentoo one). Cannot quite read it but the "a_tour_of_sage.pdf" looks legit compared to the english one.

[strogdon]

I have Gentoo here and all pdfs build with this ticket. I have installed

[I] dev-texlive/texlive-langcjk
[I] dev-texlive/texlive-langcyrillic
[I] dev-texlive/texlive-langenglish
[I] dev-texlive/texlive-langeuropean
[I] dev-texlive/texlive-langfrench
[I] dev-texlive/texlive-langgerman
[I] dev-texlive/texlive-langitalian
[I] dev-texlive/texlive-langjapanese
[I] dev-texlive/texlive-langportuguese
[I] dev-texlive/texlive-langspanish

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Replying to jdemeyer:

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found

So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

[fbissey]

Replying to strogdon:

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

[strogdon]

Replying to fbissey:

Replying to strogdon:

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

This was with vanilla and not s-o-g. I just did make doc-pdf. In doing that some japanese fonts could not be generated. I uncommented the f cid-x.map line although it could have been f ckx.map in order to get the build to complete.

[strogdon]

Replying to strogdon:

Replying to fbissey:

Replying to strogdon:

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

This was with vanilla and not s-o-g. I just did make doc-pdf. In doing that some japanese fonts could not be generated. I uncommented the f cid-x.map line although it could have been f ckx.map in order to get the build to complete.

It was %f cid-x.map that I had to uncomment.

[fbissey]

Still commented here, and yet I completed the build of the pdf documentation succesfully

%% Put additional fontmap files here (usually for Type0 fonts)
%f  cid-x.map

% the following file is generated by updmap(-sys) from the 
% KanjiMap entries in the updmap.cfg file.
f kanjix.map
% minimal example for Chinese and Korean users
% improvements please to tex-live@tug.org
f ckx.map

[strogdon]

Replying to fbissey:

Still commented here, and yet I completed the build of the pdf documentation succesfully

%% Put additional fontmap files here (usually for Type0 fonts)
%f  cid-x.map

% the following file is generated by updmap(-sys) from the 
% KanjiMap entries in the updmap.cfg file.
f kanjix.map
% minimal example for Chinese and Korean users
% improvements please to tex-live@tug.org
f ckx.map

I don't think there is an issue. I'm redoing things. It will take a while. Things are quite slow.

[strogdon]

OK, with f cid-x.map commented out I get

mktexpk --mfmode / --bdpi 600 --mag 2+180/600 --dpi 1380 gbm

in ja/tutorial/missfont.log when building ja/tutorial/tutorial-jp.pdf and I see

[docpdf] > <./a2tag.png> <./1.pngtutorial-jp.dvi -> tutorial-jp.pdf
[docpdf] > <./ab2tag.png>][1 [277
[docpdf] kpathsea: Running mktexpk --mfmode / --bdpi 600 --mag 2+180/600 --dpi 1380 gbm

...

[docpdf] Underfull \hbox (badness 10000) in paragraph at lines 22578--22581
[docpdf] []\T1/ptm/m/n/10 For de-tails, see \T1/pcr/m/n/10 ModulesWithBasis.ParentMethod
[docpdf] s.module_morphism() \T1/ptm/m/n/10 (and also \T1/pcr/m/n/10 sage.
[docpdf] [327] [328] [329] [330] [331] [332] [333] [334] [335] [336] [337] [338]gsftopk: fatal: map file `psfonts.cmz' not found.
[docpdf] 
[docpdf] Underfull \hbox (badness 7888) in paragraph at lines 23459--23467
[docpdf] []\T1/ptm/m/n/10 Again I rec-om-mend this [][]$http : / / www . scipy . org / W
[docpdf] iki / Documentation ? action = AttachFile & do = get & target = scipy _
[docpdf] [339] [340] [341] [342] [343]mktexpk: don't know how to create bitmap font for gbm.
[docpdf] mktexpk: perhaps gbm is missing from the map file.
[docpdf] kpathsea: Appending font creation commands to missfont.log.
[docpdf] 
[docpdf] dvipdfmx:warning: Could not locate a virtual/physical font for TFM "gbm".
[docpdf] dvipdfmx:warning: >> There are no valid font mapping entry for this font.
[docpdf] dvipdfmx:warning: >> Font file name "gbm" was assumed but failed to locate that font.
[docpdf] dvipdfmx:fatal: Cannot proceed without .vf or "physical" font for PDF output...
[docpdf] 
[docpdf] Output file removed.
[docpdf] make[2]: *** [Makefile:35: all-pdf-ja] Error 1
[docpdf] make[2]: Leaving directory '/64bitdev/storage/sage-git_develop/sage/local/share/doc/sage/latex/ja/tutorial'

The same missfont.log file is in ja/a_tour_of_sage. If I uncomment %f cid-x.map then everything builds. I'm not sure what's up but I can make the pdfs to build. I guess it's possible that since I didn't do

L10N="ca de en fr hu it ja pt ru tr" USE=extra emerge -1Nv texlive

I manually installed the packages - that that could be the difference? The only difference on my system is that app-text/texlive-2015 would be rebuilt with L10N="ca* de* en fr hu* it* ja* pt* ru* tr*. But all the extra languages are installed.

[fbissey]

Since I had to build anew I also needed USE="cjk xetex", but you would have been prompted for those I am sure. On the other hand I found /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm on my system and it belongs to

$ equery b /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm
 * Searching for /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm ... 
dev-texlive/texlive-langjapanese-2015 (/usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm)

Can you check that you have that?

[fbissey]

On another note:

  611  rm build/pkgs/sphinx/patches/latex_memory.patch 
  612  ./sage -f sphinx
  613  make doc-clean
  614  MAKE="make -j8" make doc-pdf

So I removed the latex_memory patch from sphinx, rebuild and the pdf doc build fine. So, at least on my system, it is unnecessary.

[strogdon]

Replying to fbissey:

Since I had to build anew I also needed USE="cjk xetex", but you would have been prompted for those I am sure. On the other hand I found /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm on my system and it belongs to

$ equery b /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm
 * Searching for /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm ... 
dev-texlive/texlive-langjapanese-2015 (/usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm)

Can you check that you have that?

Yes, this file is on my system.

[fbissey]

Hum... I don't any of the stuff you quote in my log. I am wondering if it is because I have fonts (truetype not tex) that have japanase character sets on my system.

[I] media-fonts/arphicfonts (0.2.20080216.1-r2@14/02/17): Chinese TrueType Arphic Fonts
[I] media-fonts/bitstream-cyberbit (2.0@14/02/17): Cyberbit Unicode (including CJK) font
[I] media-fonts/corefonts (1-r7@14/02/17): Microsoft's TrueType core fonts
[I] media-fonts/croscorefonts (1.23.0@14/02/17): Open licensed fonts metrically compatible with MS corefonts
[I] media-fonts/dejavu (2.35@30/12/16): DejaVu fonts, bitstream vera with ISO-8859-2 characters
[I] media-fonts/droid (113-r4@14/02/17): Font family from Google's Android project
[I] media-fonts/encodings (1.0.4@30/12/16): X.Org font encodings
[I] media-fonts/font-adobe-100dpi (1.0.3@14/02/17): X.Org Adobe bitmap fonts
[I] media-fonts/font-adobe-75dpi (1.0.3@14/02/17): X.Org Adobe bitmap fonts
[I] media-fonts/font-adobe-utopia-100dpi (1.0.4@14/02/17): X.Org Adobe Utopia bitmap fonts
[I] media-fonts/font-alias (1.0.3-r1@14/02/17): X.Org font aliases
[I] media-fonts/font-bh-lucidatypewriter-100dpi (1.0.3@14/02/17): X.Org Bigelow & Holmes Lucida bitmap fonts
[I] media-fonts/font-misc-misc (1.1.2@14/02/17): X.Org miscellaneous fonts
[I] media-fonts/font-util (1.3.1@30/12/16): X.Org font utilities
[I] media-fonts/hack (2.020@30/12/16): A typeface designed for source code
[I] media-fonts/ipamonafont (1.0.8@14/02/17): Hacked version of IPA fonts, which is suitable for browsing 2ch
[I] media-fonts/ja-ipafonts (003.02-r1@14/02/17): Japanese TrueType fonts developed by IPA (Information-technology Promotion Agency, Japan)
[I] media-fonts/liberation-fonts (2.00.1-r1@30/12/16): A Helvetica/Times/Courier replacement TrueType font set, courtesy of Red Hat
[I] media-fonts/libertine (5.3.0.20120702-r2@16/01/17): Fonts from the Linux Libertine Open Fonts Project
[I] media-fonts/noto (20160305-r1@30/12/16): Google's font family that aims to support all the world's languages
[I] media-fonts/stix-fonts (1.1.1@14/02/17): Comprehensive OpenType font set of mathematical symbols and alphabets
[I] media-fonts/urw-fonts (2.4.9@30/12/16): free good quality fonts gpl'd by URW++
[I] media-fonts/wqy-microhei (0.2.0_beta@14/02/17): A droid derived Sans-Serif style CJK font
[I] media-fonts/wqy-zenhei (0.9.46@14/02/17): WenQuanYi Hei-Ti Style (sans-serif) Chinese outline font

[strogdon]

Replying to fbissey:

Hum... I don't any of the stuff you quote in my log. I am wondering if it is because I have fonts (truetype not tex) that have japanase character sets on my system.

It appears that, for whatever reason, I had a

~/.texlive/texmf-var/fonts/map/dvipdfmx/updmap/kanjix.map

file that was getting in the way of the build. Moving the texmf-var folder out of the way and I appear to now have a sane build of the pdf docs.

[jdemeyer]

Let me remind everybody that this ticket is about upgrading from Sphinx 1.4 to 1.5. Discussions about building docs in general can be good to have, but they do not belong on this ticket (unless there is a regression, i.e. something worked with Sphinx 1.4 but no longer works).

[fbissey]

Sorry for the noise Jeroen. We wanted to be sure we had the dependencies right.

My own experience here is that: 1) doc, including pdf and japanese pdf, in particular, build fine 2) it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

I am happy for this ticket to be sent to the bots once the offending patch is removed - or someone shows it is needed in some cases.

[jdemeyer]

Replying to fbissey:

it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

[fbissey]

Replying to jdemeyer:

Replying to fbissey:

it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

I am happy with that, we may well discover that is needed on some platforms. Putting it in positive review.

[jdemeyer]

Replying to jdemeyer:

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

See #22660.