Opened 5 years ago

Closed 5 years ago

## #22252 closed enhancement (fixed)

Reported by: Owned by: jdemeyer major sage-7.6 packages: standard days85 fbissey Jeroen Demeyer François Bissey, John Palmieri Fixed upstream, but not in a stable release. f20f00a f20f00a5ce6168f34f51845c0698e4419506ea12

The newer version of Sphinx needs requests.

Tarballs:

We add a patch to Sphinx to allow running without SSL: https://github.com/sphinx-doc/sphinx/pull/3554

### comment:1 Changed 5 years ago by fbissey

Should we update docutils to 0.13 at the same time? sphinx 1.5.2 as all the fix to work with it as far as I know, can be in a separate ticket but it is better if they are done at the same time given the comments in https://github.com/cschwan/sage-on-gentoo/issues/451

### comment:2 Changed 5 years ago by jdemeyer

I would say: let's do whatever is easiest. So the question is: is it easier to make Sage and Sphinx-1.5.2 work with docutils-0.12 or with docutils-0.13?

### comment:3 Changed 5 years ago by fbissey

Thinking more about it, sphinx 1.5.2 should work with both, so we can do sphinx first then docutils afterwards.

### comment:4 Changed 5 years ago by jdemeyer

• Description modified (diff)

### comment:6 Changed 5 years ago by git

• Commit set to 6a61a49ecfede1b9648a30361fa90df52e94fb15

Branch pushed to git repo; I updated commit sha1. New commits:

 ​6a61a49 Doc fixes for Sphinx-1.5

### comment:7 Changed 5 years ago by jdemeyer

• Authors set to Jeroen Demeyer
• Description modified (diff)
• Summary changed from Upgrade to Sphinx 1.5.2 to Upgrade to Sphinx 1.5.x

### comment:8 Changed 5 years ago by jhpalmieri

The line

    import ssl


from sphinx/util/requests.py is going to cause problems on systems without SSL (like my Mac, until I ran make openssl and sage -f python2).

### comment:9 Changed 5 years ago by git

• Commit changed from 6a61a49ecfede1b9648a30361fa90df52e94fb15 to 33e4007ace5c9b9430131bee4567e4c4e8d7a6c3

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​04fe397 Upgrade to Sphinx 1.5.3 ​33e4007 Doc fixes for Sphinx 1.5.x

### comment:10 Changed 5 years ago by git

• Commit changed from 33e4007ace5c9b9430131bee4567e4c4e8d7a6c3 to 9333e9a348c27ba7db5d12312b9d707e8f08abcf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​9333e9a Docbuild fixes for Sphinx 1.5.x

### comment:11 Changed 5 years ago by git

• Commit changed from 9333e9a348c27ba7db5d12312b9d707e8f08abcf to 6c377771ac0410945c134d9e7cdc5bec0c46870a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​d8cdc76 Upgrade to Sphinx 1.5.3 ​6c37777 Docbuild fixes for Sphinx 1.5.x

### comment:13 Changed 5 years ago by git

• Commit changed from 6c377771ac0410945c134d9e7cdc5bec0c46870a to 78e671810360d3e02ec1e0fa6643a0e63f727f39

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​879c401 Upgrade to Sphinx 1.5.3 ​78e6718 Docbuild fixes for Sphinx 1.5.x

### comment:14 Changed 5 years ago by jdemeyer

This last version actually manages to compile the documentation.

### comment:15 Changed 5 years ago by jdemeyer

• Status changed from new to needs_review

### comment:16 Changed 5 years ago by jhpalmieri

What about the issue in 8? Is ssl going to become a dependency for building the Sage docs?

### comment:17 Changed 5 years ago by fbissey

Modulo 8 it works for me.

### comment:18 Changed 5 years ago by jpflori

I think SSl is going to be a Sage dependency soon anyway so that we can upgrade R.

### comment:19 Changed 5 years ago by jpflori

In fact, SSL is already a Sage dependency: #22189.

### comment:20 Changed 5 years ago by fbissey

• Reviewers set to François Bissey
• Status changed from needs_review to positive_review

That's a broad interpretation but I am ready to give this a positive review anyway.

### comment:21 Changed 5 years ago by jdemeyer

• Status changed from positive_review to needs_work

### comment:22 Changed 5 years ago by jdemeyer

I'm not convinced by the openSSL story.

### comment:23 Changed 5 years ago by git

• Commit changed from 78e671810360d3e02ec1e0fa6643a0e63f727f39 to ced49200aefa25b91ff96bb2cbc03fe418ec425b

Branch pushed to git repo; I updated commit sha1. New commits:

 ​ced4920 Allow running Sphinx without SSL

### comment:24 Changed 5 years ago by jdemeyer

• Description modified (diff)
• Report Upstream changed from N/A to Reported upstream. No feedback yet.
• Status changed from needs_work to needs_review

### comment:25 Changed 5 years ago by jhpalmieri

• Reviewers changed from François Bissey to François Bissey, John Palmieri
• Status changed from needs_review to positive_review

This builds (and the docs build) for me even when import ssl fails in Sage's Python.

Last edited 5 years ago by jhpalmieri (previous) (diff)

### comment:26 Changed 5 years ago by jhpalmieri

By the way, if I build the docs and then switch to this branch and build again, it fails, with the message

TypeError: ('__init__() takes exactly 4 arguments (1 given)', <class 'sphinx.environment.BuildEnvironment'>, ())


This suggests that #19882 might still be useful.

### comment:27 Changed 5 years ago by hivert

It seems that you removed the patch giving extra memory to LaTeX to compile tp PDF file the huge Sage doc. We should keep this somewhere. I'm waiting for the patchbot which apparently check the compiling of the PDF doc.

As far as I understand the ssl workaround is sound. And I confirm that the patch_domain_init function in src/sage_setup/docbuild/__init__.py should indeed be removed. So this looks good to me except:

• the huge tex problem above
• the patch in build/pkgs/sphinx/patches/environment.patch which deals with changing some config value.

### comment:28 Changed 5 years ago by jhpalmieri

• Status changed from positive_review to needs_work

I can't comment on the TeX memory problem because I can't get the PDF docs to build with this branch. The first one attempted is reference/manifolds, which very early produces an error

[docpdf]
[docpdf] Package hyperref Message: Driver (autodetected): hpdftex.
[docpdf]
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/hyperref/hpdftex.def
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/rerunfilecheck.sty))
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/hypcap.sty)
[docpdf] (/usr/local/texlive/2016/texmf-dist/tex/latex/jknapltx/mathrsfs.sty)
[docpdf] ! Undefined control sequence.
[docpdf] l.48 \ifPDFTeX
[docpdf]


Trying again with the patch

• ## src/doc/common/conf.py

diff --git a/src/doc/common/conf.py b/src/doc/common/conf.py
index 564407b..7ec52df 100644
 a latex_elements['preamble'] = r""" \usepackage{amssymb} \usepackage{textcomp} \usepackage{mathrsfs} \usepackage{iftex} % Only declare unicode characters when compiling with pdftex; E.g. japanese % tutorial does not use pdftex

### comment:29 Changed 5 years ago by jhpalmieri

Based on https://github.com/sphinx-doc/sphinx/issues/2727, for example, maybe we should consider using latex_engine to determine the preamble. Maybe for another ticket, if we can get the PDF docs to build without much hard work.

### comment:30 Changed 5 years ago by jhpalmieri

Okay, the PDF docs build for me after adding \usepackage{iftex} to the preamble, without any other changes to the current branch, and in particular without the "huge tex" changes from previous versions.

### comment:31 Changed 5 years ago by jdemeyer

• Report Upstream changed from Reported upstream. No feedback yet. to Fixed upstream, but not in a stable release.

### comment:32 Changed 5 years ago by git

• Commit changed from ced49200aefa25b91ff96bb2cbc03fe418ec425b to 67f3b42a18bb98b54da28b5b323d2f45af6c7374

Branch pushed to git repo; I updated commit sha1. New commits:

 ​67f3b42 Add missing patches back

### comment:33 follow-ups: ↓ 41 ↓ 43 Changed 5 years ago by jdemeyer

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found


So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

Last edited 5 years ago by jdemeyer (previous) (diff)

### comment:34 Changed 5 years ago by jhpalmieri

On my OS X machine, extractbb seems to be part of TeXLive 2016, and is just a symlink to xdvipdfmx:

$ls -l /usr/local/texlive/2016/bin/universal-darwin/extractbb lrwxr-xr-x 1 palmieri wheel 9 Sep 6 2016 /usr/local/texlive/2016/bin/universal-darwin/extractbb@ -> xdvipdfmx  Maybe if you install some package of TeX extras, it will be there? ### comment:35 Changed 5 years ago by jhpalmieri In any case, everything works with the current branch on OS X if • I also apply the patch in 28. • and I run LC_ALL=C make doc-pdf, as suggested by Volker on sage-release. ### comment:36 Changed 5 years ago by git • Commit changed from 67f3b42a18bb98b54da28b5b323d2f45af6c7374 to f20f00a5ce6168f34f51845c0698e4419506ea12 Branch pushed to git repo; I updated commit sha1. New commits:  ​f20f00a Use TeX package iftex ### comment:37 follow-up: ↓ 39 Changed 5 years ago by jdemeyer I think the LC_ALL issue is orthogonal to this ticket, so I didn't do anything for that. I did apply the patch from 28 ### comment:38 Changed 5 years ago by jdemeyer • Status changed from needs_work to needs_review ### comment:39 in reply to: ↑ 37 Changed 5 years ago by jhpalmieri Replying to jdemeyer: I think the LC_ALL issue is orthogonal to this ticket, so I didn't do anything for that. I agree. ### comment:40 Changed 5 years ago by jhpalmieri I am happy with this, but it needs testing on platforms other than OS X. ### comment:41 in reply to: ↑ 33 Changed 5 years ago by fbissey Replying to jdemeyer: On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch: [docpdf] /bin/sh: extractbb: command not found  So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it. So I would appreciate if somebody else could finish this ticket. I should check that I can build it now. I changed system so I am starting from a cleaner slate. I used to have problem with russian documentation because of tex fonts problems. According to my ebuild I need the following from tex L10N="ca de en fr hu it ja pt ru tr" USE=extra emerge -1Nv texlive  I usually do VARTEX=$somelocations make doc-pdf


Gentoo doesn't install "unfolded" russian (and japanese?) tex fonts. If you don't specify VARTEX, it may try to do it in a system location.

### comment:42 Changed 5 years ago by fbissey

Japanese pdf doc build OK here with this ticket as is (a vanilla build rather than a sage-on-gentoo one). Cannot quite read it but the "a_tour_of_sage.pdf" looks legit compared to the english one.

### comment:43 in reply to: ↑ 33 ; follow-up: ↓ 44 Changed 5 years ago by strogdon

I have Gentoo here and all pdfs build with this ticket. I have installed

[I] dev-texlive/texlive-langcjk
[I] dev-texlive/texlive-langcyrillic
[I] dev-texlive/texlive-langenglish
[I] dev-texlive/texlive-langeuropean
[I] dev-texlive/texlive-langfrench
[I] dev-texlive/texlive-langgerman
[I] dev-texlive/texlive-langitalian
[I] dev-texlive/texlive-langjapanese
[I] dev-texlive/texlive-langportuguese
[I] dev-texlive/texlive-langspanish


/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map


in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found


So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

### comment:44 in reply to: ↑ 43 ; follow-up: ↓ 45 Changed 5 years ago by fbissey

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map


in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

### comment:45 in reply to: ↑ 44 ; follow-up: ↓ 46 Changed 5 years ago by strogdon

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map


in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

This was with vanilla and not s-o-g. I just did make doc-pdf. In doing that some japanese fonts could not be generated. I uncommented the f cid-x.map line although it could have been f ckx.map in order to get the build to complete.

Last edited 5 years ago by strogdon (previous) (diff)

### comment:46 in reply to: ↑ 45 Changed 5 years ago by strogdon

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map


in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

This was with vanilla and not s-o-g. I just did make doc-pdf. In doing that some japanese fonts could not be generated. I uncommented the f cid-x.map line although it could have been f ckx.map in order to get the build to complete.

It was %f cid-x.map  that I had to uncomment.

Last edited 5 years ago by strogdon (previous) (diff)

### comment:47 follow-up: ↓ 48 Changed 5 years ago by fbissey

Still commented here, and yet I completed the build of the pdf documentation succesfully

%% Put additional fontmap files here (usually for Type0 fonts)
%f  cid-x.map

% the following file is generated by updmap(-sys) from the
% KanjiMap entries in the updmap.cfg file.
f kanjix.map
% minimal example for Chinese and Korean users
f ckx.map


### comment:48 in reply to: ↑ 47 Changed 5 years ago by strogdon

Still commented here, and yet I completed the build of the pdf documentation succesfully

%% Put additional fontmap files here (usually for Type0 fonts)
%f  cid-x.map

% the following file is generated by updmap(-sys) from the
% KanjiMap entries in the updmap.cfg file.
f kanjix.map
% minimal example for Chinese and Korean users
f ckx.map


I don't think there is an issue. I'm redoing things. It will take a while. Things are quite slow.

### comment:49 Changed 5 years ago by strogdon

OK, with f cid-x.map commented out I get

mktexpk --mfmode / --bdpi 600 --mag 2+180/600 --dpi 1380 gbm


in ja/tutorial/missfont.log when building ja/tutorial/tutorial-jp.pdf and I see

[docpdf] > <./a2tag.png> <./1.pngtutorial-jp.dvi -> tutorial-jp.pdf
[docpdf] > <./ab2tag.png>][1 [277
[docpdf] kpathsea: Running mktexpk --mfmode / --bdpi 600 --mag 2+180/600 --dpi 1380 gbm

...

[docpdf] Underfull \hbox (badness 10000) in paragraph at lines 22578--22581
[docpdf] []\T1/ptm/m/n/10 For de-tails, see \T1/pcr/m/n/10 ModulesWithBasis.ParentMethod
[docpdf] s.module_morphism() \T1/ptm/m/n/10 (and also \T1/pcr/m/n/10 sage.
[docpdf] [327] [328] [329] [330] [331] [332] [333] [334] [335] [336] [337] [338]gsftopk: fatal: map file psfonts.cmz' not found.
[docpdf]
[docpdf] Underfull \hbox (badness 7888) in paragraph at lines 23459--23467
[docpdf] []\T1/ptm/m/n/10 Again I rec-om-mend this [][]$http : / / www . scipy . org / W [docpdf] iki / Documentation ? action = AttachFile & do = get & target = scipy _ [docpdf] [339] [340] [341] [342] [343]mktexpk: don't know how to create bitmap font for gbm. [docpdf] mktexpk: perhaps gbm is missing from the map file. [docpdf] kpathsea: Appending font creation commands to missfont.log. [docpdf] [docpdf] dvipdfmx:warning: Could not locate a virtual/physical font for TFM "gbm". [docpdf] dvipdfmx:warning: >> There are no valid font mapping entry for this font. [docpdf] dvipdfmx:warning: >> Font file name "gbm" was assumed but failed to locate that font. [docpdf] dvipdfmx:fatal: Cannot proceed without .vf or "physical" font for PDF output... [docpdf] [docpdf] Output file removed. [docpdf] make[2]: *** [Makefile:35: all-pdf-ja] Error 1 [docpdf] make[2]: Leaving directory '/64bitdev/storage/sage-git_develop/sage/local/share/doc/sage/latex/ja/tutorial'  The same missfont.log file is in ja/a_tour_of_sage. If I uncomment %f cid-x.map then everything builds. I'm not sure what's up but I can make the pdfs to build. I guess it's possible that since I didn't do L10N="ca de en fr hu it ja pt ru tr" USE=extra emerge -1Nv texlive  I manually installed the packages - that that could be the difference? The only difference on my system is that app-text/texlive-2015 would be rebuilt with L10N="ca* de* en fr hu* it* ja* pt* ru* tr*. But all the extra languages are installed. ### comment:50 follow-up: ↓ 52 Changed 5 years ago by fbissey Since I had to build anew I also needed USE="cjk xetex", but you would have been prompted for those I am sure. On the other hand I found /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm on my system and it belongs to $ equery b /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm
* Searching for /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm ...
dev-texlive/texlive-langjapanese-2015 (/usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm)


Can you check that you have that?

### comment:51 Changed 5 years ago by fbissey

On another note:

  611  rm build/pkgs/sphinx/patches/latex_memory.patch
612  ./sage -f sphinx
613  make doc-clean
614  MAKE="make -j8" make doc-pdf


So I removed the latex_memory patch from sphinx, rebuild and the pdf doc build fine. So, at least on my system, it is unnecessary.

### comment:52 in reply to: ↑ 50 Changed 5 years ago by strogdon

Since I had to build anew I also needed USE="cjk xetex", but you would have been prompted for those I am sure. On the other hand I found /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm on my system and it belongs to

\$ equery b /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm
* Searching for /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm ...
dev-texlive/texlive-langjapanese-2015 (/usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm)


Can you check that you have that?

Yes, this file is on my system.

### comment:53 follow-up: ↓ 54 Changed 5 years ago by fbissey

Hum... I don't any of the stuff you quote in my log. I am wondering if it is because I have fonts (truetype not tex) that have japanase character sets on my system.

[I] media-fonts/arphicfonts (0.2.20080216.1-r2@14/02/17): Chinese TrueType Arphic Fonts
[I] media-fonts/bitstream-cyberbit (2.0@14/02/17): Cyberbit Unicode (including CJK) font
[I] media-fonts/corefonts (1-r7@14/02/17): Microsoft's TrueType core fonts
[I] media-fonts/croscorefonts (1.23.0@14/02/17): Open licensed fonts metrically compatible with MS corefonts
[I] media-fonts/dejavu (2.35@30/12/16): DejaVu fonts, bitstream vera with ISO-8859-2 characters
[I] media-fonts/droid (113-r4@14/02/17): Font family from Google's Android project
[I] media-fonts/encodings (1.0.4@30/12/16): X.Org font encodings
[I] media-fonts/font-alias (1.0.3-r1@14/02/17): X.Org font aliases
[I] media-fonts/font-bh-lucidatypewriter-100dpi (1.0.3@14/02/17): X.Org Bigelow & Holmes Lucida bitmap fonts
[I] media-fonts/font-misc-misc (1.1.2@14/02/17): X.Org miscellaneous fonts
[I] media-fonts/font-util (1.3.1@30/12/16): X.Org font utilities
[I] media-fonts/hack (2.020@30/12/16): A typeface designed for source code
[I] media-fonts/ipamonafont (1.0.8@14/02/17): Hacked version of IPA fonts, which is suitable for browsing 2ch
[I] media-fonts/ja-ipafonts (003.02-r1@14/02/17): Japanese TrueType fonts developed by IPA (Information-technology Promotion Agency, Japan)
[I] media-fonts/liberation-fonts (2.00.1-r1@30/12/16): A Helvetica/Times/Courier replacement TrueType font set, courtesy of Red Hat
[I] media-fonts/libertine (5.3.0.20120702-r2@16/01/17): Fonts from the Linux Libertine Open Fonts Project
[I] media-fonts/noto (20160305-r1@30/12/16): Google's font family that aims to support all the world's languages
[I] media-fonts/stix-fonts (1.1.1@14/02/17): Comprehensive OpenType font set of mathematical symbols and alphabets
[I] media-fonts/urw-fonts (2.4.9@30/12/16): free good quality fonts gpl'd by URW++
[I] media-fonts/wqy-microhei (0.2.0_beta@14/02/17): A droid derived Sans-Serif style CJK font
[I] media-fonts/wqy-zenhei (0.9.46@14/02/17): WenQuanYi Hei-Ti Style (sans-serif) Chinese outline font


### comment:54 in reply to: ↑ 53 Changed 5 years ago by strogdon

Hum... I don't any of the stuff you quote in my log. I am wondering if it is because I have fonts (truetype not tex) that have japanase character sets on my system.

It appears that, for whatever reason, I had a

~/.texlive/texmf-var/fonts/map/dvipdfmx/updmap/kanjix.map


file that was getting in the way of the build. Moving the texmf-var folder out of the way and I appear to now have a sane build of the pdf docs.

### comment:55 Changed 5 years ago by jdemeyer

Let me remind everybody that this ticket is about upgrading from Sphinx 1.4 to 1.5. Discussions about building docs in general can be good to have, but they do not belong on this ticket (unless there is a regression, i.e. something worked with Sphinx 1.4 but no longer works).

### comment:56 follow-up: ↓ 57 Changed 5 years ago by fbissey

Sorry for the noise Jeroen. We wanted to be sure we had the dependencies right.

My own experience here is that: 1) doc, including pdf and japanese pdf, in particular, build fine 2) it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

I am happy for this ticket to be sent to the bots once the offending patch is removed - or someone shows it is needed in some cases.

### comment:57 in reply to: ↑ 56 ; follow-ups: ↓ 58 ↓ 59 Changed 5 years ago by jdemeyer

it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

### comment:58 in reply to: ↑ 57 Changed 5 years ago by fbissey

• Status changed from needs_review to positive_review

it does so with or without latex_memory.patch` so I guess we can drop it - unless someone has a different experience.

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

I am happy with that, we may well discover that is needed on some platforms. Putting it in positive review.