cvxopt 1.1.8 fails to build with SAGE_SPKG_INSTALL_DOCS=yes

[strogdon]

This is because the pre-built HTML were removed, and building docs using sphinx was off.

[leif]

Diff between the .p0 and the .p1. For reference / review only.

[leif]

Replying to leif:

• Run more tests in spkg-check (i.e., those in src/examples/book/ as well).

P.S.: Tested with SAGE_CHECK=yes on Ubuntu 10.04.4 x86 and x86_64 (FSF GCC 4.7.2 and 4.8.0, respectively).

[dimpase]

tested with SAGE_CHECK=yes on Sage 5.9 OSX10.6.8 (native lapack/blas), all good.

I understand that cvxopt docs not integrated into Sage Reference manual, but are installed in SAGE_LOCAL/share/. Wouldn't it be good to add a link somewhere in Reference manual to SAGE_LOCAL/share/ ? Otherwise it's quite hard to find, unless you know something either not really documented, IMHO.

How does one force doc building/installing? I tried

SAGE_SPKG_INSTALL_DOCS=yes sage -f cvxopt-1.1.6.p1.spkg

to no avail.

[leif]

Replying to dimpase:

I understand that cvxopt docs not integrated into Sage Reference manual, but are installed in SAGE_LOCAL/share/. Wouldn't it be good to add a link somewhere in Reference manual to SAGE_LOCAL/share/ ? Otherwise it's quite hard to find, unless you know something either not really documented, IMHO.

How does one force doc building/installing? I tried

SAGE_SPKG_INSTALL_DOCS=yes sage -f cvxopt-1.1.6.p1.spkg

to no avail.

? They're in $SAGE_ROOT/local/share/doc/cvxopt/html/, indeed probably hard to find ...

[dimpase]

Replying to leif:

Replying to dimpase:

I understand that cvxopt docs not integrated into Sage Reference manual, but are installed in SAGE_LOCAL/share/. Wouldn't it be good to add a link somewhere in Reference manual to SAGE_LOCAL/share/ ? Otherwise it's quite hard to find, unless you know something either not really documented, IMHO.

How does one force doc building/installing? I tried

SAGE_SPKG_INSTALL_DOCS=yes sage -f cvxopt-1.1.6.p1.spkg

to no avail.

? They're in $SAGE_ROOT/local/share/doc/cvxopt/html/, indeed probably hard to find ...

I looked at the install log, and it didn't say anything :-) Please add a message saying that docs were installed (and where to)...

[dimpase]

Replying to dimpase:

Wouldn't it be good to add a link somewhere in Reference manual to SAGE_LOCAL/share/ ?

rather, to SAGE_LOCAL/share/doc/, The main problem is how to add a relative link. One can in principle do ../../../../...../local/share/doc/ somewhere in doc/<LANG>/website/templates/index.html, but I'm not sure it's a good idea (IMHO the idea to put any docs in $SAGE_LOCAL/share/doc/ is not so good... )
Perhaps at least there should be a symbolic link sage/doc/local pointing there, although I'm not sure how well this works with webservers.
If this is OK then I can create a patch (hmm, but how? Does hg understand symbolic links? Or it should be in some script, this link creation?)

[dimpase]

Harald, as this is www-related (docs structure), could you comment please?

[schilly]

Uhm, I can't really follow you, sorry … also, the website is rather independent from what you might want to do inside Sage. Is there anything already existing that's similar to what you try to do here? Also, the documentation is usually maintained by minh, I've CCed him.

What I found is this dedicated page: ​http://www.sagemath.org/doc/numerical_sage/cvxopt.html

[vbraun]

The docs should link to upstream (​http://cvxopt.org) and not to a local file that isn't even installed unless you build with SAGE_SPKG_INSTALL_DOCS=yes. Also, the link in ​http://www.sagemath.org/doc/numerical_sage/cvxopt.html is broken.

[dimpase]

Replying to vbraun:

The docs should link to upstream (​http://cvxopt.org) and not to a local file that isn't even installed unless you build with SAGE_SPKG_INSTALL_DOCS=yes. Also, the link in ​http://www.sagemath.org/doc/numerical_sage/cvxopt.html is broken.

we can have SAGE_LOCAL/share/doc/cvxopt/index.html redirecting to cvxopt.org by default, and overwrite it with the local docs if SAGE_SPKG_INSTALL_DOCS=yes during the cvxopt installation.

But there is much more stuff there than just cvxopt docs. E.g. I have currently, on Sage 5.9,

NTL				ipython				ppl
cvxopt				mpfr				pwl
eclib				networkx-1.7.dev_20130506064154	sagetex

and again these docs are largely lost to a typical user. I wonder why are they (except cvxopt, that was explicitly installed with SAGE_SPKG_INSTALL_DOCS on) there in the first place. It looks like the other packages ignore the value of SAGE_SPKG_INSTALL_DOCS, no?

[dimpase]

Replying to schilly:

Uhm, I can't really follow you, sorry … also, the website is rather independent from what you might want to do inside Sage.

oops, I meant the setup when the sagenb server is run on a remote host (rather than on localhost); then the docs are served somehow via a web interface, isn't it?

Generally speaking, do Unix webservers allow symbolic links in the docs they serve?

[vbraun]

Any feature requests beyond installing cvxopt should go to a different ticket.

Some packages just install their docs by default. Unfortunately, correct usage of autotools often eludes upsteam, requiring manual copying of documentation if desired. The latter is only attempted if SAGE_SPKG_INSTALL_DOCS is set.

Apache disallows symlinks by default. Not that that would help with ​http://sagemath.org, there we serve only the Sage docs and not a complete Sage installation afaik.

[schilly]

Replying to dimpase:

oops, I meant the setup when the sagenb server is run on a remote host (rather than on localhost); then the docs are served somehow via a web interface, isn't it?

yes, but i don't know enough about sagenb/notebook server to give you any answer, sorry. there are also related issues, e.g. that the sagenb/notebook server doesn't show documentation in other languages – but that's in my eyes outside of the scope of this ticket.

Generally speaking, do Unix webservers allow symbolic links in the docs they serve?

yes and no. the documentation on the sage website is served as static file via apache. Following symlinks is explicitly enabled and it's basically just a copy/paste of the output files. What the notebook server does is different, because it serves the files via its own server – there is also this live-documentation served via /doc/live/…

So, in conclusion, there are (at least?) three independent things moving around:

• local installation of Sage
• Sage Documentation at sagemath.org/doc (that's where mvgnu and I can weight in, we also have our own script that updates sagemath.org/help.html)
• sagenb.org [and others] /doc/live/ <- and the menu that's in the help popup window.

[dimpase]

Replying to vbraun:

The docs should link to upstream (​http://cvxopt.org) and not to a local file that isn't even installed unless you build with SAGE_SPKG_INSTALL_DOCS=yes.

actually my idea about the main index.html is not listing individual packages, but just a general link to SAGE_LOCAL/share/doc for "more stuff"... OK, I will open a separate ticket, and set this one to positive review, as soon as my request for a message saying that docs were installed (and where to) in spkg-install is honoured.

PS. I've opened #14646 to deal with the main index.html, etc.

[leif]

Replying to dimpase:

OK, I will open a separate ticket, and set this one to positive review, as soon as my request for a message saying that docs were installed (and where to) in spkg-install is honoured.

I was actually first going to add that, but then did not for a couple of reasons:

• I hate absolute paths there (as have unfortunately been added back to (additional) messages in the "main" docbuilding, although the messages there are even worse, since they claim the documents had been built even upon errors), but was 100% sure at least some would complain about a relative path in the message.

• Most other packages don't.

• Then I'd also added a message in case SAGE_SPKG_INSTALL_DOCS!=yes, telling that the [HTML] docs have not been installed (and why), and how one can change that, which is pretty redundant, as that should be the same for all spkgs.

Orthogonal to that:

Should I delete previous (potentially outdated) docs regardless of SAGE_SPKG_INSTALL_DOCS, which could probably be considered unexpected behaviour? (E.g. if one reinstalls an spkg just with different flags, or in order to run its test suite, but forgets to set SAGE_SPKG_INSTALL_DOCS to the previous setting? Does anyone use that variable on a per-package basis? To me it seems a few people have it set in their shell rcs, and most people don't use it, probably just because they don't know about... B) )

[dimpase]

Replying to leif:

Replying to dimpase:

OK, I will open a separate ticket, and set this one to positive review, as soon as my request for a message saying that docs were installed (and where to) in spkg-install is honoured.

I was actually first going to add that, but then did not for a couple of reasons:

• I hate absolute paths there (as have unfortunately been added back to (additional) messages in the "main" docbuilding, although the messages there are even worse, since they claim the documents had been built even upon errors), but was 100% sure at least some would complain about a relative path in the message.

well, how about just saying "installed in SAGE_SHARE/doc" ?

• Most other packages don't.

fix them :–) (not on this ticket...)

• Then I'd also added a message in case SAGE_SPKG_INSTALL_DOCS!=yes, telling that the [HTML] docs have not been installed (and why), and how one can change that, which is pretty redundant, as that should be the same for all spkgs.

Orthogonal to that:

Should I delete previous (potentially outdated) docs regardless of SAGE_SPKG_INSTALL_DOCS, which could probably be considered unexpected behaviour? (E.g. if one reinstalls an spkg just with different flags, or in order to run its test suite, but forgets to set SAGE_SPKG_INSTALL_DOCS to the previous setting? Does anyone use that variable on a per-package basis? To me it seems a few people have it set in their shell rcs, and most people don't use it, probably just because they don't know about... B) )

if they are in "wrong" location, yes. Otherwise, that's probably too much trouble --- although I as proposed on #14646, there ought to be a default SAGE_SHARE/doc/cvxopt/index.html forwarding to cvxopt website, so, ideally, SAGE_SPKG_INSTALL_DOCS!=yes would mean putting this forwarding back in place... Well, even more ideally, this functionality shouldn't be on the spkg-install level, but above, at sage -i level.

[leif]

Replying to dimpase:

Replying to leif:

I was actually first going to add that, but then did not for a couple of reasons:

• I hate absolute paths there (as have unfortunately been added back to (additional) messages in the "main" docbuilding, although the messages there are even worse, since they claim the documents had been built even upon errors), but was 100% sure at least some would complain about a relative path in the message.

well, how about just saying "installed in SAGE_SHARE/doc" ?

That's kind of a relative path... [I would have used] ;-)

Orthogonal to that:

Should I delete previous (potentially outdated) docs regardless of SAGE_SPKG_INSTALL_DOCS, which could probably be considered unexpected behaviour? (E.g. if one reinstalls an spkg just with different flags, or in order to run its test suite, but forgets to set SAGE_SPKG_INSTALL_DOCS to the previous setting? Does anyone use that variable on a per-package basis? To me it seems a few people have it set in their shell rcs, and most people don't use it, probably just because they don't know about... B) )

if they are in "wrong" location, yes. Otherwise, that's probably too much trouble --- although I as proposed on #14646, there ought to be a default SAGE_SHARE/doc/cvxopt/index.html forwarding to cvxopt website, so, ideally, SAGE_SPKG_INSTALL_DOCS!=yes would mean putting this forwarding back in place... Well, even more ideally, this functionality shouldn't be on the spkg-install level, but above, at sage -i level.

So, should I move the rm -rf ... out of the if [ "$SAGE_SPKG_INSTALL_DOCS" = yes ] ... branch? (With a separate message on what's going on ... :-) )

[dimpase]

Replying to leif: right, go ahead :)

[mkoeppe]

This needs updating for the current (new-style package) 1.1.8p1.

[leif]

Replying to mkoeppe:

This needs updating for the current (new-style package) 1.1.8p1.

Yes, people keep updating packages without fixing long-standing issues.

[leif]

Let me add that I won't waste my time with rebasing more than three years old patches the next days.

[mkoeppe]

Right.

[dimpase]

IMHO the idea is that html docs should be built, and then installed, not just copied from provided html/* ?

[dimpase]

this is a minimal fix to docs being built/installed. Feel free to add more stuff, or just set it to positive review...

---

New commits:

​22ac42a

minimal fix to ensure that docs build/install

[mderickx]

Merge conflict with sage 8.1.beta3 needs to be resolved.

[fbissey]

From experience on the sphinx upgrade ticket, you may want to add typing along with sphinx. And it should be on the other side of the | since you don't need to rebuild cvxopt each time sphinx or typing is upgraded.

[fbissey]

You may have to depend on #23023 because of typing. The ticket is merged but not in a released beta yet.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​ba68e21	minimal fix to ensure that docs build/install
​2ea32c1	Upgrade to Sphinx 1.6.3
​81253e8	Merge branch 'develop' into sphinx16
​7bbfe54	conform typing to the new boiler plate.
​6930d06	Remove language specific targeting as upstream has removed at least the only one we are using.
​56bffcc	trac 23023:
​039c7f9	Merge branch 'develop' into sphinx-1.6.3
​c488967	Add typing to DOC_DEPENDENCIES
​83dd36e	Merge branch 'u/fbissey/sphinx1.6.3' of trac.sagemath.org:sage into cvxdocs

[dimpase]

rebased and merged with the branch on #23023, but I am getting now

[cvxopt-1.1.8.p2] pkg_resources.DistributionNotFound: The 'sphinxcontrib-websupport' distribution was not found and is required by Sphinx
[cvxopt-1.1.8.p2] make[2]: *** [Makefile:30: html] Error 1

if I try to build with SAGE_SPKG_INSTALL_DOCS=yes

Any idea what sphinxcontrib-websupport is all about? Is it another sphinx package we need to distribute, but we don't?

[fbissey]

Replying to dimpase:

rebased and merged with the branch on #23023, but I am getting now

[cvxopt-1.1.8.p2] pkg_resources.DistributionNotFound: The 'sphinxcontrib-websupport' distribution was not found and is required by Sphinx
[cvxopt-1.1.8.p2] make[2]: *** [Makefile:30: html] Error 1

if I try to build with SAGE_SPKG_INSTALL_DOCS=yes

Any idea what sphinxcontrib-websupport is all about? Is it another sphinx package we need to distribute, but we don't?

Yes it is a runtime dependency. It used to be part of sphinx proper. I had to write a new ebuild for it for Gentoo (actually it was a bit more complicated than that but I digress). So far we didn't appear to have a use for it in sage, cvxopt is the first package to require it.

[dimpase]

Indeed, build with SAGE_SPKG_INSTALL_DOCS=yes completes after

./sage --pip install sphinxcontrib-websupport

Do we thus open another ticket to make it a standard (pip) package?

[fbissey]

It would be acceptable to me for it to be part of this ticket.

[dimpase]

#23764 deals with the apparent problem with adding sphinxcontrib-websupport as a package...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f97adda

added sphinxcontrib_websupport

[dimpase]

added sphinxcontrib_websupport as a standard package. (As I don't see how a standard package may depend on an optional one).

[jdemeyer]

1. The dependencies are not quite right: sphinx depends on sphinxcontrib-websupport, not the other way around. Also, it makes little sense to add sphinxcontrib-websupport as direct dependency of cvxopt. Adding sphinx should be sufficient.

In cvxopt/spkg-install:

2. You can simplify

       if [ -d $SAGE_LOCAL/share/doc/cvxopt/html ] ; then
          rm -rf $SAGE_LOCAL/share/doc/cvxopt/html
       fi
   

to

       rm -rf "$SAGE_LOCAL/share/doc/cvxopt/html"

3. You should quote "$SAGE_LOCAL"

4. It would be good to fix the indentation: use 4 spaces and also indent the comments.

[gh-dimpase]

Replying to jdemeyer:

1. The dependencies are not quite right: sphinx depends on sphinxcontrib-websupport, not the other way around.

I don't understand, are you saying one can install sphinxcontrib-websupport without installing sphinx first? This sounds strange to me, as the former is a component of the latter.

Also, it makes little sense to add sphinxcontrib-websupport as direct dependency of cvxopt. Adding sphinx should be sufficient.

Only if the dependencies are changed as you proposed.

[jdemeyer]

Replying to gh-dimpase:

I don't understand, are you saying one can install sphinxcontrib-websupport without installing sphinx first?

Yes indeed.

This sounds strange to me, as the former is a component of the latter.

It's not really a component. It's more like an extra package.

[gh-dimpase]

Replying to jdemeyer:

Replying to gh-dimpase:

I don't understand, are you saying one can install sphinxcontrib-websupport without installing sphinx first?

Yes indeed.

This sounds strange to me, as the former is a component of the latter.

It's not really a component. It's more like an extra package.

sphinxcontrib-webuspport provides a Python API to easily integrate 
Sphinx documentation into your Web application.

From this one concludes that it is useless without sphinx, and thus it is its optional component. Do you know other uses for it? I don't.

[jdemeyer]

Replying to gh-dimpase:

From this one concludes that it is useless without sphinx, and thus it is its optional component.

It's not optional. Even if you don't use it, Sphinx won't run without sphinxcontrib-websupport.

[gh-dimpase]

huh? This ticket adds sphinxcontrib-websupport as a package! We never shipped it before.

[fbissey]

Because in sphinx 1.5.x and under it was a part of sphinx. It has been split off in 1.6.

[gh-dimpase]

Replying to fbissey:

Because in sphinx 1.5.x and under it was a part of sphinx. It has been split off in 1.6.

thanks for clarification. This seems to explain the mutual confusion over the dependence hierarchy.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c853c73	Merge branch 'public/14645' of trac.sagemath.org:sage into t14645
​20a3353	reviewer's comments on cvxopt/spkg-install

[gh-dimpase]

I have fixed spkg-install as requested, and left the deps as they were.

[jdemeyer]

Replying to gh-dimpase:

left the deps as they were.

Why? They are wrong!

[gh-dimpase]

Replying to jdemeyer:

Replying to gh-dimpase:

left the deps as they were.

Why? They are wrong!

No they are not! Why are you saying that Sphinx won't run without sphinxcontrib-websupport while it is perfectly doing this on the current beta? Do you think I have added a link to the tarball merely as work of art?

[dimpase]

English please.

[jdemeyer]

Please wait... I will explain but you need to give me time to do that.

[jdemeyer]

I'm splitting the "package sphinxcontrib-websupport" part of this ticket as #24972.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9abfc71	minimal fix to ensure that docs build/install
​6cb6c93	Further fixes to docbuilding of cvxopt

[dimpase]

I would most appreciate no silent switches of the ticket status back to "needs work". I find this very rude.

[jdemeyer]

Replying to dimpase:

I would most appreciate no silent switches of the ticket status back to "needs work".

I had to set it back to "needs work" because it was wrong and I didn't want anybody accidentally setting this to "positive review" in that state.

The reason for the silence is that I need more time to test things, gather evidence and explain you properly why it is wrong.

[dimpase]

How hard is to add a comment saying "I need more time to explain"? Otherwise I take it as "I have already explained this" and it makes makes me feel stupid.

And I am most probably not the only one who gets upset this way.

[jdemeyer]

Sorry for that.

[jdemeyer]

I tried to explain on #24972.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​f5dd6db	added sphinxcontrib_websupport
​8ecd93c	Minor fixes to sphinxcontrib_websupport
​4bfdc4a	minimal fix to ensure that docs build/install
​61f5908	Further fixes to docbuilding of cvxopt

[gh-dimpase]

I wonder whether the call to the sphinx-build command needs some MathJax-specific options. This is cause I get source TeX formulas in e.g. SAGE_LOCAL/share/doc/cvxopt/fftw.html (compare it with ​http://cvxopt.org/userguide/fftw.html)

It could also be that some other things need to be done to get MathJax working here, I don't know (perhaps it's not relevant to this ticket)

[gh-dimpase]

On ​​http://cvxopt.org/userguide/fftw.html there is no MathJax, they generate png images for formulae. E.g.

<dd><p>Replaces the columns of a dense complex matrix with their inverse
discrete Fourier transforms: if <code class="docutils literal"><span class="pre">X</span></code> has <img class="math" src="_images/math/e11f2701c4a39c7fe543a6c4150b421d50f1c159.png" alt="n"/> rows,</p>
<div class="math">
<p><img src="_images/math/ea898f4dc35c3010225ae66a6d7993bae88b3a56.png" alt="X[k,:] :=
    \frac{1}{n} \sum_{j=0}^{n-1} e^{2\pi j k \sqrt{-1}/n} X[j,:],
    \qquad k=0,\ldots,n-1."/></p>
</div></dd></dl>

I don't know whether we should follow the suit, or leave it as it is, or make it use MathJax...

[dimpase]

ping...

[dimpase]

it all works now.