Opened 8 years ago

Last modified 4 years ago

#11197 needs_info defect

Make spkg docs be built and installed after the full build

Reported by: jason Owned by: tbd
Priority: major Milestone: sage-6.4
Component: build Keywords: sd32
Cc: ddrake Merged in:
Authors: John Palmieri Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jhpalmieri)

Following on some concerns brought up about dependencies in building spkg docs, as implemented in #10823, here is a ticket for reimplementing #10823 so that docs are built after the full Sage build. In this way, spkg docs are built when the normal Sage docs are built, and we don't have any worries about having python or sphinx or other dependencies installed.

From Dan's message to sage-devel:

How about, in addition to spkg-install and spkg-check, we have spkg-install-docs? We would move the logic that checks the SAGE_SPKG_INSTALL_DOCS variable to the build/install script, which would run the script if the env var is set. This would also allow the build process to determine if a spkg has documentation that can be built, using something roughly like tar jtf foo.spkg | grep spkg-install-docs

As a data point, doing that for all spkgs in 4.7.alpha4 (except the Sage library, which we handle separately) takes about 70 seconds on a mildly-loaded sagenb.kaist.ac.kr, which is an 8-core Xeon machine.

See http://groups.google.com/group/sage-devel/browse_thread/thread/d46ddccff06d8670 and various tickets associated with #10823

Here are a few updated spkgs:


Apply either

Attachments (2)

trac_11197-spkg-install-docs.patch (3.1 KB) - added by jhpalmieri 8 years ago.
root repo
trac_11197-spkg-install-docs.v2.patch (4.5 KB) - added by jhpalmieri 8 years ago.
root repo; implement "make spkg-doc" from SAGE_ROOT

Download all attachments as: .zip

Change History (29)

comment:1 Changed 8 years ago by jason

  • Description modified (diff)

comment:2 Changed 8 years ago by jason

  • Description modified (diff)

comment:3 Changed 8 years ago by jason

  • Description modified (diff)

comment:4 Changed 8 years ago by jason

  • Description modified (diff)

comment:5 Changed 8 years ago by jason

  • Cc ddrake added

comment:6 follow-up: Changed 8 years ago by jhpalmieri

  • Authors set to John Palmieri
  • Status changed from new to needs_review

Here's a patch for the Sage root repo (the file spkg/install). This should perhaps be considered a first draft, because there are a few issues/questions.

  • Right now, it records a log for the docbuilding in spkg/logs/numpy-1.5.1.p0-docs.log, for example, instead of just appending to the already existing spkg/logs/numpy-1.5.1.p0.log. It's easy enough to switch it around. Opinions?
  • Right now, it stores a placeholder file in spkg/installed if the docs seem to have been built correctly. (Since after building each spkg, we delete the directory where it was built (spkg/build/numpy-...), we can't just rely on 'make' recognizing that it's already been installed, I think.) Is this the right way to do it?

A few things which I think belong on other tickets or which I don't know how to solve:

  • With the approach here, running "sage -i ..." or "sage -f ..." won't install the docs. Should we add something to the sage-spkg script which tries to build the docs if SAGE_SPKG_INSTALL_DOCS is "yes", but fails gracefully if it can't? Or only build the docs in sage-spkg if we can tell that the Sage library has already been installed, or some other criterion like this? This may end up with some code duplication, or separating out the doc-building stuff into a separate script (and then gets us entangled in the base repo vs. scripts repo debate...).
  • I noticed that the most recent cython spkg (the one in 4.7.alpha4) has files for building the docs, at least some of which weren't added to the Mercurial repository. I haven't checked the other spkgs for this sort of issue, but someone should...

Changed 8 years ago by jhpalmieri

root repo

comment:7 in reply to: ↑ 6 ; follow-up: Changed 8 years ago by jason

  • Description modified (diff)

Replying to jhpalmieri:

Here's a patch for the Sage root repo (the file spkg/install). This should perhaps be considered a first draft, because there are a few issues/questions. - Right now, it records a log for the docbuilding in spkg/logs/numpy-1.5.1.p0-docs.log, for example, instead of just appending to the already existing spkg/logs/numpy-1.5.1.p0.log. It's easy enough to switch it around. Opinions?

  • Right now, it stores a placeholder file in spkg/installed if the docs seem to have been built correctly. (Since after building each spkg, we delete the directory where it was built (spkg/build/numpy-...), we can't just rely on 'make' recognizing that it's already been installed, I think.) Is this the right way to do it? A few things which I think belong on other tickets or which I don't know how to solve:

To me, it seems okay to think about the docs install as a separate thing from the spkg install, so it makes sense to me to have a separate log file and separate placeholder file. I don't know much about this part of the build process, though, so my +1 doesn't count for much.

  • With the approach here, running "sage -i ..." or "sage -f ..." won't install the docs. Should we add something to the sage-spkg script which tries to build the docs if SAGE_SPKG_INSTALL_DOCS is "yes", but fails gracefully if it can't? Or only build the docs in sage-spkg if we can tell that the Sage library has already been installed, or some other criterion like this? This may end up with some code duplication, or separating out the doc-building stuff into a separate script (and then gets us entangled in the base repo vs. scripts repo debate...).

Hmm...thinking about this one still...

  • I noticed that the most recent cython spkg (the one in 4.7.alpha4) has files for building the docs, at least some of which weren't added to the Mercurial repository. I haven't checked the other spkgs for this sort of issue, but someone should...

Are you talking about the docs directory containing the documentation? That directory is part of the upstream source, so shouldn't be in the spkg repository. However, as of a week or two ago, that docs directory has been merged into the main Cython source tree, so future versions of the spkg will not have that docs directory.

comment:8 in reply to: ↑ 7 Changed 8 years ago by jhpalmieri

Replying to jason:

Are you talking about the docs directory containing the documentation?

I mean that if I unpack cython-0.14.1.p2.spkg, cd into the directory and type "hg status", I see

? SPKG.txt.orig
? doc/.hgignore
? doc/Makefile
? doc/README
? doc/TODO
? doc/_static/cython-logo-light.png
? doc/_static/cythonlogo.png
? doc/_static/favicon.ico
? doc/_templates/layout.html
? doc/conf.py
? doc/examples/tutorial/fib1/fib.pyx
? doc/examples/tutorial/fib1/setup.py
? doc/examples/tutorial/great_circle/c1.pyx
? doc/examples/tutorial/great_circle/c2.pyx
? doc/examples/tutorial/great_circle/p1.py
? doc/examples/tutorial/primes/primes.py
? doc/examples/tutorial/primes/primes.pyx
? doc/examples/tutorial/primes/setup.py
? doc/index.rst
? doc/sphinxext/cython_highlighting.py
? doc/sphinxext/ipython_console_highlighting.py
? doc/todo.txt
? doc/welcome.rst

So I guess those need to be added to .hginore, or the whole doc directory needs to added, or if everything in there has since been moved to src, we only need to get rid of SPKG.txt.orig.

comment:9 follow-up: Changed 8 years ago by jason

Good point; the doc directory should be added to .hgignore. And SPKG.txt.orig should be deleted. Sorry; that seems pretty sloppy of me.

comment:10 Changed 8 years ago by jason

I've noted these changes on #10827.

comment:11 in reply to: ↑ 9 Changed 8 years ago by jhpalmieri

Replying to jason:

Good point; the doc directory should be added to .hgignore. And SPKG.txt.orig should be deleted. Sorry; that seems pretty sloppy of me.

I've made those kinds of mistakes lots of times myself...

comment:12 follow-up: Changed 8 years ago by vbraun

If we are moving to a two-step binary / documentation build process, how about we remove the SAGE_SPKG_INSTALL_DOCS environment variable and, instead, provide a "doc" target in the main Makefile. So you can just "make doc", thats much easier to discover than a cryptic environment variable.

comment:13 in reply to: ↑ 12 Changed 8 years ago by jason

Replying to vbraun:

If we are moving to a two-step binary / documentation build process, how about we remove the SAGE_SPKG_INSTALL_DOCS environment variable and, instead, provide a "doc" target in the main Makefile. So you can just "make doc", thats much easier to discover than a cryptic environment variable.

+1

comment:14 Changed 8 years ago by jason

Maybe we should clearly distinguish between sage's documentation and spkg documentation. I'd expect make doc to just build Sage's documentation. Maybe we could have two targets: make doc (build and install Sage docs) and make spkg-doc (build and install any spkg docs).

comment:15 Changed 8 years ago by jhpalmieri

  • Description modified (diff)
  • Status changed from needs_review to needs_info

Here's a new patch. This does the Makefile approach. I think I like the Makefile approach, but I'm not positive, so I'm marking this as "needs info". Please provide opinions about the two approaches and also about the specific patches.

Changed 8 years ago by jhpalmieri

root repo; implement "make spkg-doc" from SAGE_ROOT

comment:16 Changed 8 years ago by was

  • Keywords sd32 added

comment:17 Changed 8 years ago by leif

Hmmm, one problem I see is that a couple of spkgs would need a (re)configure before e.g. make install-docs etc. would work.

Also, many packages provide [alternate] documentation in various formats (man pages, texinfo files, HTML, dvi / PDF etc.), some of which require tools (i.e., prerequisites either provided by the system or Sage), and some even some small build process just to prepare the sources for other tools. So it isn't always clear what would be installed [in addition]. Re-unpacking and re-configuring an spkg just to install docs is IMHO also a bit odd, just like keeping the build directories until eventually some tools are available after the normal build has finished.

The easiest way would be to simply include some pre-built versions of documentation requiring external tools, and just copy / install them if SAGE_SPKG_INSTALL_DOCS=yes or an spkg's spkg-install-docs is run. This could at least in some cases increase an spkg's size significantly though.

comment:18 follow-up: Changed 8 years ago by leif

We could also support some kind of SAGE_SPKG_INSTALL_DOCS=no (and =later) in spkg-install, but that still doesn't tell which kind of documentation.

(I personally would for example always install man pages by default, if available.)

No matter if we build / install any documents for some spkg (or whether they are available at all in the spkg itself), we could create HTML pages with links to upstream [documentation] if appropriate.

Same could be done for licenses.

comment:19 in reply to: ↑ 18 Changed 8 years ago by jhpalmieri

Right now, I'm more in favor of adding a target to the Makefile, rather than the environment variable approach.

Replying to leif:

Hmmm, one problem I see is that a couple of spkgs would need a (re)configure before e.g. make install-docs etc. would work.

Well, this would be taken care of by the doc install script for the particular spkg. The whole point of this ticket is to install the docs separately from the full build, so I'm not sure I understand your concerns. If you want to build the docs during the build, use the approach in #10823, and the spkg-install file can take care of the proper configurations to do things efficiently. (So maybe some spkgs would behave differently depending on whether you ran make spkg-doc compared to make with SAGE_SPKG_INSTALL_DOCS=yes: in the first case, a special script spkg-install-docs would be run, while in the first case, the code would be in spkg-install. Or is that too complicated?)

Also, many packages provide [alternate] documentation in various formats (man pages, texinfo files, HTML, dvi / PDF etc.), some of which require tools (i.e., prerequisites either provided by the system or Sage), and some even some small build process just to prepare the sources for other tools. So it isn't always clear what would be installed [in addition]. Re-unpacking and re-configuring an spkg just to install docs is IMHO also a bit odd, just like keeping the build directories until eventually some tools are available after the normal build has finished.

There could be several possible targets: make spkg-doc which would build whatever the spkg maintainers thought was a sensible default, but also make spkg-doc-html, make spkg-doc-man, etc. Perhaps the script spkg/install-spkg-docs should pass an argument ("html", "pdf", "default", etc.) to each individual spkg-install-docs script.

The easiest way would be to simply include some pre-built versions of documentation requiring external tools, and just copy / install them if SAGE_SPKG_INSTALL_DOCS=yes or an spkg's spkg-install-docs is run. This could at least in some cases increase an spkg's size significantly though.

I don't think the size increase would be acceptable.

Replying to leif:

We could also support some kind of SAGE_SPKG_INSTALL_DOCS=no (and =later) in spkg-install, but that still doesn't tell which kind of documentation.

or SAGE_SPKG_INSTALL_DOCS=html

No matter if we build / install any documents for some spkg (or whether they are available at all in the spkg itself), we could create HTML pages with links to upstream [documentation] if appropriate.

Sure, that sounds like a good idea. Probably for another ticket, once we get this framework set up. At least, setting up these html pages would seem to require another set of ideas and therefore add another level of complications.

comment:20 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:21 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:22 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:23 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:24 in reply to: ↑ description Changed 4 years ago by jdemeyer

Replying to jason:

Following on some concerns brought up about dependencies in building spkg docs, as implemented in #10823

What are these "concerns"? Can these "concerns" be fixed in a simpler way than adding more complexities to the Sage build procedure?

comment:25 Changed 4 years ago by jdemeyer

  • Component changed from packages: standard to build

comment:26 Changed 4 years ago by jdemeyer

I think it needs to be clarified why this ticket is needed.

comment:27 Changed 4 years ago by jhpalmieri

I think the situation is:

  • we want to be able to install documentation for any of the various packages that provide it, but
  • building the documentation may have different prerequisites (e.g., Sphinx) than building the package itself.

It sounds like the plan in this ticket was to build documentation for the packages at the same time as building the regular Sage documentation.

Note: See TracTickets for help on using tickets.