Opened 8 years ago

Closed 8 years ago

#11665 closed defect (fixed)

Don't delete built Sage documentation until Sphinx has been successfully (re)installed

Reported by: leif Owned by: tbd
Priority: major Milestone: sage-4.7.2
Component: packages: standard Keywords: spkg doc/output html latex
Cc: Merged in: sage-4.7.2.alpha2
Authors: Leif Leonhardy Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #11659 Stopgaps:

Description (last modified by leif)

As the title says.


New spkg: http://spkg-upload.googlecode.com/files/sphinx-1.0.4.p8.spkg

md5sum: 5529e918d8a73e1476c0366fd5fcee35 sphinx-1.0.4.p8.spkg

sphinx-1.0.4.p8 (Leif Leonhardy, August 17th 2011)

  • #11665: Don't delete previously built Sage documentation (in $SAGE_ROOT/devel/sage-*/doc/output/) until Sphinx has been successfully reinstalled.
  • Also, don't delete a previous Sphinx installation until Sphinx has been successfully (re)built. To achieve this, we now first run 'python setup.py build', then delete any previously installed version, and after that run 'python setup.py install'.
  • Quote the filename of patches.
  • Some clean-up, add some messages (and blank lines to separate the output).
  • Added some more of Sphinx' dependencies, corrected minimal version numbers required.

Attachments (1)

sphinx-1.0.4.p7-p8.diff (6.4 KB) - added by leif 8 years ago.
Diff between Jeroens's p7 spkg (#11659) and the p8. For reference / review.

Download all attachments as: .zip

Change History (22)

comment:1 Changed 8 years ago by leif

I wonder if we should delete previously built Sage documentation at all (in all devel branches btw.), since rebuilding the documentation takes a fair amount of time, and if someone's going to rebuild it (from scratch), he'll certainly be smart enough to delete whatever generated output he wants (or needs) to.

The current behaviour is IMHO somewhat surprising, at least in that it affects all branches as mentioned above.

And worse, this even happens whenever Sphinx gets reinstalled just because some of its (direct or indirect) dependencies changed (when using make with SAGE_UPGRADING=yes to also rebuild all dependent packages).

comment:2 Changed 8 years ago by leif

  • Authors set to Leif Leonhardy
  • Description modified (diff)
  • Status changed from new to needs_review

New spkg is up.

comment:3 Changed 8 years ago by jhpalmieri

I know this isn't your change, but setup.py says

requires = ['Pygments>=0.8', 'Jinja2>=2.2', 'docutils>=0.5']

So why are the prerequisites "Pygments >= 1.3.1" and "docutils >= 0.4"?

The changes look okay at first glance, otherwise. I'll keep looking at them, as should other people.

comment:4 follow-up: Changed 8 years ago by jhpalmieri

Another pre-existing condition (i.e., this was a problem with the previous version of the spkg):

Processing dependencies for Sphinx==1.0.4
Searching for docutils==0.7
Best match: docutils 0.7
Adding docutils 0.7 to easy-install.pth file

Using /Applications/sage_builds/sage-4.7.1.rc2/local/lib/python2.6/site-packages

Why wasn't docutils in easy-install.pth already? The other prerequisites look okay, for example

Searching for Jinja2==2.5.5
Best match: Jinja2 2.5.5
Processing Jinja2-2.5.5-py2.6.egg
Jinja2 2.5.5 is already the active version in easy-install.pth

Using /Applications/sage_builds/sage-4.7.1.rc2/local/lib/python2.6/site-packages/Jinja2-2.5.5-py2.6.egg

Looking at easy-install.pth, indeed there is no entry for docutils, either before or after (!) the Sphinx installation. Before installing Sphinx, this is perhaps the fault of the docutils spkg, although I'm not sure how to fix it. After installing Sphinx, either there is a problem with the Sphinx installation process, or the line "Adding docutils 0.7 to easy-install.pth file" is just wrong.

comment:5 in reply to: ↑ 4 ; follow-up: Changed 8 years ago by leif

Replying to jhpalmieri:

Why wasn't docutils in easy-install.pth already? [...]
Looking at easy-install.pth, indeed there is no entry for docutils, either before or after (!) the Sphinx installation. Before installing Sphinx, this is perhaps the fault of the docutils spkg, although I'm not sure how to fix it. After installing Sphinx, either there is a problem with the Sphinx installation process, or the line "Adding docutils 0.7 to easy-install.pth file" is just wrong.

Perhaps ticket:10176:easy_install.py.debug.patch helps in debugging this.

(I don't have an entry for docutils in easy-install.pth in any Sage installation since [at least] Sage 4.5.3; haven't checked earlier versions just because they're located on other machines or filesystems.)

comment:6 in reply to: ↑ 5 Changed 8 years ago by leif

Replying to leif:

(I don't have an entry for docutils in easy-install.pth in any Sage installation since [at least] Sage 4.5.3; haven't checked earlier versions just because they're located on other machines or filesystems.)

FWIW, in no Sage installation since [at least] Sage 4.2.1, so I guess it's never been there.

comment:7 follow-up: Changed 8 years ago by jhpalmieri

Do you understand why the Sphinx installation process says "Adding docutils 0.7 to easy-install.pth file" but doesn't actually do anything?

Otherwise, I'm happy with the changes, except that the list of prerequisites in SPKG.txt should be changed (or explained, if it should stay as is). The issue with docutils and easy-install.pth can be deferred to another ticket, if it even needs to be addressed at all. (Sphinx seems to work, so I don't know how important it is.)

comment:8 in reply to: ↑ 7 Changed 8 years ago by leif

  • Reviewers set to John Palmieri

Replying to jhpalmieri:

Do you understand why the Sphinx installation process says "Adding docutils 0.7 to easy-install.pth file" but doesn't actually do anything?

Well, Sphinx just uses distutils / distribute, and IIRC that prints that message.

A problem we previously had was that packages found in Python's current path won't get added, despite of the message (which is printed in advance).


Otherwise, I'm happy with the changes, except that the list of prerequisites in SPKG.txt should be changed (or explained, if it should stay as is).

Yes, I can correct them. AFAIK a couple of spkgs have rather "random" numbers as the minimal version of some other package required; usually simply the current version in Sage is taken. (And not even the upstream pages always give the correct versions required, some not even the full list of requirements.)

comment:9 Changed 8 years ago by leif

P.S.:

As long as the packages are installed in .../site-packages/<package-basename>/ (like docutils are), this doesn't matter or cause any trouble.

Just the message is perhaps confusing.

comment:10 Changed 8 years ago by jhpalmieri

Okay. Once you upgrade the list of prerequisites, I'll give this a positive review. It doesn't sound like it's necessary to open a follow-up ticket on docutils and easy-install.pth.

Changed 8 years ago by leif

Diff between Jeroens's p7 spkg (#11659) and the p8. For reference / review.

comment:11 follow-up: Changed 8 years ago by leif

  • SPKG.txt

    diff -r 2febeec86a8f -r be96c2edf57b SPKG.txt
    a b  
    1111
    1212== License ==
    1313
    14 Modified BSD
     14Modified BSD; see e.g. its egg-info file for other options
    1515
    1616== SPKG Maintainers ==
    1717
     
    1919
    2020== Upstream Contact ==
    2121
    22 Author:  Georg Brandl <georg at python org>
    23 Home Page: http://sphinx.pocoo.org
     22Author: Georg Brandl <georg at python org>
     23Home Page: http://sphinx.pocoo.org,
     24  see also http://pypi.python.org/pypi/Sphinx
    2425
    2526== Dependencies ==
    26  * Jinja >= 2.2
    27  * Pygments >= 1.3.1
    28  * docutils >= 0.4
    29  * distutils
     27 * Jinja2 >= 2.2
     28 * Pygments >= 0.8
     29 * docutils >= 0.5
     30 * setuptools / distribute
    3031 * Python
    31  * GNU patch
     32 * GNU patch (shipped with Sage)
    3233
    3334== Special Update/Build Instructions ==
    3435
     
    6364
    6465== Changelog ==
    6566
    66 === sphinx-1.0.4.p8 (Leif Leonhardy, August 8th 2011) ===
     67=== sphinx-1.0.4.p8 (Leif Leonhardy, August 17th 2011) ===
    6768 * #11665: Don't delete previously built Sage documentation (in
    6869   $SAGE_ROOT/devel/sage-*/doc/output/) until Sphinx has been successfully
    6970   reinstalled.
     
    7374   that run 'python setup.py install'.
    7475 * Quote the filename of patches.
    7576 * Some clean-up, add some messages (and blank lines to separate the output).
    76  * Added some more of Sphinx' dependencies.
     77 * Added some more of Sphinx' dependencies, corrected minimal version numbers
     78   required.
    7779
    7880=== sphinx-1.0.4.p7 (Jeroen Demeyer, 2011-08-08) ===
    7981 * Ticket #11659: Increase LaTeX SAVE_SIZE from 20000 to 50000

I've updated the attached diff (between the p7 and the p8); I'll replace the spkg on Google code soon.

comment:12 in reply to: ↑ 11 Changed 8 years ago by leif

  • Description modified (diff)

Replying to leif:

I'll replace the spkg on Google code soon.

New spkg is up now. Funny how many people already downloaded the previous one.

comment:13 follow-up: Changed 8 years ago by leif

P.S.: I still wonder whether we should delete previously built Sage documentation at all, and if so, of which branch(es).

I believe doing so has more potential of annoyance than not doing so.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 8 years ago by jhpalmieri

Replying to leif:

P.S.: I still wonder whether we should delete previously built Sage documentation at all, and if so, of which branch(es).

I believe doing so has more potential of annoyance than not doing so.

There have certainly been cases where old doctree directories were incompatible between Sphinx versions, resulting in crashes for sage -docbuild ....

comment:15 in reply to: ↑ 14 Changed 8 years ago by leif

Replying to jhpalmieri:

Replying to leif:

I believe doing so has more potential of annoyance than not doing so.

There have certainly been cases where old doctree directories were incompatible between Sphinx versions, resulting in crashes for sage -docbuild ....

Well, that's easily fixed by a rm -rf devel/sage/doc/output; the deletion of work which perhaps took hours (or more) of CPU time isn't.

comment:16 Changed 8 years ago by jhpalmieri

If #6495 gets reviewed and merged, it will speed up generation of the reference manual on machines with many processors. So deleting the output wouldn't be as much of an issue then. (The reference manual is the real bottleneck; everything else takes just a few minutes.)

The deletion of the output was introduced in #7286. Some options:

  • Delete the output, as is done now.
  • Don't delete the output, but print a message after installation about potential problems. Such a message may get lost among many others during a run of sage -upgrade.
  • Delete the output if we've updated the version of Sphinx (which we can tell by looking at the name of the Sphinx directory in SAGE_LOCAL/lib/python/site-packages), but not otherwise. But Sage-specific changes in Sphinx, which wouldn't affect the Sphinx version number, could conceivably lead to incompatible pickles.

The first of these is in some sense the safest, since it is the status quo. It will make some operations take more time, but it won't cause things to fail in confusing ways. The others could lead to failures and confusing error messages. So your comment that deleting the docs has "more potential of annoyance than not doing so" depends on your point of view: is it better to make things work but take longer, or to have them be faster but possibly fail in confusing ways.

Speaking of confusing error messages, for a while now, running "sage -docbuild tutorial pickle" (or replace 'pickle' by 'json') has failed...

comment:17 Changed 8 years ago by leif

The funny thing is that #6495's description says:

"Building the Sage reference manual can use significant computer resources. Easing the burden could speed up Sage development."

Building the documentation in parallel doesn't take less resources, but even more. It only makes things faster if you have more resources available.

comment:18 follow-up: Changed 8 years ago by jhpalmieri

That's a bit misleading. Building the entire reference manual in one piece takes more memory, for example, than building it in chunks — it doesn't scale up well — so building with two threads should use less memory, even with both threads combined, than building before the patches at #6495. This is especially true when building in !LaTeX format, it seems to me. Based on my anecdotal experiences: building, say, the PDF version of the reference manual was pretty painful before the patches on my home computer: it slowed other things down noticeably. Building after the patches is more pleasant.

(I didn't write the description at #6495, for what that's worth.)

comment:19 in reply to: ↑ 18 Changed 8 years ago by leif

Replying to jhpalmieri:

(I didn't write the description at #6495, for what that's worth.)

I know, Mitesh did.

comment:20 Changed 8 years ago by jhpalmieri

  • Status changed from needs_review to positive_review

The current spkg gets a positive review.

comment:21 Changed 8 years ago by jdemeyer

  • Merged in set to sage-4.7.2.alpha2
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.