Opened 8 years ago
Closed 8 years ago
#11665 closed defect (fixed)
Don't delete built Sage documentation until Sphinx has been successfully (re)installed
Reported by: | leif | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-4.7.2 |
Component: | packages: standard | Keywords: | spkg doc/output html latex |
Cc: | Merged in: | sage-4.7.2.alpha2 | |
Authors: | Leif Leonhardy | Reviewers: | John Palmieri |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | #11659 | Stopgaps: |
Description (last modified by )
As the title says.
New spkg: http://spkg-upload.googlecode.com/files/sphinx-1.0.4.p8.spkg
md5sum: 5529e918d8a73e1476c0366fd5fcee35 sphinx-1.0.4.p8.spkg
sphinx-1.0.4.p8 (Leif Leonhardy, August 17th 2011)
- #11665: Don't delete previously built Sage documentation (in $SAGE_ROOT/devel/sage-*/doc/output/) until Sphinx has been successfully reinstalled.
- Also, don't delete a previous Sphinx installation until Sphinx has been successfully (re)built. To achieve this, we now first run 'python setup.py build', then delete any previously installed version, and after that run 'python setup.py install'.
- Quote the filename of patches.
- Some clean-up, add some messages (and blank lines to separate the output).
- Added some more of Sphinx' dependencies, corrected minimal version numbers required.
Attachments (1)
Change History (22)
comment:1 Changed 8 years ago by
comment:2 Changed 8 years ago by
- Description modified (diff)
- Status changed from new to needs_review
New spkg is up.
comment:3 Changed 8 years ago by
I know this isn't your change, but setup.py says
requires = ['Pygments>=0.8', 'Jinja2>=2.2', 'docutils>=0.5']
So why are the prerequisites "Pygments >= 1.3.1" and "docutils >= 0.4"?
The changes look okay at first glance, otherwise. I'll keep looking at them, as should other people.
comment:4 follow-up: ↓ 5 Changed 8 years ago by
Another pre-existing condition (i.e., this was a problem with the previous version of the spkg):
Processing dependencies for Sphinx==1.0.4 Searching for docutils==0.7 Best match: docutils 0.7 Adding docutils 0.7 to easy-install.pth file Using /Applications/sage_builds/sage-4.7.1.rc2/local/lib/python2.6/site-packages
Why wasn't docutils in easy-install.pth
already? The other prerequisites look okay, for example
Searching for Jinja2==2.5.5 Best match: Jinja2 2.5.5 Processing Jinja2-2.5.5-py2.6.egg Jinja2 2.5.5 is already the active version in easy-install.pth Using /Applications/sage_builds/sage-4.7.1.rc2/local/lib/python2.6/site-packages/Jinja2-2.5.5-py2.6.egg
Looking at easy-install.pth, indeed there is no entry for docutils, either before or after (!) the Sphinx installation. Before installing Sphinx, this is perhaps the fault of the docutils spkg, although I'm not sure how to fix it. After installing Sphinx, either there is a problem with the Sphinx installation process, or the line "Adding docutils 0.7 to easy-install.pth file" is just wrong.
comment:5 in reply to: ↑ 4 ; follow-up: ↓ 6 Changed 8 years ago by
Replying to jhpalmieri:
Why wasn't docutils in
easy-install.pth
already? [...]
Looking at easy-install.pth, indeed there is no entry for docutils, either before or after (!) the Sphinx installation. Before installing Sphinx, this is perhaps the fault of the docutils spkg, although I'm not sure how to fix it. After installing Sphinx, either there is a problem with the Sphinx installation process, or the line "Adding docutils 0.7 to easy-install.pth file" is just wrong.
Perhaps ticket:10176:easy_install.py.debug.patch helps in debugging this.
(I don't have an entry for docutils
in easy-install.pth
in any Sage installation since [at least] Sage 4.5.3; haven't checked earlier versions just because they're located on other machines or filesystems.)
comment:6 in reply to: ↑ 5 Changed 8 years ago by
Replying to leif:
(I don't have an entry for
docutils
ineasy-install.pth
in any Sage installation since [at least] Sage 4.5.3; haven't checked earlier versions just because they're located on other machines or filesystems.)
FWIW, in no Sage installation since [at least] Sage 4.2.1, so I guess it's never been there.
comment:7 follow-up: ↓ 8 Changed 8 years ago by
Do you understand why the Sphinx installation process says "Adding docutils 0.7 to easy-install.pth file" but doesn't actually do anything?
Otherwise, I'm happy with the changes, except that the list of prerequisites in SPKG.txt should be changed (or explained, if it should stay as is). The issue with docutils and easy-install.pth can be deferred to another ticket, if it even needs to be addressed at all. (Sphinx seems to work, so I don't know how important it is.)
comment:8 in reply to: ↑ 7 Changed 8 years ago by
- Reviewers set to John Palmieri
Replying to jhpalmieri:
Do you understand why the Sphinx installation process says "Adding docutils 0.7 to easy-install.pth file" but doesn't actually do anything?
Well, Sphinx just uses distutils / distribute, and IIRC that prints that message.
A problem we previously had was that packages found in Python's current path won't get added, despite of the message (which is printed in advance).
Otherwise, I'm happy with the changes, except that the list of prerequisites in SPKG.txt should be changed (or explained, if it should stay as is).
Yes, I can correct them. AFAIK a couple of spkgs have rather "random" numbers as the minimal version of some other package required; usually simply the current version in Sage is taken. (And not even the upstream pages always give the correct versions required, some not even the full list of requirements.)
comment:9 Changed 8 years ago by
P.S.:
As long as the packages are installed in .../site-packages/<package-basename>/
(like docutils
are), this doesn't matter or cause any trouble.
Just the message is perhaps confusing.
comment:10 Changed 8 years ago by
Okay. Once you upgrade the list of prerequisites, I'll give this a positive review. It doesn't sound like it's necessary to open a follow-up ticket on docutils and easy-install.pth.
comment:11 follow-up: ↓ 12 Changed 8 years ago by
-
SPKG.txt
diff -r 2febeec86a8f -r be96c2edf57b SPKG.txt
a b 11 11 12 12 == License == 13 13 14 Modified BSD 14 Modified BSD; see e.g. its egg-info file for other options 15 15 16 16 == SPKG Maintainers == 17 17 … … 19 19 20 20 == Upstream Contact == 21 21 22 Author: Georg Brandl <georg at python org> 23 Home Page: http://sphinx.pocoo.org 22 Author: Georg Brandl <georg at python org> 23 Home Page: http://sphinx.pocoo.org, 24 see also http://pypi.python.org/pypi/Sphinx 24 25 25 26 == Dependencies == 26 * Jinja >= 2.227 * Pygments >= 1.3.128 * docutils >= 0. 429 * distutils27 * Jinja2 >= 2.2 28 * Pygments >= 0.8 29 * docutils >= 0.5 30 * setuptools / distribute 30 31 * Python 31 * GNU patch 32 * GNU patch (shipped with Sage) 32 33 33 34 == Special Update/Build Instructions == 34 35 … … 63 64 64 65 == Changelog == 65 66 66 === sphinx-1.0.4.p8 (Leif Leonhardy, August 8th 2011) ===67 === sphinx-1.0.4.p8 (Leif Leonhardy, August 17th 2011) === 67 68 * #11665: Don't delete previously built Sage documentation (in 68 69 $SAGE_ROOT/devel/sage-*/doc/output/) until Sphinx has been successfully 69 70 reinstalled. … … 73 74 that run 'python setup.py install'. 74 75 * Quote the filename of patches. 75 76 * Some clean-up, add some messages (and blank lines to separate the output). 76 * Added some more of Sphinx' dependencies. 77 * Added some more of Sphinx' dependencies, corrected minimal version numbers 78 required. 77 79 78 80 === sphinx-1.0.4.p7 (Jeroen Demeyer, 2011-08-08) === 79 81 * Ticket #11659: Increase LaTeX SAVE_SIZE from 20000 to 50000
I've updated the attached diff (between the p7 and the p8); I'll replace the spkg on Google code soon.
comment:12 in reply to: ↑ 11 Changed 8 years ago by
- Description modified (diff)
Replying to leif:
I'll replace the spkg on Google code soon.
New spkg is up now. Funny how many people already downloaded the previous one.
comment:13 follow-up: ↓ 14 Changed 8 years ago by
P.S.: I still wonder whether we should delete previously built Sage documentation at all, and if so, of which branch(es).
I believe doing so has more potential of annoyance than not doing so.
comment:14 in reply to: ↑ 13 ; follow-up: ↓ 15 Changed 8 years ago by
Replying to leif:
P.S.: I still wonder whether we should delete previously built Sage documentation at all, and if so, of which branch(es).
I believe doing so has more potential of annoyance than not doing so.
There have certainly been cases where old doctree directories were incompatible between Sphinx versions, resulting in crashes for sage -docbuild ...
.
comment:15 in reply to: ↑ 14 Changed 8 years ago by
Replying to jhpalmieri:
Replying to leif:
I believe doing so has more potential of annoyance than not doing so.
There have certainly been cases where old doctree directories were incompatible between Sphinx versions, resulting in crashes for
sage -docbuild ...
.
Well, that's easily fixed by a rm -rf devel/sage/doc/output
; the deletion of work which perhaps took hours (or more) of CPU time isn't.
comment:16 Changed 8 years ago by
If #6495 gets reviewed and merged, it will speed up generation of the reference manual on machines with many processors. So deleting the output wouldn't be as much of an issue then. (The reference manual is the real bottleneck; everything else takes just a few minutes.)
The deletion of the output was introduced in #7286. Some options:
- Delete the output, as is done now.
- Don't delete the output, but print a message after installation about potential problems. Such a message may get lost among many others during a run of
sage -upgrade
. - Delete the output if we've updated the version of Sphinx (which we can tell by looking at the name of the Sphinx directory in SAGE_LOCAL/lib/python/site-packages), but not otherwise. But Sage-specific changes in Sphinx, which wouldn't affect the Sphinx version number, could conceivably lead to incompatible pickles.
The first of these is in some sense the safest, since it is the status quo. It will make some operations take more time, but it won't cause things to fail in confusing ways. The others could lead to failures and confusing error messages. So your comment that deleting the docs has "more potential of annoyance than not doing so" depends on your point of view: is it better to make things work but take longer, or to have them be faster but possibly fail in confusing ways.
Speaking of confusing error messages, for a while now, running "sage -docbuild tutorial pickle" (or replace 'pickle' by 'json') has failed...
comment:17 Changed 8 years ago by
The funny thing is that #6495's description says:
"Building the Sage reference manual can use significant computer resources. Easing the burden could speed up Sage development."
Building the documentation in parallel doesn't take less resources, but even more. It only makes things faster if you have more resources available.
comment:18 follow-up: ↓ 19 Changed 8 years ago by
That's a bit misleading. Building the entire reference manual in one piece takes more memory, for example, than building it in chunks — it doesn't scale up well — so building with two threads should use less memory, even with both threads combined, than building before the patches at #6495. This is especially true when building in !LaTeX format, it seems to me. Based on my anecdotal experiences: building, say, the PDF version of the reference manual was pretty painful before the patches on my home computer: it slowed other things down noticeably. Building after the patches is more pleasant.
(I didn't write the description at #6495, for what that's worth.)
comment:19 in reply to: ↑ 18 Changed 8 years ago by
Replying to jhpalmieri:
(I didn't write the description at #6495, for what that's worth.)
I know, Mitesh did.
comment:20 Changed 8 years ago by
- Status changed from needs_review to positive_review
The current spkg gets a positive review.
comment:21 Changed 8 years ago by
- Merged in set to sage-4.7.2.alpha2
- Resolution set to fixed
- Status changed from positive_review to closed
I wonder if we should delete previously built Sage documentation at all (in all devel branches btw.), since rebuilding the documentation takes a fair amount of time, and if someone's going to rebuild it (from scratch), he'll certainly be smart enough to delete whatever generated output he wants (or needs) to.
The current behaviour is IMHO somewhat surprising, at least in that it affects all branches as mentioned above.
And worse, this even happens whenever Sphinx gets reinstalled just because some of its (direct or indirect) dependencies changed (when using
make
withSAGE_UPGRADING=yes
to also rebuild all dependent packages).