[with spkg, positive review] update Sphinx to version 0.6.2 or 0.6.3

[jhpalmieri]

As the summary says. The spkg can be found at

​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg

After installing it and before building any docs, you will probably need to delete the directory SAGE_ROOT/devel/sage/doc/output -- this holds cached information which may not be compatible between different versions of Sphinx.

Sphinx 0.6.3 also uses Jinja2. Sage includes Jinja, not Jinja2. As long as you're connected to a network, the installation of Sphinx 0.6.3 downloads Jinja2 and installs it -- it can coexist with Jinja, so it shouldn't mess anything up. If you're not connected to a network, and for putting in the Sage tar ball, there is an spkg for Jinja2:

​http://sage.math.washington.edu/home/palmieri/SPKG/jinja2-2.1.1.p0.spkg

For producing a new Sage tarball, since we're adding a new spkg, I think we need to modify the files spkg/install and spkg/standard/deps; I've attached new versions of these.

Finally, the patch below ("inheritance_example.patch") is not required. It provides an illustration of a new capability of Sphinx 0.6.3 (and I think it's quite nice), but it's not something that needs to be installed or reviewed for this ticket.

[jhpalmieri]

Oh, I forgot: you need to apply the patch at #6585 for the reference manual to build. This version of Sphinx also prints a lot of new warnings, too.

[mvngu]

I installed the updated SPKG with

./sage -f http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p0.spkg

and applied the patch at #6585. When doing

./sage -docbuild reference html

I got the following errors:

mvngu@sage sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux]$ ./sage -docbuild reference html
Traceback (most recent call last):
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 667, in <module>
    getattr(get_builder(name), type)()
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 348, in _wrapper
    for module_name in self.get_modified_modules():
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 412, in get_modified_modules
    env = self.get_sphinx_environment()
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/devel/sage/doc/common/builder.py", line 403, in get_sphinx_environment
    return BuildEnvironment.frompickle(config, env_pickle)
  File "/scratch/mvngu/sage-4.1.1.alpha0-sage.math.washignton.edu-x86_64-Linux/local/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py", line 210, in frompickle
    env = pickle.load(picklefile)
AttributeError: 'module' object has no attribute 'RedirStream'

[jhpalmieri]

I think I got this error, too. Once I deleted doc/output, things worked for me. (I think there is something stored in the old version of the documentation which confuses the new version of Sphinx.)

[mvngu]

The upgraded Sphinx SPKG can't handle binary encodings in the file:

sage/combinat/partition.py

In particular, the following sections from that source file:
line:2233

sage: p = loads('x\x9c\x85\x8e\xbd\xaa\xc2@\x10\x85\x89\xff.>\xc4\x94\xda\x04\x15|\x04\xb1\xb1\x90\x0b[\x87I\x18\x935\xc9\xae\xb33\xda\t\xd7\xc2\xf76"biw\x0e\x9c\x9f\xef\xbfW\x08\x96\x94\x16\xa1\xcd\x9dGM\xcf\x18\xd5\xa9\x0b\xde\x1c>Jv\x91PIt\xbf\xcd|m8Y\xdc\xb9w\xe3\xfe\xdc&\xf5\xbb\x1d\x9d/%u^\xa9\xa4hZ\xac)\xfb\x18\x1e\xd8d\xfd\xf8\xe3\xa1\x1df\x1e[\xe2\x91\xdd|\x97!\x1ca\xb5\x84\n\xaf\xdd\x02\xbc\xbe\x05\x1a\x12\x01\xad\xd0C\x88@|\xc1\x064\xc0\x9a\xc7v\x16\xf2\x13\x15\x9a\x15\r\x8a\xf0\xe47\xf9;ixj\x13_u \xd8\x81\x98K\x9e>\x01\x13iVH')

line:2294

sage: p = loads('x\x9c\x85\x8e\xbd\x0e\x820\x14\x85\x03\x8a?\x8d\x0f\xd1Q\x97\x06y\x07\xe3\xaa&\x9d\xc9\xa5\xb9\x96\n\xb4^Z\xdcLt\xf0\xbd\xc5 qt;\'9?\xdf#V\x1e4\n\xe5\x9a\xc2X\x08\xe2\nm0\xc18\xcb\x0e\xa3\xf2\xfb\x16!\xa0\x0f\xbbcn+F\xd1\xe6I\xf1\x9d&k\x19UC\xbb5V{al@\x8d-k\xa0\xc2|44\x95Q\xf6:Q"\x93\xdcB\x834\x93\xe9o\x99\xbb3\xdf\xa6\xbc\x84[\xbf\xc0\xf5\xf7\x87\x7f 8R\x075\x0f\x8eg4\x97+W\\P\x85\\\xd5\xe0=-\xfeC\x0fIFK\x19\xd9\xb2g\x80\x9e\x81u\x85x\x03w\x0eT\xb1')

line:2839

sage: dmp = 'x\x9ck`J.NLO\xd5K\xce\xcfM\xca\xccK,\xd1+H,*\xc9,\xc9\xcc\xcf\xe3\n\x80\xb1\x8a\xe3\x93\x81DIQbf^I1W!\xa3fc!Sm!\xb3F(7\x92x!Km!k(GnbE<\xc8\x88B6\x88\xb9E\x99y\xe9\xc5z@\x05\xa9\xe9\xa9E\\\xb9\x89\xd9\xa9\xf10N!{(\xa3QkP!Gq(c^\x06\x90c\x0c\xe4p\x96&\xe9\x01\x00\xc2\xe53\xfd'

When these lines are removed, the HTML version of the reference manual builds OK. So a possible fix for line 2233 is to replace everything in the TEST block starting from that line by:

sage: p = PartitionsGreatestLE(10,2)
sage: p == loads(dumps(p))
True

We can do something similar for line 2294; replace everything in the TEST block starting from line 2294 by:

sage: p = PartitionsGreatestEQ(10,2)
sage: p == loads(dumps(p))
True

As for the TEST block starting from line 2839, I have no idea how to fix it so that the reference manual would successfully builds. Someone on the sage-combinat team to the rescue?

[jhpalmieri]

Replying to jhpalmieri:

Oh, I forgot: you need to apply the patch at #6585 for the reference manual to build. This version of Sphinx also prints a lot of new warnings, too.

I think that this patch fixes the issue with binary encodings; at least it did for me.

[mvngu]

With the patch at #6585 and the SPKG, building the reference manual on sage.math takes much longer than previously.

[jhpalmieri]

Replying to mvngu:

With the patch at #6585 and the SPKG, building the reference manual on sage.math takes much longer than previously.

I agree: on my mac, it took 28 minutes to build the html version with the old one, and 43 minutes with the new one. With the --jsmath option on, the old one took 6 minutes and the new one took 15 minutes. I was getting similar comparisons on sage.math. (This is with me deleting devel/sage/doc/output between each build, to level the playing field.)

It's strange: for the tutorial and developer's guide, building is about the same or slightly faster with the new version. I also made a modified version of the developer's guide by duplicating some sections (so that the resulting PDF file was 559 pages); the new version was slightly faster there, too. Maybe it's all of the math in the reference manual? Maybe it's the large number of files? Maybe it's the autogeneration of files?

[mpatel]

Cursory ​profiling with #6187's "testreference" target indicates that the rewritten autodoc extension is significantly slower than before. I think autodoc now also uses far more memory. Attempting to build the full reference manual on my machine, Sphinx "stalled" at 97% of modules read. I think the kernel was too busy swapping everything but Sphinx to disk.

Perhaps there's a more efficient, parallelizable representation. Can we subclass and/or Cythonize ​sphinx.ext.autodoc.Documenter?

[mpatel]

Minor progress: It seems that a custom ​autodoc-skip-member method now overrides Sphinx's default behavior, which is to skip private methods (__init__, etc.). To keep these out of the reference manual, we might use, e.g.,

def skip_NestedClass(app, what, name, obj, skip, options):
    """docstring"""
    skip_nested = str(obj).find("sage.misc.misc") != -1 and name.find("MainClass.NestedClass") != -1
    return skip or skip_nested

in conf.py. Unfortunately, this does not significantly reduce the build time or memory usage.

[mpatel]

Apparently, the culprit is sphinx.pycode's Python source analyzer, which sphinx.ext.autodoc uses to extract attribute docstrings. From Sphinx-0.6.2-py2.6.egg/sphinx/pycode/__init__.py:

    The docstrings can either be in special '#:' comments before the assignment
    or in a docstring after it.

According to the analyzer, there are just two places to find such docstrings in the Sage library:

• combinat/designs/block_design.py
• schemes/elliptic_curves/ell_field.py

Still, resource use is high, because there is a lot of caching going on. We can reduce it with ​weak references:

--- __init__.py.orig    2009-08-01 09:58:27.303927239 -0700
+++ __init__.py 2009-08-01 11:35:17.477865562 -0700
@@ -120,10 +120,11 @@ class PycodeError(Exception):
             res += ' (exception was: %r)' % self.args[1]
         return res
 
-
+import weakref
 class ModuleAnalyzer(object):
     # cache for analyzer objects -- caches both by module and file name
-    cache = {}
+#    cache = {}
+    cache = weakref.WeakValueDictionary()
 
     @classmethod
     def for_string(cls, string, modname, srcname='<string>'):

This does reduce build time and memory usage, but I don't know if the output is identical.

[mpatel]

For sage -docbuild reference html --jsmath, at least, the results are close:

$ du -s output_v0*
95550   output_v051
97392   output_v062
97388   output_v062_mod

The differences are in environmental.pickle or stem from using a function as a default argument in

• combinat/words/word_content.py
• rings/polynomial/groebner_fan.py

[mpatel]

Clearly, it's time for a break.

[jhpalmieri]

Nice observation. Making the change with weak_references does indeed speed things up:

0.5.1     28 minutes
0.6.2     43 minutes
0.6.2     33 minutes   (version using weakref)

With no other changes, it includes all of the private methods, as already pointed out (and I assume it did before using weakref -- this may account for some of the increased time, since it's producing more output), but it also includes (for some classes, anyway), an entry for __weakref)_. See the documentation for sage/algebras/steenrod_algebra.py, for instance: it lists as an attribute of SteenrodAlgebraFactory?

__weakref__
   list of weak references to the object (if defined)

[mpatel]

Actually, noticing __weakref__ for that module is what led to the comment about autodoc-skip-member. I think this was before I "patched" Sphinx, but I may be mistaken. I'm also not sure about why __weakref__ appears spontaneously. Perhaps a super class?

[mpatel]

See #6664 for the autodoc-skip-member issue.

[mpatel]

Also, I've asked about the merits of the weakref workaround on ​sphinx-dev.

[mpatel]

Example inheritance diagram for "The Steenrod algebra". This is just for fun.

[mpatel]

The new ​inheritance diagram extension has potential. I've attached a minimal example. Please do not merge it.

The options are limited, however. For example, I could not find a directive that auto-generates a diagram for each module. Also, it seems there's no way to color-code the nodes/classes, e.g., to indicate their modular provenance. Maybe it's possible to do one or more of these ourselves, in builder.py, conf.py, or by patching Sphinx.

[mpatel]

I should add that this requires a local installation of ​GraphViz.

[mpatel]

Replying to mpatel:

Also, I've asked about the merits of the weakref workaround on ​sphinx-dev.

The ​thread. Preliminary tests indicate that the proposed alternative does not reduce memory use significantly.

[mpatel]

Replying to mpatel:

Replying to mpatel:

Also, I've asked about the merits of the weakref workaround on ​sphinx-dev.

The ​thread. Preliminary tests indicate that the proposed alternative does not reduce memory use significantly.

​Here, actually.

[mpatel]

There's been no further activity about the workaround on sphinx-dev. Since weakrefs appear to work for us, should we add a patch to the spkg?

[mvngu]

Replying to mpatel:

There's been no further activity about the workaround on sphinx-dev. Since weakrefs appear to work for us, should we add a patch to the spkg?

Hmm... could you make an spkg with that workaround?

[jhpalmieri]

Replying to mvngu:

Replying to mpatel:

There's been no further activity about the workaround on sphinx-dev. Since weakrefs appear to work for us, should we add a patch to the spkg?

Hmm... could you make an spkg with that workaround?

I can certainly do it, but not until Monday.

[mpatel]

Does the problem described ​here recur with the new spkg?

[jhpalmieri]

The new spkg is at

​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p1.spkg

(The old one is still there; it's "...p0.spkg", as listed in the description above.)

[jhpalmieri]

By the way, note that since this includes the private methods in the reference manual, there are a lot of new warnings when building that document. (People haven't written all of the docstrings for their __init__ methods, etc., to conform with proper reST format, since they didn't need to before.) These should be dealt with in a separate ticket: otherwise, we might feel obliged to keep rebasing this ticket until it was merged.

[mpatel]

Does #6664 suppress the private methods?

[jhpalmieri]

Does #6664 suppress the private methods?

Yes, but do we *want* to suppress the private methods? I actually prefer that they be included in the reference manual.

[jhpalmieri]

Uh oh, it seems as though with this spkg, changes to pyx files are not detected. That is, build the documentation (say, with the jsmath option). Then edit a .py file and a .pyx file (I chose sage/misc/latex.py and sage/matrix/matrix0.pyx). Run 'sage -docbuild reference print_modified_modules': the .py file will be listed, but not the .pyx file. I can't figure out why this is happening :(

[mpatel]

I also don't know why. builder.py uses Sphinx's get_outdated_files() method to get an iterable of modified modules. However, this method's code is the same in v0.5.1 and v0.6.2. I did find that the newer Sphinx does not include .so files for .pyx files in environment.pickle's dependencies dictionary. This may "explain" the observed behavior.

[jhpalmieri]

I played around with the Sphinx code for a while, and I could not figure out what's going on. Well, I believe that "matrix0.so" was listed as a dependency of "matrix0.pyx" in version 0.5.1, so changing the .pyx file resulted in the modification time for the .so file to be wrong, triggering a rebuild of the docs, but in 0.6.2, the .so file no longer appears to be a dependency of the .pyx file. I've posted a message to sphinx-dev about how dependencies are determined; we'll see if there are any helpful responses.

[jhpalmieri]

Okay, I now have a patch which seems to recognize changes to pyx files. The dependency information is stored in the pickle, so to test this, first delete SAGEDOC/output/doctrees/. The new spkg is available from ​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p2.spkg.

Please test thoroughly. (One thing to watch out for: I thought it wasn't working because I was changing sage/matrix/action.pyx and seeing no notification when the reference manual was built, but then I realized that action.pyx isn't even included in the reference manual. Doh.)

[jhpalmieri]

In case anyone downloaded the version posted yesterday, there is a slightly modified version today: the author of Sphinx posted a patch for the Cython problem to the ​Sphinx mercurial site, and so the version today uses that patch instead. (Mine was similar to his, but why not use the official one?)

I've replaced the old one with the new one, so the link is the same: ​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p2.spkg.

[jhpalmieri]

Okay, here's another one. Sphinx was just updated to version 0.6.3, and this is supposed to fix our problems. In my testing, it doesn't have the Cython problem that the original version of 0.6.2 had; it incorporates the patch in the most recent 0.6.2 spkg. The 0.6.3 spkg is a bit slower than 0.5.1 and also than our version of 0.6.2 with the unauthorized weakref patch. It's not nearly as slow and doesn't seem to use nearly as much memory as the unpatched 0.6.2. Here's the spkg: ​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p0.spkg.

Some timings in minutes for "sage -docbuild reference html --jsmath", with the patch at #6664 applied:

                    Mac      sage.math  ubuntu
Sphinx 0.5.1         6           5        10
Sphinx 0.6.2.p2      6           6        10
Sphinx 0.6.3         8           9        11

"Mac" is my Intel iMac running OS X 10.5; "sage.math" is sage.math, and "ubuntu" is an old ubuntu box I have access to.

As I said above, the memory usage seems to stay within reason for all of these, but I'm not using anything sophisticated to test this: with the terrible version, my computer would become very sluggish, and with the versions tested here, that doesn't happen.

[mpatel]

For Sage 4.1.2.alpha0's HTML reference manual, at least, the new spkg seems to work just as Georg indicated. I'll try to do more testing soon.

By the way, it might be useful to update spkg-install along the lines of #6598 and ​this example. For example,

#!/usr/bin/env bash

if [ -z "$SAGE_LOCAL" ]; then
   echo "SAGE_LOCAL undefined ... exiting";
   echo "Maybe run 'sage -sh'?"
   exit 1
fi

success() {
    if [ $? -ne 0 ]; then
        echo "Error building Sphinx: '$1'"
        exit 1
    fi
}

set -e

echo "Removing old version..."
rm -rf "$SAGE_LOCAL"/lib/python/site-packages/Sphinx-*
success 'deleting previous version'

[...]

[jhpalmieri]

Replying to mpatel:

By the way, it might be useful to update spkg-install along the lines of #6598 and ​this example. For example,

Okay, try this one: ​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg. (Putting "set -e" in other places made the script exit without printing the custom error message. With this one, if, for example, I make the directory "$SAGE_LOCAL"/lib/python/site-packages/Sphinx-* undeletable, I get the appropriate message.)

[mpatel]

Thanks very much for pointing that out. I may need to update the spkgs at #5447...

[timdumol]

Forgive me if I'm mistaken, but doesn't Sphinx>=0.6 depend on Jinja2?

Incompatible changes:

• Templating now requires the Jinja2 library, which is an enhanced version of the old Jinja1 engine.

​http://sphinx.pocoo.org/changes.html#release-0-6-mar-24-2009

[mpatel]

Yes. I think site-packages/Jinja2-2.1.1-py2.6-linux-x86_64.egg, say, is installed automatically (via network) by python setup.py install in spkg-install.

Should we make it a separate spkg, to supplement or replace jinja-1.2.p0.spkg?

[mpatel]

I checked that Sphinx now incorporates changes to included .py and .pyx files.

For the HTML builder, both jsmath and pngmath modes appear to work as expected.

The PDF reference manual's table of contents is missing. This may stem from #6885 and my invoking LaTeX's batch mode with b.

[timdumol]

Replying to mpatel:

Yes. I think site-packages/Jinja2-2.1.1-py2.6-linux-x86_64.egg, say, is installed automatically (via network) by python setup.py install in spkg-install.

Should we make it a separate spkg, to supplement or replace jinja-1.2.p0.spkg?

I think it would probably best to make a separate spkg so that installation will not require network access. I've made an spkg that I can post if you want.

Also, the spkg for this states that it depends on Jinja>=1.1, which is no longer the case.

[jhpalmieri]

Okay, I've updated the file SPKG.txt (in ​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg) to say that it depends on Jinja >= 2. If you want to post a link to an spkg for jinja, that would be great.

[timdumol]

Here you go: ​http://drop.io/jinja2_spkg

[jhpalmieri]

A few issues with the Jinja spkg: the source is under version control but it shouldn't be. When I install it, I get jinja2-2.1.1, not jinja2-2.2, so the name is wrong. Also, the part of the spkg-install file which is supposed to remove the old version wasn't removing the right thing (it said "Jinja-2*", but the directory is actually "Jinja2-2.1.1*").

Anyway, here's a modified spkg, much smaller since src is not in the mercurial repository. This is an updated version of the version 1 jinja spkg, with the new src dropped in, replacing the old src: ​http://sage.math.washington.edu/home/palmieri/SPKG/jinja2-2.1.1.p0.spkg.

Should this have its own ticket? Will it screw up anything from #6568? I don't know anything about Jinja, so I don't know what to do to test it; all I see is that the docs are built correctly with the new versions of Sphinx and Jinja.

[timdumol]

Thanks for the fixes.

Jinja and Jinja2 are totally different, and so it shouldn't affect any current templating routines. In any case, once Sphinx 0.6.3 is included, we can remove the dependency on Jinja by replacing all instances of import jinja with import jinja2, etc. The API's are pretty much unchanged, and from what I know, none of the changed interfaces are currently used.

[cremona]

I'm not at all sure what you people are doing here, so perhaps I should have kept away, but (in trying to help with #6820) I tried installing the new sphinx spkg using

./sage -f http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.2.p0.spkg

and now

john@ubuntu%sage -docbuild reference html
Traceback (most recent call last):
  File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 667, in <module>
    getattr(get_builder(name), type)()
  File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 348, in _wrapper
    for module_name in self.get_modified_modules():
  File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 412, in get_modified_modules
    env = self.get_sphinx_environment()
  File "/home/john/sage-4.1.1/devel/sage/doc/common/builder.py", line 403, in get_sphinx_environment
    return BuildEnvironment.frompickle(config, env_pickle)
  File "/home/john/sage-4.1.1/local/lib/python2.6/site-packages/Sphinx-0.6.2-py2.6.egg/sphinx/environment.py", line 210, in frompickle
    env = pickle.load(picklefile)
AttributeError: 'module' object has no attribute 'RedirStream'

This is in the main branch, so a vanilla 4.1.1 build. Should that be 4.1.2.alpha0?

[timdumol]

Hey.

The first spkg had problems. There are newer spkgs. ​http://sage.math.washington.edu/home/palmieri/SPKG/sphinx-0.6.3.p1.spkg is the latest.

[jhpalmieri]

To cremona: sorry about that. When you switch versions of Sphinx, you have to delete the old output in the docs directory -- it caches some information there, apparently in ways which are not compatible between versions. So after you install the spkg, delete the directory SAGE_ROOT/devel/sage/doc/output and try again. I should change the summary on this ticket to reflect this...

To timdumol: I see, jinja and jinja2 can coexist on the same system, so for now we can have both. That's the best solution now, I agree. (I tried replacing all of the "import jinja" lines with "import jinja2", but there was a problem with one of them: in sage.ext.gen_interpreters, the line from jinja.datastructure import ComplainingUndefined doesn't translate, because there is no jinja2.datastructure.)

[cremona]

Thanks -- that worked! (Except that I cannot read and deleted all SAGE_ROOT/devel/sage/doc and not just the output directory, so had to copy the rest from another clone...)

I like the percentage meter and the fact that the output overwrites one line rather than running off the screen. Minor point here: it does not overwrite long lines when they are followed by short ones so you see strange garbage characters on the end of te line like this:

reading sources... [ 49%] sage/misc/explain_pickle      _densensephismsmpecies

Of course this is a triviality.

Otherwise all went well and (from the point of view of an ordinary user/developer) I'm happy with this spkg. (There are lots of warnings and at least one error in the docstrings, but that is another matter). I have no objection to this getting a positive review, assuming tha the other issues discussed above have been dealt with.

[jhpalmieri]

new version the file spkg/install (incorporating jinja2)

[jhpalmieri]

new version of the file spkg/standard/deps (incorporating jinja2)

[jhpalmieri]

diff between old and new install

[jhpalmieri]

diff between old and new deps

[mpatel]

Both the latest Sphinx and Jinja2 packages appear to install without network activity. The changes to install and deps look good to me, but I haven't done a complete installation from scratch.

To the extent that it counts, my review is positive.

[timdumol]

Documentation builds perfectly. Sphinx 0.6.3 takes only 2 minutes more sys time than 0.5.1 to build all the docs, for me.

Haven't done a complete install yet, but I'll do so once I finish downloading the 4.1.2.alpha2 tarball.

[mpatel]

I updated spkg/install and spkg/standard/deps, replaced the old Sphinx spkg with the new one in spkg/standard, and copied the Jinja2 package to spkg/standard. With these changes, sage -bdist 100 generates the expected binary distributions on sage.math from both binary and compiled source distributions. Moreover, sage -sdist 200 makes a source distribution that compiles as expected.

There's a failed test:

mpatel@sage sage-200$ ./sage -t devel/sage/sage/misc/trace.py 
sage -t  "devel/sage/sage/misc/trace.py"                    
**********************************************************************
File "/scratch/mpatel/source/sage-200/devel/sage/sage/misc/trace.py", line 61:
    sage: print s.before[s.before.find('-'):]
Expected:
    ---...
    ipdb> c
    2 * 5
Got:
    --Call--
    > /scratch/mpatel/source/sage-200/local/lib/python2.6/site-packages/sage/rings/arith.py(1944)factor()
       1943 
    -> 1944 def factor(n, proof=None, int_=False, algorithm='pari', verbose=0, **kwds):
       1945     """
    
    ipdb> c
    2 * 5
    <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of  10 in __main__.example_1
***Test Failed*** 1 failures.
For whitespace errors, see the file /scratch/mpatel/source/sage-200/tmp/.doctest_trace.py
         [3.7 s]
exit code: 1024
 
----------------------------------------------------------------------
The following tests failed:


        sage -t  "devel/sage/sage/misc/trace.py"
Total time for all tests: 3.7 seconds

I think it's not related to this ticket.

[mpatel]

Problem: Somehow, Sphinx 0.6.3 breaks the sidebar toggle (cf. #6507) in the live docs. In Firebug's console:

unexpected end of XML source
<span class="math">('#searchbox').show(0);

[mpatel]

Patching Sphinx 0.6.3 should fix this and perhaps also a similar problem reported on ​sage-devel:

sphinx/themes/basic/layout.html

@@ -81 +81 @@
               {{ _('Enter search terms or a module, class or function name.') }}
               </p>
           </div>
-          <script type="text/javascript">$('#searchbox').show(0);</script>
+          <script type="text/javascript">jQuery('#searchbox').show(0);</script>
           {%- endif %}
           {%- endblock %}
         </div>

[mpatel]

Or perhaps not.

[timdumol]

mpatel: The problem with the sidebar is caused by compression. Line comments ("") are used in the sidebar code, therefore when the code is compressed, the entire function is commented out. Changing the line comments to block comments ("/* */") fixed the problem for me.

[timdumol]

A patch for the doc sidebar problem is now up at #7126.

[mpatel]

If this comment at #7126 is correct, we should still patch sphinx/themes/basic/layout.html. I can do this.

[timdumol]

Replying to mpatel:

Problem: Somehow, Sphinx 0.6.3 breaks the sidebar toggle (cf. #6507) in the live docs. In Firebug's console:

unexpected end of XML source
<span class="math">('#searchbox').show(0);

#7141 adds a patch that fixes this.

[mhansen]

Merged both the jinja2 and sphinx in spkg.