Opened 10 years ago

Closed 9 years ago

#7549 closed enhancement (fixed)

Reference manual: class hierarchy, inherited members, underscore variables

Reported by: mpatel Owned by: mvngu
Priority: minor Milestone: sage-4.4
Component: documentation Keywords:
Cc: hivert, jhpalmieri, nthiery, mvngu, jason Merged in: sage-4.4.alpha0
Authors: Mitesh Patel Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

In the Sage reference manual, it might be useful to

  • List the base classes for a class.
  • List inherited members but not their docstrings.
  • Include docstrings for "private" variables (e.g., __init__, _foo).
  • Include inheritance diagrams.

Sphinx extensions of interest: autodoc, inheritance_diagram.

See sage-devel for some discussions.

Attachments (8)

trac_7549-doc_inheritance_underscore.patch (7.4 KB) - added by mpatel 10 years ago.
Doc build options for inherited members, private variables. Inheritance diagrams. Apply to sage repo.
trac_7549-doc_inheritance_underscore_v2.patch (8.8 KB) - added by mpatel 10 years ago.
Set up intersphinx for Python docs. Replaces previous patch. sage repo.
trac_7549-doc_inheritance_underscore_v3.patch (8.9 KB) - added by mpatel 10 years ago.
Rebased for 4.3.alpha1. Replaces previous patches.
trac_7549-doc_inheritance_underscore_v4.patch (9.4 KB) - added by mpatel 9 years ago.
Rebased vs. 4.3.3.alpha0. See comment. Apply only this patch. sage repo.
trac_7549-doc_inheritance_underscore_v5.patch (9.0 KB) - added by mpatel 9 years ago.
Rebase vs. 4.3.4.alpha1 + #8457. Apply only this patch.
trac_7549-small-fixes.patch (850 bytes) - added by jhpalmieri 9 years ago.
apply on top of other patch
trac_7549-doc_inheritance_underscore_v6.patch (9.8 KB) - added by mpatel 9 years ago.
Fix recursion problem. Apply only this combined patch.
trac_7549-doc_inheritance_underscore_v7.patch (9.9 KB) - added by mpatel 9 years ago.
Help notes. Replaces V6.

Download all attachments as: .zip

Change History (44)

Changed 10 years ago by mpatel

Doc build options for inherited members, private variables. Inheritance diagrams. Apply to sage repo.

comment:1 Changed 10 years ago by mpatel

  • Cc hivert jhpalmieri nthiery added

The patch, which may depend weakly on #7473's "builder" patch,

  • Adds a sage -docbuild option -i to include inherited members, mostly without docstrings.
  • Adds a sage -docbuild option -u to include "private" variables.
  • Uses Sphinx's :show-inheritance: autodoc option to list the base classes of a class.

It's likely that the attached patch needs work. In particular, sage.doc.common.conf.process_inherited would benefit from expert intervention.

Note: To force a rebuild when toggling -u, add -S -aE to the command-line. For example,

sage -docbuild reference html -juv3 -S -aE

Toggling -i should automatically trigger a rebuild.

comment:2 Changed 10 years ago by mpatel

  • Status changed from new to needs_review

comment:3 Changed 10 years ago by mpatel

Running

sage -docbuild reference html -juiv3

I encountered

reading sources... [ 27%] sage/combinat/lyndon_wordtor_weightedturesnssis_basis
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 811: ordinal not in range(128)

Are #6674 and #6682 relevant?

comment:4 Changed 10 years ago by mpatel

  • Cc mvngu added

Putting

# -*- coding: utf-8 -*-                                                         

at the beginning of sage.combinat.lyndon_word seems to placate Sphinx.

Should we do this with every Sage library .py and .pyx file?

comment:5 follow-up: Changed 10 years ago by jhpalmieri

After adding the "utf-8" line to lyndon_word.py, I got this error:

reading sources... [ 51%] sage/interfaces/singular                 ismsm
reST markup error:
/Applications/sage/local/lib/python2.6/site-packages/sage/interfaces/singular.py:docstring of sage.interfaces.singular.SingularFunctionElement._sage_doc_:8: (SEVERE/4) Unexpected section title.

After fixing this (by ripping out a doctest), things built fine. The html version of the reference manual looks good, and seems to have all of the methods we want, with the inherited ones having no docstrings.

Here are some timings on my iMac. Between each build, I removed the directory doc/output:

sage -docbuild reference html -j        11 minutes
sage -docbuild reference html -ji       22 minutes
sage -docbuild reference html -ju      13 minutes
sage -docbuild reference html -jiu     63 minutes!

Also

sage -docbuild reference pdf -iu  

took too long on my iMac. On sage.math, it took about 2 hours and then bombed when trying to run pdflatex: there was one error (a missing "r" before a triple-quoted docstring with backslashes); once this was fixed, it ran for a while and then said

! TeX capacity exceeded, sorry [pool size=1165810].

I don't know if there is a way of fixing this, short of #6495. (I think that the tex installation on sage.math is somewhat old, which doesn't help.)

comment:6 Changed 10 years ago by jhpalmieri

I have yet to succeed making a pdf version of the reference manual. On the two machines I've tried, it takes ages and then crashes with "TeX capacity exceeded". This happened just with the "-i" option on my iMac. I haven't tried all of the possibilities on all of the machines.

So I wonder if it makes sense, at least until something like #6495 is in place, to disable the "-i" and "-u" flags for the pdf version of the manual. If not disable them, then at least print a warning saying it's going to take a long time and then might very well crash?

comment:7 Changed 10 years ago by mpatel

Sure, I have no problem with either course. We could also make -i and -u "Advanced" options.

Aside: I can't even build the HTML manual with -iu on my local machine. It requires too much memory.

comment:8 Changed 10 years ago by mpatel

Maybe we can use Sphinx's intersphinx extension to implement a HTML analogue of #6495?

comment:9 Changed 10 years ago by mpatel

For links to Sage tickets, we could use Sphinx's extlinks extension.

comment:10 in reply to: ↑ 5 Changed 10 years ago by hivert

Also

sage -docbuild reference pdf -iu  

took too long on my iMac. On sage.math, it took about 2 hours and then bombed when trying to run pdflatex: there was one error (a missing "r" before a triple-quoted docstring with backslashes); once this was fixed, it ran for a while and then said

! TeX capacity exceeded, sorry [pool size=1165810].

I don't know if there is a way of fixing this, short of #6495. (I think that the tex installation on sage.math is somewhat old, which doesn't help.)

Did you try to launch lex with more memory ? I remember doing a

export extra_mem_bot=8000000; latex ...

in a similar situation.

Changed 10 years ago by mpatel

Set up intersphinx for Python docs. Replaces previous patch. sage repo.

comment:11 Changed 10 years ago by mpatel

  • Authors set to Mitesh Patel

Version 2, which replaces the previous patch,

  • Enables intersphinx for the Python docs. This is easy to use, e.g.,
    For more information, please consult :mod:`subprocess`.  Blah, blah, whatever...
    
  • Adds # -*- coding: utf-8 -*- to the top of sage.combinat.lyndon_word.
  • Drops a sage.interfaces.singular.SingularFunctionElement._sage_doc_ doctest.
  • Prepares for extlinks. Sphinx 1.0 will include this new extension.

comment:12 Changed 10 years ago by mpatel

It seems that :show-inheritance: shows just the immediate super classes, not the entire hierarchy. I do not know whether this is easy to change.

comment:13 Changed 10 years ago by mpatel

Please see #6495 for progress on incrementally building the reference manual.

Changed 10 years ago by mpatel

Rebased for 4.3.alpha1. Replaces previous patches.

comment:14 follow-up: Changed 10 years ago by mpatel

  • Cc jason added

I should add that we can also link directly to matplotlib's docs from the Sage documentation. We just need to update doc.common.conf.intersphinx_mapping (the object inventory is here). Perhaps there are others of interest in this list?

comment:15 Changed 9 years ago by mpatel

Against 4.3.2.alpha1:

applying trac_7549-doc_inheritance_underscore_v3.patch
patching file doc/common/conf.py
Hunk #5 succeeded at 378 with fuzz 2 (offset 5 lines).
now at: trac_7549-doc_inheritance_underscore_v3.patch

comment:16 in reply to: ↑ 14 Changed 9 years ago by jhpalmieri

Replying to mpatel:

I should add that we can also link directly to matplotlib's docs from the Sage documentation. We just need to update doc.common.conf.intersphinx_mapping (the object inventory is here). Perhaps there are others of interest in this list?

Cython, mpmath, and Sphinx are possibilities. Building Sage shouldn't rely on an internet connection, though, and it seems to me that intersphinx does. So should we also include the object inventories so that we can build the docs without an internet connection? This would rely on keeping those up to date. Can we have it check for an internet connection and fail gracefully if there isn't one?

Right now I don't know what I'm doing wrong, but I can't get private methods to be included in the docs. This is with 4.3.2. I put in a print statement in skip_underscore to make sure it was finding the correct methods, and it was. But I didn't see them in the documentation afterwards. I'll try again later.

Changed 9 years ago by mpatel

Rebased vs. 4.3.3.alpha0. See comment. Apply only this patch. sage repo.

comment:17 Changed 9 years ago by mpatel

V4

  • Rebased vs. 4.3.3.alpha0.
  • Combines the skip member handlers, since Sphinx takes the first answer that's not None. (I suppose we could keep them separate but return None when a test is inconclusive).
  • Detects changes in -u, too. Toggling either new option should trigger a rebuild.

Should we move the new options to the "Advanced" section? Add "may use lots of memory"?

Note: I still haven't done a complete build in each configuration.

comment:18 Changed 9 years ago by mpatel

I should add that I've disabled the intersphinx extension, for now. It can cache inventories, but I don't know whether it fails gracefully if the cache has "expired" but there's no connection.

comment:19 Changed 9 years ago by mpatel

The HTML manual builds with -i, -u, and -iu.

Building the PDF manual with -u on sage.math yields

[4811] [4812] [4813] [4814] [4815] [4816]
! TeX capacity exceeded, sorry [pool size=1165811].
\Hy@pstringdef ...->\edef #1{\pdfescapestring {#2}
                                                  }
l.392012 ...matrix0.Matrix.multiplicative_order}{}
                                                  \begin{methoddesc}{multipl...

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on reference.log.
make: *** [reference.pdf] Error 1

after I suppress problems with underscored methods in

  • sagenb.notebook.worksheet
  • structure.formal_sum
  • sage.combinat.crystals.tensor_product
  • sage.combinat.species.set_species
  • sage.combinat.words.word_generators
  • sage.rings.number_field.number_field
  • sage.rings.number_field.number_field_ideal
  • sage.rings.polynomial.multi_polynomial_libsingular
  • sage.rings.polynomial.infinite_polynomial_ring
  • sage.rings.polynomial.infinite_polynomial_element

I'll fix the first problem in a separate ticket and leave the others for someone else.

comment:20 follow-up: Changed 9 years ago by jhpalmieri

I get the html manual to build, too, although with 1517 warnings when built with "-u", and more than 28,000 warnings with "-iu": either because the docstrings for underscore methods are sloppily formatted (not a surprise) or messages along the lines of those in #8244.

I noticed that after I built with "-j -iu", edited a few files and ran "sage -b", then when I built the manual again with "-j -iu", in the "reading sources" portion of the build, it only processes the changed files, but in the "writing output" phase, it rewrites all of the files. This could be a little annoying; can it be avoided?

I haven't had the time to try a pdf build yet, but I'll work on that later today.

comment:21 Changed 9 years ago by jhpalmieri

Now I've tried building the pdf, and I run into the same "TeX capacity exceeded" message. I've tried altering "extra_mem_top" and "extra_mem_bot", but it's not helping. (I may not be doing this right, though.)

I had to fix more problems to get the PDF to build: everywhere there was an email address, pdflatex was crashing on the @ sign, so I had to enclose the address in double backquotes. So I ended up changing these files (in addition to sagenb.notebook.worksheet: I applied the patch from #8265 to deal with this one):

  • sage/combinat/species/set_species.py
  • sage/combinat/words/word_generators.py
  • sage/misc/hg.py
  • sage/misc/lazy_attribute.py
  • sage/misc/preparser.py
  • sage/rings/number_field/number_field.py
  • sage/rings/number_field/number_field_ideal.py
  • sage/rings/padics/padic_generic.py
  • sage/rings/padics/tutorial.py
  • sage/rings/polynomial/infinite_polynomial_element.py
  • sage/rings/polynomial/infinite_polynomial_ring.py
  • sage/rings/polynomial/multi_polynomial_libsingular.pyx
  • sage/rings/polynomial/pbori.pyx
  • sage/rings/polynomial/symmetric_ideal.py
  • sage/rings/polynomial/symmetric_reduction.pyx
  • sage/structure/formal_sum.py
  • sage/structure/sequence.py

I don't think that fixing these is a high priority, since the PDF manual doesn't build anyway with the "-u" option.


Aside from my question about why "-iu" rewrites all of the output when it shouldn't have to, I'm happy with this. (I haven't tested this with "-i" or "-u".) I agree that there should be a comment about "-i" and "-u" taking a lot of memory, and perhaps saying that they don't work with the pdf build. I don't think they need to be in the "advanced" options.

comment:22 Changed 9 years ago by hivert

Hi !

Just to make you aware of #7448 (Sphinx rendering of nested classes) which is related. Notice that our patches certainly do not commute but rebasing should be trivial.

Cheers,

Florent

comment:23 in reply to: ↑ 20 ; follow-up: Changed 9 years ago by mpatel

Replying to jhpalmieri:

I noticed that after I built with "-j -iu", edited a few files and ran "sage -b", then when I built the manual again with "-j -iu", in the "reading sources" portion of the build, it only processes the changed files, but in the "writing output" phase, it rewrites all of the files. This could be a little annoying; can it be avoided?

I've also seen this behavior, but I've been unable reproduce it with certainty --- at least, with a version of #6187's "testreference" patch. Are there particular types of changes that usually trigger the problem?

I don't think Sphinx is especially smart about updating files affected by a change in other files, but this may be irrelevant.

comment:24 Changed 9 years ago by mpatel

To do: Rebase the patch against 4.3.3.alpha1 + #8244 + #7448.

comment:25 in reply to: ↑ 23 Changed 9 years ago by jhpalmieri

Replying to mpatel:

Replying to jhpalmieri:

I noticed that after I built with "-j -iu", edited a few files and ran "sage -b", then when I built the manual again with "-j -iu", in the "reading sources" portion of the build, it only processes the changed files, but in the "writing output" phase, it rewrites all of the files. This could be a little annoying; can it be avoided?

I've also seen this behavior, but I've been unable reproduce it with certainty --- at least, with a version of #6187's "testreference" patch. Are there particular types of changes that usually trigger the problem?

I'm not sure, but I just saw this message at the beginning of a docbuild:

WARNING: unsupported build info format in '/Applications/sage/devel/sage/doc/output/html/en/reference/.buildinfo', building all

comment:26 Changed 9 years ago by mpatel

I've seen this message, too. It might be a Sphinx bug.

The HTML builder writes a hash of the html_* settings to .buildinfo in the output directory. It appears that StandaloneHTMLBuilder.get_outdated_docs contains nearly all of the relevant code.

If, for some reason, this get_outdated_docs doesn't get called, the builder writes empty strings for the hashes. For example, sage -docbuild reference html -j -S -a does this (-a tells Sphinx to write all files). The next build attempt triggers the warning.

But I don't know if this is responsible for what we see.

Changed 9 years ago by mpatel

Rebase vs. 4.3.4.alpha1 + #8457. Apply only this patch.

comment:27 Changed 9 years ago by mpatel

V5 is rebased for 4.3.4.alpha1 + #8457. I'm not sure what we should do about the many -iu warnings. Should we handle them in sage_getargspec or in conf?

comment:28 Changed 9 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

If I run sage -docbuild reference html -i, I get

reading sources... [  8%] sage/algebras/iwahori_hecke_algebra_element
Exception occurred:
  File "/Applications/sage/devel/sage/doc/common/conf.py", line 354, in skip_member
    if 'SAGE_CHECK_NESTED' in os.environ:
RuntimeError: maximum recursion depth exceeded
The full traceback has been saved in ...

I scanned the traceback but didn't see anything obviously helpful.

By the way, I think we can put back the doctest from interfaces/singular.py, by changing the triple quotes at the beginning of the docstring to r""". The attached patch does this.

Also, I used all of the extra error messages to figure out where the warnings

<autodoc>:0: (ERROR/3) Unexpected indentation.

were coming from, and the attached patch fixes the problems. So now, combined with #8457, I get no error messages when building the reference manual (without the -i or -u switches, of course).

For the new warnings when building with -i or -u or -iu, I say we deal with them on another ticket. It's a big job revising all of the docstrings for underscore functions so that they are in proper reST format. (Also, right now they're useful -- see the previous paragraph. :)

Finally, do we want some sort of warning for the -i switch, and possibly the -u switch? ("may be slow, may not be compatible with PDF output" as part of its help string, for example)?

comment:29 Changed 9 years ago by jhpalmieri

(You may need to apply #8511 to get rid of one of the "unexpected indentation" warnings.)

comment:30 Changed 9 years ago by jhpalmieri

On second thought, I moved the "unexpected indentation" fix to ticket #8511, so the "small fixes" patch just reinstates the interfaces/singular.py doctest.

Changed 9 years ago by jhpalmieri

apply on top of other patch

Changed 9 years ago by mpatel

Fix recursion problem. Apply only this combined patch.

comment:31 follow-up: Changed 9 years ago by mpatel

  • Authors changed from Mitesh Patel to Mitesh Patel, Florent Hivert
  • Status changed from needs_work to needs_review

I broke the module check in #7448's V4 when I made #7448's V5. I apologize for that. I've restored Florent's code and folded in the small fixes patch in this ticket's V6.

comment:32 Changed 9 years ago by mpatel

The only change is in sage_autodoc.py.

Changed 9 years ago by mpatel

Help notes. Replaces V6.

comment:33 in reply to: ↑ 31 Changed 9 years ago by hivert

  • Authors changed from Mitesh Patel, Florent Hivert to Mitesh Patel

Replying to mpatel:

I broke the module check in #7448's V4 when I made #7448's V5. I apologize for that. I've restored Florent's code and folded in the small fixes patch in this ticket's V6.

Hi Mitesh ! I don't see any code from me here except a few lines who where already in #7448, and which you decide to improve (thanks by the way). So I don't think I deserve to be in the author.

Also I'm Ok with reviewing this but they are three more tickets in combinatoric which I committed myself to review. So don't rely on me for reviewing this very soon... However, I'd like to see this merged soon, so if you imagine someone who is willing to review, please ask him. Anyway, thanks for this work. I really this sage's doc is going to greatly improve.

comment:34 Changed 9 years ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review

Sorry for not getting to this sooner. Positive review.

comment:35 Changed 9 years ago by jhpalmieri

  • Milestone set to sage-4.4

comment:36 Changed 9 years ago by jhpalmieri

  • Merged in set to sage-4.4.alpha0
  • Resolution set to fixed
  • Status changed from positive_review to closed

Merged in 4.4.alpha0:

  • trac_7549-doc_inheritance_underscore_v7.patch
Note: See TracTickets for help on using tickets.