Extend the capabilities of Sage's introspection

[SimonKing]

The following used to fail:

sage: x??
Error getting source: could not find class definition
sage: P.<x,y> = QQ[]
sage: P??
Error getting source: could not find class definition
sage: I??
Error getting source: could not find class definition

Similarly, the edit command did not work in these case.

With my patch, all that works. Moreover, both sage.misc.edit_module and sage.misc.sageinspect are put into the reference manual.

Note that this may be related with #11287: sage.misc.sage_getfile and sage_getsourcelines could be a reliable tool to get information on how and from where to import a given object. Therefore Cc to the participants of that ticket.

Moreover, Cc to #9976. Reason:

Depends on #9976

Apply

• trac11298_introspection_and_edit.patch​
• trac11298_singular_standard_options.rebased.patch​
• trac_11298-ref.patch​

[SimonKing]

For the patchbot:

Depends on #9976

[SimonKing]

Correcting a misspelling.

Depends on #9976

[SimonKing]

Note that, with the patch, functools.partial objects can be reasonably edited as well:

edit(sage.combinat.partition_algebra.SetPartitionsAk,'vim')

opens vim at line 31 of sage/combinat/partition_algebra.py:

def create_set_partition_function(letter, k):
    """
    EXAMPLES::
    
        sage: from sage.combinat.partition_algebra import create_set_partition_function
        sage: create_set_partition_function('A', 3)
        Set partitions of {1, ..., 3, -1, ..., -3}
    """
    from sage.functions.all import floor
    if isinstance(k, (int, Integer)):
        if k > 0:
            return globals()['SetPartitions' + letter + 'k_k'](k)
    elif is_RealNumber(k):
        if k - math.floor(k) == 0.5:
            return globals()['SetPartitions' + letter + 'khalf_k'](floor(k))

    raise ValueError, "k must be an integer or an integer + 1/2"

and indeed that is the underlying definition of SetPartitionsAk; see line 53, which is

SetPartitionsAk = functools.partial(create_set_partition_function,"A")

[SimonKing]

I found one more case where source code inspection failed: the (lib)singular_standard_options wrapper.

With the new patch, we have

sage: P.<x,y> = QQ[]
sage: I = P*[x,y]
sage: edit(I.interreduced_basis,'vim')
    ...
    @singular_standard_options
    @libsingular_standard_options
    def interreduced_basis(self):
        r"""
        If this ideal is spanned by `(f_1, ..., f_n)` this method
        returns `(g_1, ..., g_s)` such that:
        
        - `(f_1,...,f_n) = (g_1,...,g_s)`
        
        - `LT(g_i) != LT(g_j)` for all `i != j`
   ...

which would previously just show the code of the wrapper, not of the wrapped method.

I wonder why the patchbot complains. In particular, why does the patchbot mention my patch from trac ticket #11115 - it shouldn't be a dependency, or should it? Let's see.

For now:

Depends on #9976

[SimonKing]

I just verified: In my patch queue, the patches from #11115 come after trac11298_introspection_and_edit.patch​

In other words, the behaviour of the patchbot seems very strange to me.

[SimonKing]

I still think that the patchbot goofed it, but I did as well: The old version of the second patch would really be on top of #11115 (but not the first). I was just updating it, so that we have:

Depends on #9976

and nothing else. Could be that I need to fix some tests, though...

[SimonKing]

Adding "r" to one doc string (for making the documentation nicely formatted) and correcting one doc test.

The tests in sage.misc and in sage.rings.polynomial all pass. Next, I'll run long tests.

Depends on #9976

[SimonKing]

I'll try to trigger the patchbot by changing to "needs work"...

[SimonKing]

... and returning to needs review...

Depends on #9976

[SimonKing]

Extend the capabilities of introspection and interactive source code edition

[SimonKing]

OK, that did not help. Hence, I submitted the patches again. I wish the red blob would finally vanish!

Depends on #9976

[SimonKing]

Ooops, I forgot that since recently there is a form field in which to state dependencies. Let's try it!

[SimonKing]

FWIW, long tests pass for me, and I believe that the patchbot has a serious flaw if it can not even deal with a single dependency.

[SimonKing]

Now I understand what the patchbot is doing (resp. not doing): It believes that #9976 does not need to be applied, since it is merged. But it does not understand that #9976 is merged in 4.7.1.alpha0, not in 4.7.rc1 (which is what it is starting with).

[SimonKing]

I noticed that building the documentation crashes when the second patch is applied. It is as follows:

Running Sphinx v1.0.4
loading pickled environment... done
building [html]: targets for 39 source files that are out of date
updating environment: 0 added, 39 changed, 0 removed
reading sources... [ 33%] sage/rings/polynomial/multi_polynomial_ideal_libsingular                           
Exception occurred:
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/config.py", line 207, in __getattr__
    raise AttributeError('No such config value: %s' % name)
AttributeError: No such config value: autodoc_default_flags
The full traceback has been saved in /tmp/sphinx-err-1tNBmd.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-dev/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
Build finished.  The built documents can be found in /mnt/local/king/SAGE/sage-4.7.rc2/devel/sage/doc/output/html/en/reference

and the traceback is

# Sphinx version: 1.0.4
# Docutils version: 0.5 release
# Jinja2 version: 2.5.5
Traceback (most recent call last):
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/cmdline.py", line 173, in main
    app.build(force_all, filenames)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/application.py", line 207, in build
    self.builder.build_update()
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/builders/__init__.py", line 198, in build_update
    'out of date' % len(to_build))
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/builders/__init__.py", line 218, in build
    purple, length):
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/builders/__init__.py", line 120, in status_iterator
    for item in iterable:
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/environment.py", line 518, in update_generator
    self.read_doc(docname, app=app)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/environment.py", line 658, in read_doc
    pub.publish()
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/core.py", line 204, in publish
    self.settings)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/readers/__init__.py", line 69, in read
    self.parse()
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/readers/__init__.py", line 75, in parse
    self.parser.parse(self.input, document)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/__init__.py", line 157, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 170, in run
    input_source=document['source'])
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/statemachine.py", line 232, in run
    context, state, transitions)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/statemachine.py", line 420, in check_line
    return method(match, context, next_state)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 2658, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 308, in section
    self.new_subsection(title, lineno, messages)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 376, in new_subsection
    node=section_node, match_titles=1)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 266, in nested_parse
    node=node, match_titles=match_titles)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/statemachine.py", line 232, in run
    context, state, transitions)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/statemachine.py", line 420, in check_line
    return method(match, context, next_state)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 2241, in explicit_markup
    self.explicit_list(blank_finish)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 2269, in explicit_list
    match_titles=self.state_machine.match_titles)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 300, in nested_list_parse
    node=node, match_titles=match_titles)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/statemachine.py", line 232, in run
    context, state, transitions)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/statemachine.py", line 420, in check_line
    return method(match, context, next_state)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 2542, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 2251, in explicit_construct
    return method(self, expmatch)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 1994, in directive
    directive_class, match, type_name, option_presets)
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/docutils/parsers/rst/states.py", line 2043, in run_directive
    result = directive_instance.run()
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/ext/autodoc.py", line 1154, in run
    if flag in self.env.config.autodoc_default_flags and \
  File "/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/Sphinx-1.0.4-py2.6.egg/sphinx/config.py", line 207, in __getattr__
    raise AttributeError('No such config value: %s' % name)
AttributeError: No such config value: autodoc_default_flags

I have no idea how this can be debugged.

[SimonKing]

Perhaps it would be worth while to follow a more conceptual approach.

Currently, singular_standard_options and magma_standard_options applied to a method func simply return a function wrapper, that does not know about func (except its doc string, but I think that's a bad idea as well since copying that string costs time). So, introspection does not work. I tried to provide wrapper with the tools needed to make it work, but apparently it was not so easy.

But there is the sage_wraps factory, and I think I should try to use it!

[SimonKing]

Introspection for sage.misc.decorators.sage_wraps, applied for singular_standard_options. Put decorators into references.

[SimonKing]

The second patch has now been updated.

The problem seems solved: I touched all files in sage/rings/polynomial, did sage -b and then built the documentation without problem.

Moreover, I fixed some syntax error in the documentation, and I've put sage.misc.decorators into the reference manual.

So, it is needing review again!

[burcin]

rebased Simon's patch to 4.7.1.alpha1

[SimonKing]

Thank you, Burcin!

For the patchbot, that seems to ignore the ticket description:

Apply trac11298_introspection_and_edit.patch, trac11298_singular_standard_options.rebase4.7.1.a1.patch

[jhpalmieri]

This looks pretty good to me. I have one question: in sage_getsourcelines, lines 1408-1411, you have the code

            except IOError:
                if obj.__class__ != type:
                    return sage_getsourcelines(obj.__class__)
                raise

What is type here? Can you give me an example which will reach this code?

[jhpalmieri]

(Is it just the built-in Python type function?)

[SimonKing]

Replying to jhpalmieri:

This looks pretty good to me. I have one question: in sage_getsourcelines, lines 1408-1411, you have the code ...

What is type here? Can you give me an example which will reach this code?

By type, I mean the type type, such as here:

sage: 1.__class__
<type 'sage.rings.integer.Integer'>
sage: 1.__class__.__class__
<type 'type'>

The idea of the code lines in question is as follows:

Let obj be the object whose source lines we want to determine. It is not necessarily true that the correct source lines for obj coincide with the source lines of its class (notorious counter example: decorated methods). Therefore, we need to try harder.

• First, it is attempted to get the position of the source code from what inspect.getdoc returns (lines 1400-1401)

• Second (if "first" fails), the same is attempted with what _sage_getdoc_unformatted returns (lines 1403-1404). For a reason that I don't understand, one can not replace the first step by the second step: The comment that I inserted states that a mysterious import error would occur at startup.

• Third (if "second" fails as well), it is tested whether the inspect module can provide the source lines of the object obj (line 1407).

• Last resort (if "third" fails), we replace obj by obj.__class__. That is not desired for decorated methods, but these are supposed to provide the source lines in a different way, so that one of the first three methods should succeed. I think as a last resort it is ok... (line 1410).

There is one obvious problem with the "last resort". If the source lines of obj.__class__ can not be found (e.g., if obj.__class__ is defined in an spkg, not in the Sage library) then as a last resort one would try to get obj.__class__.__class__, but that is the type type (see example above), which is defined in the Python spkg and can thus not be found. Infinite recursion.

In order to avoid the infinite recursion, an error is raised if obj.__class__ is the type type, which is, in particular, the case if obj is not a class instance but a class.

I hope that answers your question...

[SimonKing]

Replying to jhpalmieri:

What is type here? Can you give me an example which will reach this code?

A concrete example (using the meataxe wrapper from my cohomology spkg):

sage: from sage.misc.sageinspect import sage_getsourcelines
sage: from pGroupCohomology.mtx import MTX
sage: M = MTX(5,[[1,2],[3,4]])
sage: sage_getsourcelines(M)
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)

/home/king/<ipython console> in <module>()

/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/sage/misc/sageinspect.pyc in sage_getsourcelines(obj, is_binary)
   1407         if pos is None:
   1408             try:
-> 1409                 return inspect.getsourcelines(obj)
   1410             except IOError:
   1411                 if obj.__class__ != type:

/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python/inspect.pyc in getsourcelines(object)
    676     original source file the first line of code was found.  An IOError is
    677     raised if the source code cannot be retrieved."""
--> 678     lines, lnum = findsource(object)
    679
    680     if ismodule(object): return lines, 0

/mnt/local/king/SAGE/sage-4.7.rc2/local/lib/python2.6/site-packages/IPython/ultraTB.pyc in findsource(object)
    175             return lines, candidates[0][1]
    176         else:
--> 177             raise IOError('could not find class definition')
    178
    179     if ismethod(object):

IOError: could not find class definition

Since M can not provide its source lines, the source lines of M.__class__ (i.e., of MTX) are sought.

Since the source lines of MTX can not be found (with the first three methods), the inspect module raises a IOError. It is caught, but reraised, since MTX.__class__ is the type type and hence an infinite recursion must be prevented.

[jhpalmieri]

Okay, I'm happy with this. I'm attaching a referee's patch with a few small changes. If you're happy with my changes, please switch the ticket to "positive review".

This makes a very nice companion to #9976.

[jhpalmieri]

referee patch; apply on top of others

[jhpalmieri]

(The only aspect of the referee's patch worth mentioning is the removal of the import of sage_getsource. This is already done at the top-level of that file, so it doesn't need to be done again.)

[SimonKing]

Replying to jhpalmieri:

Okay, I'm happy with this. I'm attaching a referee's patch with a few small changes. If you're happy with my changes, please switch the ticket to "positive review".

I am happy with the changes. Thus I change it into a positive review. Thank you very much!

[jdemeyer]

This needs to be rebased to #11287.

[jhpalmieri]

After applying the patch at #11287, these still apply for me, although one does have fuzz. Is the fuzz an issue or are patches actually being rejected?

[SimonKing]

<Bump>

Jeroen, could you please elaborate why you think that it should be rebased?

[jhpalmieri]

(If all else fails, we can wait until 4.7.2.alpha0 and rebase it with respect to that.)

[jdemeyer]

Replying to jhpalmieri:

After applying the patch at #11287, these still apply for me, although one does have fuzz. Is the fuzz an issue or are patches actually being rejected?

My personal opinion on applying patches with fuzz is that I allow patches with fuzz 1 but not with fuzz 2. I have seen at least two cases where a "fuzz 2" patch was actually wrongly merged (I don't remember which tickets).

[SimonKing]

Replying to jdemeyer:

My personal opinion on applying patches with fuzz is that I allow patches with fuzz 1 but not with fuzz 2. I have seen at least two cases where a "fuzz 2" patch was actually wrongly merged (I don't remember which tickets).

OK. Then I'll take care of it.

[SimonKing]

By the way: When I tried to import the patch from #11287 on top of sage-4.7.rc2, application to doc/en/reference/mist.rst failed. But if I understood correctly, the problem with the patch here is somewhere else, right? And moreover, I should probably get a more recent Sage version...

[SimonKing]

That's odd!

I applied the patches from #11251, then #11287. Then I had fuzz 2 for one of the patches from #9976 (but they are already merged in 4.7.1.alpha0), and then the patches from here applied cleanly and without fuzz.

So, it seems that I should get the latest rc or alpha version of sage, and try again.

[jdemeyer]

Replying to SimonKing: Yes, sage-4.7.rc2 is in fact quite old, so please try ​sage-4.7.1.rc1 + #11287 (or wait until sage-4.7.2.alpha0 is released and rebase to that).

[SimonKing]

Introspection for sage.misc.decorators.sage_wraps, applied for singular_standard_options. Put decorators into references. Rebased against #11287

[SimonKing]

I've built sage-4.7.1.rc1 and rebased the patch.

Apply trac11298_introspection_and_edit.patch attachment:trac11298_singular_standard_options.rebased.patch​ attachment:trac_11298-ref.patch​

[SimonKing]

The change of the patch concerns doc/en/misc.rst, and the reason for the change was the fact that the context of the inserted line has changed. In particular, the change in the patch has not changed code or doctests. Therefore I believe that it should be reverted to a positive review.

Apply trac11298_introspection_and_edit.patch trac11298_singular_standard_options.rebased.patch trac_11298-ref.patch

[jhpalmieri]

I believe that it should be reverted to a positive review.

I agree. If it's a simple rebase, no new review should be needed. Thanks for taking care of this.