rewrite load and attach

[William Stein]

DEPENDS ON #7483

The goal of this ticket is to rewrite the load and attach commands to eliminate a lot of weirdness:

(1) they should both be usable as normal python functions, so people who find the load <...> form and attach <....> forms weird, can forget they exist. This also makes comprehensive doctesting way easier.

(2) it should be possible to do load DIR+'file.sage' say, and also constructions like

for i in range(3):
    load "setup%s.py"%i

The patches hopefully do the above and much more. They refactor all the relevant code, document it way, way better, clean it up, and unify it (there was tons of replication between the notebook and command line).

The notebook page (sagenb-7514.patch) is only needed if you use the ntoebook, by the way.

[William Stein]

apply to the core sage library

[Sébastien Labbé]

Reading the summary of this ticket made me remember a problem I was having with the attach command : Can we attach a file already in the sage tree that we are editing? I already tried, but I stopped using it because sometimes the changes in the file were not considered and I have been stopping sage and running sage -br ever since then.

Maybe I was doing something bad or maybe this ticket solves the problem or maybe it is not possible at all to do this...??

Sébastien

[William Stein]

Can we attach a file already in the sage tree that we are editing?

I stopped using it because sometimes the changes in the file were not

considered and I have been stopping sage and running sage -br ever since then.

You probably don't understand what attach does. All it does is execfile the file that you attached. There are situations where this happening might be perceived as "the file were not considered". E.g., if you create an install F of a class Foo defined in a file a.py, then a.py is reloaded, the object F is *not* somehow magically modified in memory to be an instance of a the new Foo that was defined in the file a.py. That's not how Python works, and it would be weird and confusing overall if things did work that way.

[William Stein]

Replying to slabbe:

Reading the summary of this ticket made me remember a problem I was having with the attach command : Can we attach a file already in the sage tree that we are editing? I already tried, but I stopped using it because sometimes the changes in the file were not considered and I have been stopping sage and running sage -br ever since then.

Maybe I was doing something bad or maybe this ticket solves the problem or maybe it is not possible at all to do this...??

You just have to *understand* what attach does. It reloads the file via execfile into the global namespace when the file changes. You can attach any file, in the tree or not. But imagine this: In the sage tree there is a file foo.py. There is another file bar.py that does "import foo" and uses some code in foo. If you attach foo.py (which results in exec'ing it in the global interpreter namespaces), then that will have no impact at all on bar.py.

Thus for some problems attach works very nicely for library code, and for others it doesn't. Which is which is clearer if you know what attach does.

[Mitesh Patel]

Combined sage repo patch. Depends on 4.3.alpha1, SageNB 0.4.6 (#7625), #7483, #7482.

[Mitesh Patel]

I've attached a combined "sagelib" patch. It depends on 4.3.alpha1, SageNB 0.4.6 (#7625), #7483, and #7482.

Out of curiosity: What if we "overload" import by keeping a stack of all imported modules. When a watched file changes, we reload it and all modules loaded since the watched file was first loaded? Would this be useful in some situations?

[Mitesh Patel]

With 4.3.alpha1 + #7625 + #7483 + #7482 + this ticket, several doctests failed:

        sage -t  devel/sage/sage/misc/sageinspect.py # 3 doctests failed
        sage -t  devel/sage/sage/misc/preparser.py # 10 doctests failed
        sage -t  devel/sage/sage/misc/reset.pyx # 2 doctests failed
        sage -t  devel/sage/sage/misc/session.pyx # 1 doctests failed

Should we stop doctesting most (all?) of sage/server?

[Mitesh Patel]

Replying to mpatel:

With 4.3.alpha1 + #7625 + #7483 + #7482 + this ticket, several doctests failed:

Three tests fail in sagenb.misc.sageinpect. Please see #7650.

Should we stop doctesting most (all?) of sage/server?

Please see #7534.

[Mitesh Patel]

I noticed

sage: f = 1
sage: save(f, 'f')
sage: attach('f.sobj')
Traceback (most recent call last)
...
ValueError: argument (='f.sobj') to load or attach must have extension py, sage, or pyx                             
sage: attached_files()
['f.sobj']

Questions about loading/attaching Cython files:

• Can they only be loaded once, if they haven't changed? For example: If zzz.pyx contains print 'Zzz!', I see

  sage: load('zzz.pyx')
  Compiling zzz.pyx...
  Zzz!
  sage: load('zzz.pyx')
  sage: load('zzz.pyx')
  sage: # Now I edit zzz.pyx
  sage: load('zzz.pyx')
  Compiling zzz.pyx...
  Awake!
  

• Can they access existing objects? For example: If I put

  try:
      b += 1
  except:
      b = 10
  print b
  

in a .pyx file, loading the file always sets b to 10, even if it's already defined.

Disclaimer: I'm still quite ignorant about Cython.

[Mitesh Patel]

Fixed doctest failures. Replaces previous.

[Mitesh Patel]

Fixed doctest failures. Depends on Sage 4.3, #7483, #7482. Replaces previous.

[Mitesh Patel]

The latest patches should fix the doctests and the attached_files problem mentioned above. I changed attach t to attach(t), since the doctesting framework appears to use a Python interpreter, i.e., not IPython, to run the examples. I also weakened [somewhat] the tests for preparser.modified_attached_files. Is there an IPython doctesting mode?

I'm not sure about the Cython file reload problem. Should we set use_cache=True when loading .pyx files explicitly?

On globals: I see that a Cython file foo.pyx is compiled to a module and loaded via from foo import *.

Can someone else please help to review this ticket?

[Mitesh Patel]

Fix(?) loading of .spyx files. Add preparser.py to reference manual. Replaces previous.

[William Stein]

rebased against sagenb-0.4.8 + trac #7483.

[William Stein]

trac 7514 sagenb-part2: proper tracebacks; make source code introspection of functions defined in \ the notebook finally work in the notebook; properly cleanup worksheet files when notebook server is killed.

[William Stein]

Right now the files that one must apply are:

To Sage Library:
 sagelib-7514_combined.3.patch 

To Notebook:
 sagenb-7514-rebase.patch  
 sagenb-7514-rebase-part2.patch

[William Stein]

The file sagenb-7514-rebase-part2.patch fixes some serious issues. In particular, it makes it so we get proper tracebacks on errors, which was broken without this patch by the new architecture (of doing everything in the worksheet process, etc.). Also, source code introspection now works again for functions defined in the notebook, which many users wanted. Finally, there was a little bug where pressing control-c to stop the notebook server didn't result in quitting all worksheet processes, hence they left junk around.

[Mitesh Patel]

• When I load/attach a .(s)pyx file in the notebook, ___code__.py appears in the cell's output HTML.

• Minor: Should we make attached_files() in the notebook report, e.g.,

  [DATA+'foo.py']
  

  instead of

  ['/full/path/to/foo.py']
  

• Typing f??<TAB> for a function f defined in the notebook shows its docstring but not its source code. Is this related to ?? not working for Cython functions defined in cells or attached DATA files?

• Should we make it possible to edit attached .pyx files, just as we can edit .py, .sage, .spyx, and .txt files?

• Minor: save_notebook appears to be called twice on shutdown. Is this a fail-safe behavior?

[William Stein]

When I load/attach a .(s)pyx file in the notebook, _code.py appears in the cell's output HTML.

That's a bug. I'll fix it soon.

Minor: Should we make attached_files() in the notebook report

I'm not sure. But I think this should be a separate ticket, since it would be a new feature.

• Typing f??<TAB> for a function f defined in the notebook shows its docstring but > not its source code.

That's really weird, because it is one of the problems that this patch definitely fixes. Are you sure? Did you define a function f in one input cell, then do f?? in another, and it didn't show the source code? Were you using a clean sagenb-0.4.8 install for testing? Maybe I messed up posting the patch.

Minor: save_notebook appears to be called twice on shutdown. Is this a fail-safe behavior?

This has been the case forever. It is not caused by this patch. I don't think it is desirable. Please do open a ticket.

OK, so I'm going to fix the _code_.py issue you reported above, triple check that f?? works, and if not, figure out what went wrong, then mark this as "needs review" again.

[Mitesh Patel]

Replying to was:

• Typing f??<TAB> for a function f defined in the notebook shows its docstring but > not its source code.

That's really weird, because it is one of the problems that this patch definitely fixes. Are you sure? Did you define a function f in one input cell, then do f?? in another, and it didn't show the source code? Were you using a clean sagenb-0.4.8 install for testing? Maybe I messed up posting the patch.

It seems that just the last line of code is omitted. The source displayed is the preparsed source, but I don't know if this is a problem.

[William Stein]

The attached patch (sagenb-7514-rebase-part3.patch) should:

1. Fixed this: "When I load/attach a .(s)pyx file in the notebook, _code.py appears in the cell's output HTML."

2. Fixed this: "Typing f??<TAB> for a function f defined in the notebook shows its docstring but not its source code." Actually, this wasn't exactly broken... the last line of the docstring was missing. I've added a newline to the end of the input file, and now the problem is gone. This isn't a hack, since I think files are *supposed* to end with a newline, according to "IEEE Standard 1003.1 (aka POSIX)". Note that this patch *only* adds support for f?? for functions specifically entered in input normal cells (not cython, not load/attached). That's another issue.

[Mitesh Patel]

Trivial docstring tweaks. Replaces previous.

[Mitesh Patel]

Positive review.

If it's OK, I'll fix (e.g., INPUT/OUTPUT blocks) a few docstrings of functions affected by the sagelib patch.

[Mitesh Patel]

Replying to mpatel:

Positive review.

If it's OK, I'll fix (e.g., INPUT/OUTPUT blocks) a few docstrings of functions affected by the sagelib patch.

I'll make a separate ticket instead. We should include load and attach in the reference manual.

[Tim Dumol]

The preparsing function introduced in #7483 lacks a "# -*- coding: utf-8 -*-" header that prevents the notebook from evaluating any string that contains unicode. This is addressed in #7835.

[Mike Hansen]

I've merged sagelib-7514_combined.patch in sage-4.3.1.alpha0.

[William Stein]

I merged the sagenb patches into sagenb-0.4.8.

[Minh Van Nguyen]

The attachment ​sagelib-7514_combined.3.patch in the Sage library.