Upgrade to IPython 0.13

[kini]

We need to upgrade IPython to the shiny new thing. Well, not really - the shiny new thing was 0.11, and 0.13 is now stable. It's been mostly rewritten and is now a lot more modular, as I understand it. Definitely worth using in Sage.

Note: To use the QT console (sage -ipython qtconsole), you need qt/qt-devel in your base OS install, the optional zeromq/pyzmq spkgs from #12843, and the optional sip/PyQt spkgs from #13022.

Install the IPython 0.13.1 spkg at ​http://sage.math.washington.edu/home/jason/ipython-0.13.1spkg

apply:

To the scripts repository:

• trac_12719-scripts.patch​
• trac_12719_remove_sage_gdb_ipython.patch​
• 12719-test-displayhook.patch​

To the root repository:

• trac_12719_ROOT_configuration_files.patch​
• trac_12719-ipythondir013.patch​

To the sage library:

• trac_12719-5.1beta1.patch​
• trac_12719-move_to_preparser.patch​
• trac_12719_ipython_fixes.patch​
• trac_12719_dedent.patch​
• trac_12719-crash.patch​
• trac_12719-cellmagics013.patch​
• 12719-newipython-take2.patch​
• 12719-displayhook.patch​
• 12719-fix-transformer.patch​
• 12719-5.4rc1.patch​
• 12719-remove-plugin.patch​
• 12719-preparse-magic.patch​
• 12719-displayhook-library.patch​
• trac_12719-pager.patch​

cd SAGE_ROOT

./sage --hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719_ROOT_configuration_files.patch
./sage --hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-ipythondir013.patch

./sage --hg -R local/bin qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-scripts.patch
./sage --hg -R local/bin qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719_remove_sage_gdb_ipython.patch
./sage --hg -R local/bin qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-test-displayhook.patch

./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-5.1beta1.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-move_to_preparser.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719_ipython_fixes.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719_dedent.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-crash.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-cellmagics013.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-newipython-take2.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-displayhook.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-fix-transformer.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-5.4rc1.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-remove-plugin.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-preparse-magic.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/12719-displayhook-library.patch
./sage --hg -R devel/sage qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12719/trac_12719-pager.patch
./sage -i http://sage.math.washington.edu/home/jason/ipython-0.13.1.spkg

./sage -br

Fixes #11223

[jhpalmieri]

See #12167 for a related ticket: how we deal with IPython configuration files.

[fbissey]

Would be nice but we need some porting, here is what typically happen if you try to use ipython-0.12 with sage (we had several reports in sage-on-gentoo before I pinned down the version used by sage):

$ sage
----------------------------------------------------------------------
| Sage Version 4.7.2, Release Date: 2011-10-29                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/sage-ipython", line 19, in <module>
    from IPython.CrashHandler import CrashHandler
ImportError: No module named CrashHandler

[kini]

Yes, obviously :) I'm just making this ticket since it is absolutely must be an eventual goal for us and nobody else has made a ticket yet, even though IPython 0.12 has been out for about three months.

[fbissey]

To be honest Fernando Perez and I talked about it on the mailing list at the time of ipython 0.11 and I said I would tell him what happens when we use ipython 0.11+ with sage because he is interested. For various reason I haven't been able to go back to it. But some sage-on-gentoo users eventually got a taste of it when it finally landed in the portage tree.

So first hurdle: do something about the crash handling.

[mhansen]

I'm working on this now, and have made quite a bit of progress. As an added bonus, making the Sage customizations to IPython is quite a bit cleaner. I should hopefully be able to clean out the mess in sage/misc/preparser_ipython.py and sage/misc/interpreter.py .

[mhansen]

This will fix #4413.

[kini]

Great, I'm glad you're on the case! :)

[vbraun]

What about zmq/pyzmq? It would really add to the functionality and I'm interested in the parallel/cluster support. But, strictly speaking, its not necessary to run ipython. Its also used in the new notebook so maybe its time for it to become a standard package?

[kini]

CCing Jason as he expressed interest on sage-devel.

[jason]

mhansen: we'd love to see your work-in-progress if it's available somewhere. I just spent some time reading the IPython docs again and thinking about how we could use it to implement the notebook hub and worker processes.

[mhansen]

I've posted patches which let Sage work with IPython 0.12 as well as cleaning up some old messy code. These should be in a state where they're ready to review.

I haven't done anything with preparing an spkg yet.

[jason]

I've updated the instructions in the description. Please correct if I messed something up.

[jason]

Just curious: do your patches work with the current 0.12, or did you base them on the current git master (or does it not matter?). On the IPython mailing list, they mentioned that 0.12 had some big bugs, so they were thinking about cutting a quick 0.12.1. Also, they expect 0.13 in about a month.

[jason]

I was thinking more about security today with the zmq messages. I think it may be more secure to have a default ipcontroller transport of IPC in a directory inside of $DOT_SAGE, rather than tcp over the loopback interface. That would be more secure in that random people on your computer wouldn't be able to try to register with the IPython hub. I emailed the IPython mailing list about it: ​http://thread.gmane.org/gmane.comp.python.ipython.devel/7789

What do you think?

[mhansen]

Replying to jason:

Just curious: do your patches work with the current 0.12, or did you base them on the current git master (or does it not matter?). On the IPython mailing list, they mentioned that 0.12 had some big bugs, so they were thinking about cutting a quick 0.12.1. Also, they expect 0.13 in about a month.

I've looked at the current git master as I was making these changes, and I think these changes should work fine with what's on there. I'd be surprised if it made a difference.

[mhansen]

I just tested things against git master and everything seems fine.

[jason]

I'm trying to apply these to 5.0beta12. I think I probably got the order wrong in the description, so it would be great if you could check that. But regardless, none of the patches seem to apply to 5.0beta12 cleanly (each has rejects).

[jason]

I just tried applying to beta6 and the -scripts patch still had problems.

[mhansen]

The -scripts patch should apply fine to the scripts repository. I've rebased the other ones to work on the newer 5.0 beta's.

[AlexanderDreyer]

After applying the patches to sage-5.0 beta12 I still have ip = IPython.ipapi.get() at the end of sage/misc/edit_module.py, which crashes Sage on startup.

[jason]

I get a crash when I apply these (with a latest git IPython). I've posted the crash report at ​http://sage.math.washington.edu/home/jason/Sage_crash_report.txt

[jason]

Is the zmq communication in ipython going through tcp/localhost sockets, or through UDS sockets? I really think we should use UDS sockets with the files in $DOTSAGE so that it is more secure (i.e., random other people on the computer can't connect to the kernel).

[vbraun]

By default we should not use zeromq. For example, if you want to fork you should do so before creating a zmq context. The zmq context creates threads and open file descriptors.

Apart from that, you should either use IPC or TCP+magic cookie, much like the X window system. Implementation wise its probably easiest to allow any socket and alway check the magic cookie.

[jason]

I don't think we have a choice of whether or not to use zmq with the new ipython. ZMQ is an integral part of how it communicates results from the backend to the frontend. They do have a system for signing messages and ignoring unsigned messages (is that the magic cookie thing?). UDS sockets is IPC, so we're good there. My problem with TCP+magic cookie is that it still allows someone else to attempt to connect to the various sockets, whereas using IPC, with the socket files in $DOTSAGE, filesystem permissions prevent others from even attempting to connect.

[jason]

It may be that I'm misunderstanding how ipython is using zmq when you just execute 'ipython'. It may be that it uses in-process communication (using zmq) instead of opening ports on localhost. I've posted to the ipython-devel mailing list about this.

[vbraun]

Given that you can build IPython without zeromq support it surely must be possible to run without?

I don't see the problem with tcp + magic cookie. Yes you can connect but your connection attempt will be ignored. And the difference in pyzmq to ipc + magic cookie is a single string. So why bother with ipc without magic cookie?

[jason]

Thomas on the ipython mailing list confirmed that the plain ipython command doesn't use zmq, so that was a big misunderstanding I had about the new ipython. So never mind what I said about sockets.

[mhansen]

Okay, I fixed the problem with edit_module.

Sorry I wasn't clear about the default IPython not using zmq. There is a experimental console shell that uses two processes which can be accessed with "ipython console", but we aren't using that at all.

[jason]

For the record, the patches now apply to my 5.0beta12 and Sage starts up.

[jason]

When doing %edit, I edit the sage library file, but the file isn't copied over to the library after editing, so the changes don't take effect. This doesn't hold back this issue, though, since it didn't work before either.

[vbraun]

Although zeromq is not, strictly speaking, necessary for IPython it might be of interest that I made zeromq and pyzmq spkgs at #12843

[PierreCagne]

Not working on sage-5.0.rc0 (system Darwin 10.8.0). I installed the spkg and applied the patches in the given order and places, but still have the following error lauching Sage :

Traceback (most recent call last):
  File "/Users/pece/sage-5.0.rc0/local/bin/sage-ipython", line 19, in <module>
    from IPython.CrashHandler import CrashHandler
ImportError: No module named CrashHandler

I even installed the patches from #12843 without any success.

[jason]

I have OSX 10.6.8: Darwin Kernel Version 10.8.0. I'm running the IPython git master and the patches, and it seems to work well. Did you do sage -b to compile the patches? Maybe the problem is in the IPython 0.12.1->0.13 changes.

[jason]

Hmmm, it looks like the scripts/ patch wasn't applied (the problematic line is deleted in the patch). Can you check again that trac_12719-scripts.patch is applied to the SAGE_ROOT/local/bin repository?

[slabbe]

Replying to jason:

Maybe the problem is in the IPython 0.12.1->0.13 changes.

On sage-5.0.rc0, I managed to apply the three patches and sage starts correctly. So I guess it's not a matter of IPython 0.13 changes. Although, the sage prompt is not in color anymore (my file .sage/ipython/ipythonrc is set to dark background color).

Is the ipython notebook suppose to work? How should I test it?

Also, doing sage -ipython notebook creates this error :

$ sage -ipython notebook
Traceback (most recent call last):
...
ImportError: The IPython Notebook requires tornado >= 2.1.0

[vbraun]

The first patch (trac_12719.patch) applies only with fuzz 2 on sage-5.0

[vbraun]

Lots of doctests fail because the display hook is not correctly set up. I'll have a look into this if nobody beats me to it...

[vbraun]

The new ipython configuration files are once more incompatible with the previous ones. My patch versions the directory now (that is, it is named .sage/ipython-VERSION) and deletes the SAGE_ROOT/ipythonrc stuff. I think all sage-specific configuration should go into sage/misc/interface.py

[vbraun]

Here is some reasonable documentation about Prefilter vs. InputSplitter: ​http://mail.scipy.org/pipermail/ipython-dev/2011-March/007334.html

[vbraun]

Updated patch

[vbraun]

The trac_12719_ipython_fixes.patch switches form Prefilter to InputSplitter.

I'm giving a positive review to everything that Mike wrote.

[vbraun]

With the new patch sage/misc/interpreter.py has 100% coverage.

[was]

never mind.

[was]

In Sage before this patch, the sage notebook, the Python command line, etc., we have:

for i in range(10):
    i

outputs the numbers up to 10.

With the new Ipython spkg, etc., we don't.

sage: for i in range(10): i
sage: 
Exiting Sage (CPU time 0m0.07s, Wall time 2m3.22s).
blastoff:sage wstein$ sage -ipython
Python 2.7.2 (default, May 17 2012, 13:12:30) 
In [1]: for i in range(10): i
In [2]: 

I'm disappointed, especially because I thought this through and in sagews I recently wrote code so this works correctly in a single cell, even if there are several instances of this. It's a matter of passing the 'single' option to exec, as is documented in the Python reference. I wonder why they screwed this up in Ipython 0.12...?

Anyway, I'm not saying this is a show stopper at all - I want the new ipython in sage. I'm just a little concerned that literally the first thing I tried shows that they made what I consider a bad design choice. Hopefully it was just bad luck.

[was]

Cool -- I tried out a bunch of other things including the new notebook (sage -ipython notebook) and didn't run into any serious issues. So I'm happy with this from from a purely functional perspective. Also, deleting tons of ugly code (done in the patches), looks great.

[mhansen]

It should be easy to get to get the correct functionality for for i in range(10): i. The shell.run_ast_nodes method just has to be called with interactivity="all" instead of interactivity="last_expr".

[mhansen]

Here's a review of Volker's ipython_fixes patch:

1. (minor) Remove \ on line 1071
2. Do one of the following:
   1. Fix/deprecate/remove sage-gdb-ipython since it depends on SAGE_CLEAN
   2. Just throw a deprecation warning with SAGE_CLEAN
3. Turn autoindent back on.
4. Why add debug=True and confirm_exit=False to default config.
5. Load the startup file: line 1151

[vbraun]

Initial patch

[vbraun]

1, 3, 5: done

4: I removed the debug, but I (and most others, I asked) definitely want to keep confirm_exit=False. Yes I really want to quit, geez!

2: I'm in favor of a) and removed it.

I've also renamed trac_13013_ROOT_configuration_files.patch (wrong ticket number) and added a patch to auto-dedent and eval in loops. Now the following works:

sage:         for i in range(3): i
0
1
2

[mhansen]

Volker's changes look good to me.

[was]

Mike's addition of trac_12719-crash.patch looks *great* to me as an addition.

[jdemeyer]

Unfortunately, this doesn't apply to sage-5.1.beta1:

applying trac_12719.patch
patching file doc/en/reference/misc.rst
Hunk #1 succeeded at 30 with fuzz 2 (offset 2 lines).
patching file sage/misc/interpreter.py
Hunk #1 FAILED at 0
Hunk #3 FAILED at 271
Hunk #4 FAILED at 385
Hunk #5 FAILED at 449
4 out of 5 hunks FAILED -- saving rejects to file sage/misc/interpreter.py.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working dir
errors during apply, please fix and refresh trac_12719.patch

[mhansen]

I've rebased this on top of #11817.

[jdemeyer]

Now it applies but I get documentation errors:

dochtml.log:/padic/scratch/jdemeyer/merger/sage-5.1.beta3/local/lib/python2.7/site-packages/sage/misc/interpreter.py:docstring of sage.misc.interpreter:12: WARNING: Block quote ends without a blank line; unexpected unindent.
dochtml.log:/padic/scratch/jdemeyer/merger/sage-5.1.beta3/local/lib/python2.7/site-packages/sage/misc/interpreter.py:docstring of sage.misc.interpreter:8: WARNING: Block quote ends without a blank line; unexpected unindent.
dochtml.log:/padic/scratch/jdemeyer/merger/sage-5.1.beta3/local/lib/python2.7/site-packages/sage/misc/interpreter.py:docstring of sage.misc.interpreter:8: WARNING: Block quote ends without a blank line; unexpected unindent.
dochtml.log:/padic/scratch/jdemeyer/merger/sage-5.1.beta3/local/lib/python2.7/site-packages/sage/misc/interpreter.py:docstring of sage.misc.interpreter:11: WARNING: Definition list ends without a blank line; unexpected unindent.
dochtml.log:/padic/scratch/jdemeyer/merger/sage-5.1.beta3/local/lib/python2.7/site-packages/sage/misc/interpreter.py:docstring of sage.misc.interpreter:13: ERROR: Unexpected indentation.

[jdemeyer]

Also, trac_12719-scripts.patch​ needs a commit message.

[jason]

It seems that double question mark, like trace??, doesn't bring up the source code anymore, but just brings up the docstring. It seems that trac_12719_ipython_fixes.patch broke this (double question marks work with just the first two patches, but not after that third patch).

[jason]

Add two more patches to get the new IPython 0.13beta1 spkg above to work.

[jason]

I rebased trac_12719_ROOT_configuration_files.patch​ to sage 5.1beta6

[jason]

I upgraded the spkg to 0.13, which was just released today.

[jason]

Replying to was:

In Sage before this patch, the sage notebook, the Python command line, etc., we have:

for i in range(10):
    i

outputs the numbers up to 10.

With the new Ipython spkg, etc., we don't.

sage: for i in range(10): i
sage: 
Exiting Sage (CPU time 0m0.07s, Wall time 2m3.22s).
blastoff:sage wstein$ sage -ipython
Python 2.7.2 (default, May 17 2012, 13:12:30) 
In [1]: for i in range(10): i
In [2]: 

I'm disappointed, especially because I thought this through and in sagews I recently wrote code so this works correctly in a single cell, even if there are several instances of this. It's a matter of passing the 'single' option to exec, as is documented in the Python reference. I wonder why they screwed this up in Ipython 0.12...?

Anyway, I'm not saying this is a show stopper at all - I want the new ipython in sage. I'm just a little concerned that literally the first thing I tried shows that they made what I consider a bad design choice. Hopefully it was just bad luck.

Fernando Perez responded:

BTW, regarding William's comment, that was not a mistake or omission: it was a very deliberate choice we made after lots of thought. Given that we manage numbered prompts and history, and that matplotlib produces so many plots as side effects, after much experimentation we found it worked much better to only execute with 'interactive' mode the last block.

Sage doesn't distinguish stdout from python sys.displayhook, so the opposite behavior may be more desirable, but we actually consider that a 'bad design choice' of sage, because it loses an important distinction and the cache management of output history.

So in summary: our choice is actually sensible and consistent given IPython's internal design :)

[jason]

Fernando: I looked in the IPython source, and it seems that IPython is a little more subtle than mentioned above. By default, it only executes the last block as interactive (compile in 'single' mode) if it is an expression. In Sage, we (by default) execute the last block always in 'single' mode. I think that is the difference here.

Given that (and if I understood correctly), could you explain again your reasoning for choosing to not compile in 'single' mode if it isn't an expression?

Note to self for future reference: for i in range(10): i invokes sys.displayhook 10 times to display i if compiled in 'single' mode.

[jason]

Also, note to William: it is an easy option to set to have IPython *always* execute the last block in interactive mode. It's just not the default.

[jason]

Replying to jason:

Fernando: I looked in the IPython source, and it seems that IPython is a little more subtle than mentioned above. By default, it only executes the last block as interactive (compile in 'single' mode) if it is an expression. In Sage, we (by default) execute the last block always in 'single' mode. I think that is the difference here.

Given that (and if I understood correctly), could you explain again your reasoning for choosing to not compile in 'single' mode if it isn't an expression?

And here is Fernando's response:

Precisely so that if the last block has a loop, it doesn't produce multiple Out [NN] cells. We want to keep a one-to-one mapping between a cell and its output. For example it's quite common to write code like:

for i in x:
  plot(x[i], label='blah_i')

If we compile that as 'single', we'd get all the Matplotlib line collection results piling up at the bottom, one for each plot call.

We played a *lot* with various options, and settled on this behavior after much experimentation. After 2 years of using it, I remain convinced it's the right solution *for us*. Not to say that Sage can't make a different choice that works better for you guys, of course!

[jason]

To change this, we just need to set ast_node_interactivity to "last" in the InteractiveShell.

[schilly]

Replying to fbissey:

Would be nice but we need some porting, here is what typically happen if you try to use ipython-0.12 with sage:

$ sage
----------------------------------------------------------------------
| Sage Version 4.7.2, Release Date: 2011-10-29                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/sage-ipython", line 19, in <module>
    from IPython.CrashHandler import CrashHandler
ImportError: No module named CrashHandler

Hi fbissey, sorry for hijacking this a bit, but i had exactly the same problem over and over again. What I just found out (and I think it really helps) is to temporarily disable the paths containing ".local" in sys.path in the sage-ipython file. The reason is, that ipython seems to import the "site package" (disable it for pure python via "-S"), because that's where my ipython 0.13 is.

So, by adding this to sage-ipython you I have both at the same time:

~~~ after import sys
oldpath = sys.path
sys.path = filter(lambda s : '.local' not in s, sys.path)
import IPython
sys.path = oldpath

[jason]

Updated patch

[jason]

I rebased the only rejected patch for 5.2beta1

[was]

My 2 cents, after I forced myself to use this for the last few weeks on my laptop. I have found the new ipython to be pretty annoying. The cut and paste behavior and lack of autoindent at the command line is pretty painful. The thing that made me finally give up and revert to the old 0.10 ipython was when I hit control-c at the wrong moment when starting Sage, and somehow the ipython history sqlite database got in a weird locked state. Nothing I could think to do (including deleting $HOME/.sage/ipython) got this unstuck. Maybe rebooting would have worked; I don't know. I'm concerned about how stable and well-tested in the wild the new ipython is. Proceed with caution.

[vbraun]

Working cut&past and working autoindent are contradictory goals. The terminal just sees a stream of characters, you either inject indentation or not. AFAIR I turned off auto-indent, we could turn that back on. Breaks pasting of indented text, of course.

The ipython dir is now versioned, you should have deleted $DOT_SAGE/ipython-0.12 / $DOT_SAGE/ipython-0.13

[vbraun]

PS: Maybe you can upload your broken ipython dir somewhere, this would be a good testcase to catch the sqlite error...

[kini]

IMO, leave auto-indent on, and use %paste or %cpaste when you want to paste. %paste tries to automatically get the contents of your clipboard. If that's not working for whatever reason, %cpaste creates a prompt where you can paste text verbatim, which IPython doesn't read until you end input with Ctrl+D or a line containing only the text "--".

[vbraun]

Replying to kini:

IMO, leave auto-indent on, and use %paste or %cpaste when you want to paste.

For the record, thats not how I use the commandline. I paste code all the time, but I almost never enter multi-line statements manually. So I'd rather have autoindent off by default (and no need for %cpaste), and use %autoindent if I want it on. Though thats of course a matter of personal preferences, and I'm happy with whatever the majority prefers...

[was]

Replying to vbraun:

Replying to kini:

IMO, leave auto-indent on, and use %paste or %cpaste when you want to paste.

For the record, thats not how I use the commandline. I paste code all the time, but I almost never enter multi-line statements manually. So I'd rather have autoindent off by default (and no need for %cpaste), and use %autoindent if I want it on. Though thats of course a matter of personal preferences, and I'm happy with whatever the majority prefers...

At least it is configurable, right? I think it is best not to change the default from what it is already in all previous versions of Sage, whether or not that is what the majority prefers (unless it is a huge majority). I'm not fan of changing UI's on people unless it is absolutely necessary.

[vbraun]

Well IPythons defaults changed, we didn't just change stuff for the fun of it. For starters, vanilla IPython now disallows any superfluous indentation, i.e.

In [1]: 1
Out[1]: 1

In [2]:  1
IndentationError: unexpected indent (<ipython-input-2-0066badbf200>, line 1)

[jhpalmieri]

Two more issues:

• when Sage starts, any lines in my init.sage file are echoed to the screen after the first "sage:" prompt. They aren't executed until I hit the return key:

  ----------------------------------------------------------------------
  | Sage Version 5.2, Release Date: 2012-07-25                         |
  | Type "notebook()" for the browser-based notebook interface.        |
  | Type "help()" for help.                                            |
  ----------------------------------------------------------------------
  sage: # Here is the beginning of init.sage.
  
  2+2
  
  # Here is another comment
  
  a = 5
  
  # Here is the end of init.sage.
  

  At this point, hitting <RETURN> inserts a 4, a newline, and the "sage:" prompt, and a is defined to be 5.

• when I evaluate or plot a graphics object, two plot windows are opened:

  sage: P = plot(sin(x), (x, 0, 10))
  sage: P  # opens up two windows showing the same plot. Note also the blank line.
  
  sage: plot(P) # also opens up two windows.
  
  sage: show(P)  # opens up only one window.
  

[iandrus]

Regarding indentation, autoindent being on breaks sage inside Emacs (dumb terminal). So I think it should be off, at least in a dumb terminal, until/unless I can figure out how to fix this in Emacs. I haven't really looked into it at all, so it may be easy to fix inside Emacs by setting some environment variables or something. See #10504.

[jason]

+1 for autoindent being on by default. That helps what is probably the vast majority of the cases---interactively typing into the command line.

[broken_symlink]

I'm having a weird problem when I use IPython.parallel in sage. Here's what happens:

from IPython.parallel import Client
rc = Client('/home/seshu/.sage/ipython-new/profile_default/security/ipcontroller-client.json')
dview = rc[:]
dview.map_sync(lambda x: x**10, range(32))
[0:apply]: 
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)<string> in <module>()
<ipython-input-4-ad199b8d76dc> in <lambda>(x)
NameError: global name 'Integer' is not defined

If I do anything that doesn't involve integers like dview.map_sync(lambda x: "hi", range(32)), it works fine. Sorry if this isn't the appropriate place to ask these questions. I wasn't sure where to ask since this is a bit bleeding edge. I would like to be able to use IPython.parallel from the sage notebook if possible.

[broken_symlink]

The ipython engines need to import the sage modules. The fix is dview.execute("from sage.all import *"). Also I did sage -sh ipcluster start.

[jason]

Just FYI, for the *next* version (so not an issue on this ticket), ​https://github.com/ipython/ipython/pull/2299 broke Sage, I think by getting rid of PrefilterTransformer? tricks.

[jason]

So I think that the only issue with this ticket is turning back on auto-indentation and getting ?? to show the source code? Any other issues?

I think the parallel issues above should be moved to another ticket.

[jason]

Note: ?? shows the source code fine, it seems, in the notebook (at least there doesn't seem to be a regression; ?? shows source when doing tab completion, but not on evaluating, just like before the python upgrade). It is a regression on the command line, though.

[jhpalmieri]

There are the issues I mentioned above. Can anyone else reproduce them?

[broken_symlink]

The parallel issues only seem to effect the notebook. I've successfully gotten ipython.parallel and sage to run on a small 24 core amazon ec2 cluster, using ssh with the ipython 0.13.1 branch on github. According to the ipython guys, the ssh launcher in 0.13 is broken. I kinda have some idea of whats going with the parallel issues and the notebook as well, but i'm not sure how to fix it. I can write something for how to setup sage with ipython engines started over ssh if this package gets updated to 0.13.1.

[jason]

Well, hopefully this gets merged before 0.13.1 comes out.

[jason]

jhpalmieri, I can confirm both of the issues that you saw. I'm seeing them too.

[jason]

actually, it looks like autoindent was turned back on: ​http://trac.sagemath.org/sage_trac/ticket/12719#comment:52

[jason]

Does anyone know where in the code init.sage is run? I've searched the entire codebase for "init.sage" and I'm not finding anything that looks like it is responsible for running the file.

[jason]

Ah, I found it. It's the last line in sage/misc/interpreter.py, the one that says self.shell.run_cell('%%load "%s"'%startup_file)

[jason]

Okay, it looks like the problem is that we punt to the ipython load, which *pastes* the content so you can edit it first: ​http://ipython.org/ipython-doc/dev/interactive/qtconsole.html#load

[jason]

So our conclusion on ​https://github.com/ipython/ipython/pull/2299 was that Sage should make %load an alias for IPython's %run, and beef up IPython's %run to be able to handle remote files. In fact, make IPython's %run use the same simple mechanism as IPython's %load would be great (though may not be possible---%run uses file-specific features in python and execfile, etc.)

[jason]

Is there a reason we didn't use a profile config file for all of our customizations?

I've started a different approach to the customizations we use here: ​https://github.com/jasongrout/sage-ipython-profile

put these into the .sage/ipython-new directory and then start with sage -ipython --profile=sage

[jason]

Okay, I think my "take2" patch significantly cleans things up and makes things more modular and more "IPythonic".

In particular, this seems to work nicely: do sage -ipython notebook to launch the ipython notebook, then evaluate %load_ext sage.misc.sage_extension. All preparsing is turned on, etc. It's great!

There should probably be a bit more documentation in the sage_extension file, and full doctests should be run. But I think it's ready for other people to hammer on it too.

Here is one issue: something broke in sage -startuptime. I'm not sure it's IPython's fault, though.

Oh, and I was working from IPython master, not IPython 0.13. I hope that doesn't put a wrench in the works...

[jason]

Preliminary tests seem to indicate that these patches and the new take work just fine with IPython 0.13 (at least when I revert my IPython git repository). So it's still probably ready for more people to hammer on this.

[jason]

My take2 patch seems to have solved the double plot window problem. It also solved the init.sage/load problems, as well as the ?? problem. So that was all of the issues listed...

[jason]

Since we now hook into the IPython display hook, we can delete our special display hook.

[jason]

Fixed the startup issue with new patch to scripts repository

[jason]

Fix a transforming issue: see ​http://mail.scipy.org/pipermail/ipython-dev/2012-September/010329.html or the commit message.

[jdemeyer]

This needs to be rebased to sage-5.4.rc0, in particular #13459.

[jason]

Also, #13361 includes the change in the startuptime.py patch, so we can delete it.

[jason]

rebased to 5.4rc1

[jason]

rebased to 5.4rc1

[jason]

Add one more patch so we don't change directories anymore.

[jason]

Something that seems to work fine now is starting up sage in the devel/sage directory. So it seems that the issue discussed in #13361 is fixed in this ipython upgrade.

[jason]

Okay, I think this may be ready for review again.

It's a big enough change that maybe it would be good to shoot for one of the first patches in the 5.5 release, so it gets plenty of testing???

[jhpalmieri]

Replying to slabbe:

Also, doing sage -ipython notebook creates this error :

$ sage -ipython notebook
Traceback (most recent call last):
...
ImportError: The IPython Notebook requires tornado >= 2.1.0

I get this too. Should I just install tornado somehow, or should this be fixed as part of this ticket?

Edit: after installing tornado (with sage --sh and easy_install tornado), it said I had to install pyzmq. So I did that, and now the notebook runs. It looks very nice, and feels pretty snappy, maybe snappier than the Sage notebook.

[jason]

Yes, tornado and zmq (pyzmq) are required for the ipython notebook. That's not part of this ticket.

And yes, the ipython notebook is very nice. After loading it up, try doing

%load_ext sage.misc.sage_extension

to get some sage goodness, like the preparser, etc.!

[jdemeyer]

Replying to jason:

Something that seems to work fine now is starting up sage in the devel/sage directory.

FYI: this was fixed by #13459.

[jason]

Replying to jdemeyer:

Replying to jason:

Something that seems to work fine now is starting up sage in the devel/sage directory.

FYI: this was fixed by #13459.

But I reverted the "import sage" that #13459 does in trac_12719-scripts.patch​, and it still works fine for me (running sage in devel/sage and then import sage actually imports the site-packages sage). Maybe IPython changed its handling of sys.path. What I'm saying is that we should test that my reverting of part of #13459 (the part where we imported sage in sage-ipython) actually does what I think it does.

[jhpalmieri]

I'm getting doctest failures because of how lists of matrices are printed. For example, with matrix2.pyx:

File ".../sage-5.4.rc1/devel/sage/sage/matrix/matrix2.pyx", line 11627:
    sage: (d, u, v)
Expected:
    (
    [     1      0]  [ 1  0]  [ 1 -w]
    [     0 -w + 9], [-w  1], [ 0  1]
    )
Got:
    ([     1      0]
    [     0 -w + 9], [ 1  0]
    [-w  1], [ 1 -w]
    [ 0  1])

That file has 7 failures altogether. Is this because you turned off Sage's displayhook?

[jason]

Interesting. I thought I used Sage's displayhook (I know I was at one time anyway), but I've also noticed the same problems on my install with printing tuples of matrices and it's been frustrating to me. I'll take a look at it.

[jhpalmieri]

12719-displayhook.patch​ looks relevant, in particular removing the lines

import sage.misc.displayhook  
sage.misc.displayhook.install() 

and removing the corresponding functions from sage/misc/displayhook.py. But I've never worked with this aspect of Sage before.

[jhpalmieri]

When I use either time or %time, the rest of the line doesn't seem to be preparsed: try

sage: srange(1/10, 1/2, 1/10)
[1/10, 1/5, 3/10, 2/5]

vs.

sage: %time srange(1/10, 1/2, 1/10)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
...
ValueError: srange() step argument must not be zero

Or even

sage: %time 1/10
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
0  <-- note the answer!

[jason]

The displayhook is activated when Sage is run (try running those matrix doctests from the command line). The problem is that the doctesting framework does *not* use the interactive version of Sage; it seems to use straight python. So the display hook is not set.

It seems that the display hook used to be set when just importing Sage as a library, maybe? If so, I think that's the wrong approach. Importing a library shouldn't obliterate the display hook.

So it sounds like we need to make the doctesting framework load the right Sage things, like the preparser, the display hook, etc.

[jason]

I'll take a look at %time. That indeed seems to be a problem where the preparser isn't being run.

[jason]

For some reason, the preparser explicitly turned itself off for magic lines. I'm not sure why but I'm posting a patch to fix that.

[jason]

John, you know the doctest framework a lot better than I do. If you start up sage-ipython and load the sage extension, it will install the display hook, import Sage, etc. John, do you know how to get the doctest runner to execute this as part of ipython: get_ipython().extension_manager.load_extension('sage.misc.sage_extension')

grout@tiny:~/sage% sage -ipython -ic "get_ipython().extension_manager.load_extension('sage.misc.sage_extension')"                    E:128
Python 2.7.3 (default, Oct  9 2012, 11:53:24) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: (identity_matrix(3), identity_matrix(5))
Out[1]: 
(
         [1 0 0 0 0]
         [0 1 0 0 0]
[1 0 0]  [0 0 1 0 0]
[0 1 0]  [0 0 0 1 0]
[0 0 1], [0 0 0 0 1]
)

[jhpalmieri]

I guess one reason to turn off the preparser for magic lines is so that in a line like

%ed file1.py

the "1" doesn't get preparsed. I don't know how the old "time" command was written; did it just call "%time"? If so, maybe that was to allow lines starting with "time" to be preparsed.

For doctesting, I don't know if IPython is used at all during doctesting. If it's possible to turn on this extension using pure Python, we can add it at the top of each file as it's doctested, with a patch like this to local/bin/sage-doctest:

sage-doctest

@@ -490 +490 @@
 
 """  % os.path.join(os.getcwd(), file_name)
     s += "from sage.all_cmdline import *; \n"
+    s += "ADD NEW CODE HERE\n"
     s += "import sage.plot.plot; sage.plot.plot.DOCTEST_MODE=True\n"  # turn off image popup
     s += r"""
 def warning_function(f):

[jason]

apply to scripts repository in local/bin/

[jason]

Updated instructions for new display hook patch that sets the display hook when doctesting.

[jason]

For the %time command, I filed an issue with IPython a while ago: ​https://github.com/ipython/ipython/pull/2403

For now, it appears like we'll have to ship that patch ourselves.

[jhpalmieri]

The matrix-printing seems to work now, but I'm seeing a few other doctest failures:

sage -t  --long -force_lib devel/sage/sage/interfaces/sage0.py
**********************************************************************
File "/scratch/palmieri/12719/sage-5.4.rc1/devel/sage-main/sage/interfaces/sage0.py", line 489:
    sage: sage0(4).gcd
Expected:
    <built-in method gcd of sage.rings.integer.Integer object at 0x...>
Got:
    <function gcd>
**********************************************************************

and several failures in sage_extension.py, which would take up too much room to include here.

[jhpalmieri]

I think the errors in sage_extension.py are because file names in %runfile foo are being preparsed: for example, the first error message is pretty long, but it ends with

IOError: did not find file '/home/palmieri/.sage/temp/sage.math.washington.edu/Integer(21508)/dir_0/run_cell.py' to load or attach

Note that Integer(21508) in the file name; this should presumably just be 21508. This preparsing doesn't happen if the number is mixed in with letters, but for files or directories with purely numeric names, it seems to pop up. So from the command line:

sage: %runfile hello2.py   # works
sage: %runfile 123/hello2.py   # fails
sage: %runfile /home/palmieri/123/hello2.py   # fails

The same thing happens with %attach, and I assume also with the other magic functions which take file names as arguments.

[jhpalmieri]

I don't understand the sage0.py failure. Does the new IPython have a different default print representation for some objects? Before the patches here:

sage: a = 4
sage: a.gcd
<built-in method gcd of sage.rings.integer.Integer object at 0x480e330>
sage: a.gcd?
Type:		builtin_function_or_method
Base Class:	<type 'builtin_function_or_method'>
String Form:	<built-in method gcd of sage.rings.integer.Integer object at 0x480e330>
Namespace:	Interactive
Definition:	a.gcd(self, n)
Docstring:
     ...

With the patches:

sage: a = 4
sage: a.gcd
<function gcd>
sage: a.gcd?
Type:       builtin_function_or_method
String Form:<built-in method gcd of sage.rings.integer.Integer object at 0x5b7c5a0>
Definition: a.gcd(self, n)
Docstring:
   ...

With the old version, the print rep for a.gcd is the same as what's labeled the "String Form" when you do a.gcd?. With the new version, the print rep has changed but the "String Form" has not. I'm also confused by this behavior (with the patches):

sage: a = 4
sage: G = a.gcd
sage: G.__repr__()
'<built-in method gcd of sage.rings.integer.Integer object at 0x4b685a0>'
sage: G
<function gcd>
sage: str(G)
'<built-in method gcd of sage.rings.integer.Integer object at 0x4b685a0>'

Why isn't G printed using its __repr__ method?

I've tried out some other objects (e.g., matrices and elements of the Steenrod algebra) and my guess is that the print representations for methods defined in pyx files are getting changed to <function ...>, while those defined in py files haven't changed. Any ideas why? Looking at ipython-0.10.2.p1/src/IPython/UserConfig/ipythonrc, I see the configuration option object_info_string_level 0, which looks relevant, but we're not using this style of configuration, right?

[broken_symlink]

Just a heads up that iPython 0.13.1 is set to be released on Friday. It fixes all the bugs I was having with ipython.parallel.

[jhpalmieri]

We need another patch, since the pager in IPython has moved. One instance of this was fixed in one of the patches, but we need to fix others.

Note the change to sagedoc.py; this needs to be tested.

[jason]

It seems like you caught all of the instances of the IPython pager:

grout@tiny:~/sage/devel/sage/sage% ack genutils *
interfaces/gap3.py
493:            import IPython.genutils
494:            IPython.genutils.page(helptext)

misc/pager.py
27:        import IPython.genutils
28:        return IPython.genutils.page

misc/sagedoc.py
833:        from IPython.genutils import page