Opened 10 years ago

Last modified 5 years ago

#12719 closed enhancement

Upgrade to IPython 0.12 — at Version 33

Reported by: kini Owned by: tbd
Priority: critical Milestone: sage-5.7
Component: packages: standard Keywords: sd40.5
Cc: iandrus, vbraun, jason, AlexanderDreyer, robertwb Merged in:
Authors: Mike Hansen Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jason)

We need to upgrade IPython to the shiny new thing. Well, not really - the shiny new thing was 0.11, and 0.12 is now stable. It's been mostly rewritten and is now a lot more modular, as I understand it. Definitely worth using in Sage.

Install IPython 0.12: preliminary spkg at http://sage.math.washington.edu/home/jason/ipython-0.12.p0.spkg

apply:

To the scripts repository: trac_12719-scripts.patch

To the sage library:

  1. trac_12719.patch
  2. trac_12719-move_to_preparser.patch

Change History (35)

comment:1 Changed 10 years ago by jhpalmieri

See #12167 for a related ticket: how we deal with IPython configuration files.

comment:2 Changed 10 years ago by fbissey

Would be nice but we need some porting, here is what typically happen if you try to use ipython-0.12 with sage (we had several reports in sage-on-gentoo before I pinned down the version used by sage):

$ sage
----------------------------------------------------------------------
| Sage Version 4.7.2, Release Date: 2011-10-29                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/sage-ipython", line 19, in <module>
    from IPython.CrashHandler import CrashHandler
ImportError: No module named CrashHandler

comment:3 Changed 10 years ago by kini

Yes, obviously :) I'm just making this ticket since it is absolutely must be an eventual goal for us and nobody else has made a ticket yet, even though IPython 0.12 has been out for about three months.

comment:4 Changed 10 years ago by fbissey

To be honest Fernando Perez and I talked about it on the mailing list at the time of ipython 0.11 and I said I would tell him what happens when we use ipython 0.11+ with sage because he is interested. For various reason I haven't been able to go back to it. But some sage-on-gentoo users eventually got a taste of it when it finally landed in the portage tree.

So first hurdle: do something about the crash handling.

comment:5 Changed 10 years ago by iandrus

  • Cc iandrus added

comment:6 Changed 10 years ago by mhansen

I'm working on this now, and have made quite a bit of progress. As an added bonus, making the Sage customizations to IPython is quite a bit cleaner. I should hopefully be able to clean out the mess in sage/misc/preparser_ipython.py and sage/misc/interpreter.py .

comment:7 Changed 10 years ago by mhansen

This will fix #4413.

comment:8 Changed 10 years ago by kini

Great, I'm glad you're on the case! :)

comment:9 Changed 10 years ago by vbraun

  • Cc vbraun added

comment:10 Changed 10 years ago by vbraun

What about zmq/pyzmq? It would really add to the functionality and I'm interested in the parallel/cluster support. But, strictly speaking, its not necessary to run ipython. Its also used in the new notebook so maybe its time for it to become a standard package?

comment:11 Changed 10 years ago by kini

  • Cc jason added

CCing Jason as he expressed interest on sage-devel.

comment:12 Changed 10 years ago by jason

mhansen: we'd love to see your work-in-progress if it's available somewhere. I just spent some time reading the IPython docs again and thinking about how we could use it to implement the notebook hub and worker processes.

comment:13 Changed 10 years ago by mhansen

I've posted patches which let Sage work with IPython 0.12 as well as cleaning up some old messy code. These should be in a state where they're ready to review.

I haven't done anything with preparing an spkg yet.

comment:14 Changed 10 years ago by jason

  • Authors set to Mike Hansen
  • Description modified (diff)

I've updated the instructions in the description. Please correct if I messed something up.

comment:15 Changed 10 years ago by jason

  • Status changed from new to needs_review

comment:16 Changed 10 years ago by jason

  • Status changed from needs_review to needs_work
  • Work issues set to spkg

comment:17 follow-up: Changed 10 years ago by jason

Just curious: do your patches work with the current 0.12, or did you base them on the current git master (or does it not matter?). On the IPython mailing list, they mentioned that 0.12 had some big bugs, so they were thinking about cutting a quick 0.12.1. Also, they expect 0.13 in about a month.

comment:18 Changed 10 years ago by jason

I was thinking more about security today with the zmq messages. I think it may be more secure to have a default ipcontroller transport of IPC in a directory inside of $DOT_SAGE, rather than tcp over the loopback interface. That would be more secure in that random people on your computer wouldn't be able to try to register with the IPython hub. I emailed the IPython mailing list about it: http://thread.gmane.org/gmane.comp.python.ipython.devel/7789

What do you think?

comment:19 in reply to: ↑ 17 Changed 10 years ago by mhansen

Replying to jason:

Just curious: do your patches work with the current 0.12, or did you base them on the current git master (or does it not matter?). On the IPython mailing list, they mentioned that 0.12 had some big bugs, so they were thinking about cutting a quick 0.12.1. Also, they expect 0.13 in about a month.

I've looked at the current git master as I was making these changes, and I think these changes should work fine with what's on there. I'd be surprised if it made a difference.

comment:20 Changed 10 years ago by mhansen

I just tested things against git master and everything seems fine.

comment:21 Changed 10 years ago by jason

  • Description modified (diff)

I'm trying to apply these to 5.0beta12. I think I probably got the order wrong in the description, so it would be great if you could check that. But regardless, none of the patches seem to apply to 5.0beta12 cleanly (each has rejects).

comment:22 Changed 10 years ago by jason

  • Description modified (diff)
  • Work issues changed from spkg to patches don't apply

I just tried applying to beta6 and the -scripts patch still had problems.

Changed 10 years ago by mhansen

comment:23 Changed 10 years ago by mhansen

The -scripts patch should apply fine to the scripts repository. I've rebased the other ones to work on the newer 5.0 beta's.

comment:24 Changed 10 years ago by AlexanderDreyer

  • Cc AlexanderDreyer added

After applying the patches to sage-5.0 beta12 I still have ip = IPython.ipapi.get() at the end of sage/misc/edit_module.py, which crashes Sage on startup.

comment:25 Changed 10 years ago by jason

I get a crash when I apply these (with a latest git IPython). I've posted the crash report at http://sage.math.washington.edu/home/jason/Sage_crash_report.txt

comment:26 Changed 10 years ago by jason

Is the zmq communication in ipython going through tcp/localhost sockets, or through UDS sockets? I really think we should use UDS sockets with the files in $DOTSAGE so that it is more secure (i.e., random other people on the computer can't connect to the kernel).

comment:27 Changed 10 years ago by vbraun

By default we should not use zeromq. For example, if you want to fork you should do so before creating a zmq context. The zmq context creates threads and open file descriptors.

Apart from that, you should either use IPC or TCP+magic cookie, much like the X window system. Implementation wise its probably easiest to allow any socket and alway check the magic cookie.

comment:28 Changed 10 years ago by jason

I don't think we have a choice of whether or not to use zmq with the new ipython. ZMQ is an integral part of how it communicates results from the backend to the frontend. They do have a system for signing messages and ignoring unsigned messages (is that the magic cookie thing?). UDS sockets is IPC, so we're good there. My problem with TCP+magic cookie is that it still allows someone else to attempt to connect to the various sockets, whereas using IPC, with the socket files in $DOTSAGE, filesystem permissions prevent others from even attempting to connect.

comment:29 Changed 10 years ago by jason

It may be that I'm misunderstanding how ipython is using zmq when you just execute 'ipython'. It may be that it uses in-process communication (using zmq) instead of opening ports on localhost. I've posted to the ipython-devel mailing list about this.

comment:30 Changed 10 years ago by vbraun

Given that you can build IPython without zeromq support it surely must be possible to run without?

I don't see the problem with tcp + magic cookie. Yes you can connect but your connection attempt will be ignored. And the difference in pyzmq to ipc + magic cookie is a single string. So why bother with ipc without magic cookie?

comment:31 Changed 10 years ago by jason

Thomas on the ipython mailing list confirmed that the plain ipython command doesn't use zmq, so that was a big misunderstanding I had about the new ipython. So never mind what I said about sockets.

Changed 10 years ago by mhansen

comment:32 Changed 10 years ago by mhansen

Okay, I fixed the problem with edit_module.

Sorry I wasn't clear about the default IPython not using zmq. There is a experimental console shell that uses two processes which can be accessed with "ipython console", but we aren't using that at all.

comment:33 Changed 10 years ago by jason

  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • Work issues patches don't apply deleted
Note: See TracTickets for help on using tickets.