Repair math-readline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script

[jdemeyer]

The script src/bin/math-readline (from #16703), which is used by the Mathematica pexpect interface and by mathematica_console, puts the terminal in a bad state (due to #20173). There is also this problem reported by John Palmieri:

If I run

   $ sage --sh
   $ math-readline

then 19 out of 20 times, it fails immediately, but every now and then, maybe when the computer is more heavily loaded, it does something else: it says

   /bin/sh: math: command not found

(as with the immediate failures) and then it waits for input.

The patch on this ticket:

• makes math-readline useful again for interactive use (this also repairs echoing of mathematica_console).
• Changes the Mathematica pexpect interface so that it does not rely on a helper script any more.

[jdemeyer]

I don't actually have Mathematica installed, so I'm just shooting in the dark here. Please test!

---

New commits:

​b2ac334	Run plain Mathematica in pexpect interface
​cfa1c55	Fix math-readline script (reverting Trac #20173)

[jdemeyer]

By the way, I cannot reproduce nor explain John's problem. So I'm not claiming to fix that, only the terminal issue.

[jhpalmieri]

This seems to solve one of my problems, which is that the issue with math-readline was causing a doctest to time out, and since the doctest now uses math instead of math-readline, the doctest seems to work better. It's hard to know for sure since it was an intermittent problem, but with the current branch and a heavily loaded system, I had no timeouts in many attempts for src/doctest/external.py. When I reverted the change so as to run Mathematica using math-readline, I immediately got a timeout.

Summarizing: since I don't have mathematica installed, I really only care about the timeout in the doctest. The math-readline script may still be broken, but I don't care. Now people with mathematica need to test this.

[mkoeppe]

Changing mathematica pexpect from math-readline to math breaks the pexpect interface -- it undoes the work that was done in #16703.

[jdemeyer]

Neither 8 nor #16703 explains how the Mathematica breaks, what goes wrong?

It's really hard to fix a problem when you don't even know what the problem is.

[mkoeppe]

Replying to jdemeyer:

Neither 8 nor #16703 explains how the Mathematica breaks, what goes wrong?

It's really hard to fix a problem when you don't even know what the problem is.

With math, mathematica('1+2') hangs indefinitely. IIRC, the reason is that there are no prompts, and consequently pexpect hangs.

[jdemeyer]

Replying to mkoeppe:

the reason is that there are no prompts

If so, how would the math-readline script fix that? That script is not writing prompts.

I feel like the patch at #16703 is a bad work-around for the Mathematica issue that #16703 fixes.

Anyway, there is not much that I can do with Mathematica being closed source...

[mkoeppe]

Replying to jdemeyer:

Replying to mkoeppe:

the reason is that there are no prompts

If so, how would the math-readline script fix that? That script is not writing prompts.

Without investigating further, I think that Mathematica is trying to be helpful and does not write prompts in "non-interactive" situations. I do not know the precise mechanism. The math-readline script seems to trick Mathematica into interactive mode.

I feel like the patch at #16703 is a bad work-around for the Mathematica issue that #16703 fixes.

Anyway, there is not much that I can do with Mathematica being closed source...

I agree with both.

[jdemeyer]

Replying to mkoeppe:

The math-readline script seems to trick Mathematica into interactive mode.

The strange thing is that pexpect is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between a pexpect session or, say, an xterm or SSH session.

[mkoeppe]

Replying to jdemeyer:

Replying to mkoeppe:

The math-readline script seems to trick Mathematica into interactive mode.

The strange thing is that pexpect is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between a pexpect session or, say, an xterm or SSH session.

I have investigated further and have implemented two solutions, which can be selected by use_pipe=False or True. In either mode, the math-readline script is no longer used by the Mathematica pexpect interface (except for mathematica_console). See comments in mathematica.py for more info.

Eliminating the use of a helper script also makes it more convenient to work with a remote Mathematica server and is also slightly faster. I have tested with Mathematica 11 on Mac OS X, both locally and with server=localhost.

This could use some testing on different platforms and with different Mathematica versions.

---

New commits:

​04f69a2

Mathematica.__init__: Implement two modes that work without helper script math-readline

[dimpase]

testing on Linux with Mathematica 9.0, I get a timeout:

sage -tp --long --optional=database_gap,gap_packages,mpir,python2,mathematica,sage src/sage/interfaces/mathematica.py 
...
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 901 ##
0
sage: from sage.repl.rich_output import get_display_manager ## line 917 ##
sage: dm = get_display_manager() ## line 918 ##
sage: P = mathematica('Plot[Sin[x],{x,-2Pi,4Pi}]')   # optional - mathematica ## line 919 ##
sage: P._rich_repr_(dm)                              # optional - mathematica ## line 920 ##

**********************************************************************
----------------------------------------------------------------------
sage -t --long --warn-long 73.2 src/sage/interfaces/mathematica.py  # Timed out
----------------------------------------------------------------------
Total time for all tests: 1800.0 seconds

this might have to do with the fact that I do it in screen, as apparently it gets stuck while trying to display stuff.

[dimpase]

On the other hand, if I run tests as in comment:18 in a "normal" terminal, they work just fine. Thus I don't know whether or not this is a problem---I'm inclined to say that it's OK. Or perhaps one should tag this test like:

sage: P._rich_repr_(dm)    # optional - mathematica,goodconsole

and add a comment saying that this is to prevent a hang if running in screen.

[mkoeppe]

This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:

sage: P._rich_repr_(dm)
OutputImagePng container

[dimpase]

Replying to mkoeppe:

This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:

sage: P._rich_repr_(dm)
OutputImagePng container

In a screen session, this just hangs. (P.show() hands too)

In terminal, it works---there is a "flashing" movement going through the display, as if it quickly pops something out and closes it. And P.show() displays the graph. Don't know, it might be something to do with X11 in general, or X11 specifics on this particular host.

[mkoeppe]

Is the DISPLAY variable set in your screen session? Could you check if this problem goes away if you unset DISPLAY first? In my opinion, the doctests should never do anything using X11.

[dimpase]

Replying to mkoeppe:

Is the DISPLAY variable set in your screen session? Could you check if this problem goes away if you unset DISPLAY first?

same behaviour.

In my opinion, the doctests should never do anything using X11.

I don't really know why this happens, but it has to do with Mathematica interaction, apparently. If I instead do

P=plot(sin(x))
P._rich_repr_(dm)

then it does work, outputting OutputImagePng container, as expected.

[mkoeppe]

These seem relevant:

• ​https://mathematica.stackexchange.com/questions/5026/generating-figures-over-remote-connection-using-terminal
• ​https://groups.google.com/forum/#!topic/comp.soft-sys.math.mathematica/qILC4ZCEvKk

Essentially, these doctests should be marked "on Linux, need DISPLAY"...

[slabbe]

Replying to mkoeppe:

Cc slabbe dunfield added

I do not have access to a mathematica licence anymore since I finished my postdoc in Liège some months ago.

[dimpase]

I would be happy give it positive review after a change to this iffy doctest.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d021b5e	Run plain Mathematica in pexpect interface
​f7ab81d	Fix math-readline script (reverting Trac #20173)
​86234ea	Mathematica.__init__: Implement two modes that work without helper script math-readline
​b313a76	Mark up iffy doctest as optional - mathematica mathematicafrontend

[mkoeppe]

OK, I've invented an optional tag for this. Can't test whether more doctests in this file need this markup as I don't have a Mathematica license on Linux. (Rebased on 8.0)

[dimpase]

OK, great, thanks!

[vbraun]

has_mathematica() now times out when it is not installed

sage -t --long src/sage/doctest/external.py
    Timed out
**********************************************************************
Tests run before process (pid=10167) timed out:
sage: from sage.doctest.external import has_internet ## line 38 ##
sage: has_internet() # random ## line 39 ##
False
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 41 ##
0
sage: from sage.doctest.external import has_latex ## line 55 ##
sage: has_latex() # random ## line 56 ##
Error: PDFLaTeX does not seem to be installed.  Download it from
ctan.org and try again.
True
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 58 ##
0
sage: from sage.doctest.external import has_magma ## line 77 ##
sage: has_magma() # random ## line 78 ##
False
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 80 ##
0
sage: from sage.doctest.external import has_matlab ## line 94 ##
sage: has_matlab() # random ## line 95 ##
False
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 97 ##
0
sage: from sage.doctest.external import has_mathematica ## line 111 ##
sage: has_mathematica() # random ## line 112 ##
------------------------------------------------------------------------

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e1ddf30	Expect: Optional arg ssh_options for __init__ and set_server_and_command
​40228c0	Mathematica: For use_pipe, use ssh -T on server and math-pipe locally

[mkoeppe]

Here is a solution.

But there may be a much simpler solution for this whole problem addressed in #16703, #20173 that may have been overlooked - namely to use math -rawterm (​http://reference.wolfram.com/language/tutorial/UsingATextBasedInterface.html.en). (Not tried yet.)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6980e18	Run plain Mathematica in pexpect interface
​2e8f304	Fix math-readline script (reverting Trac #20173)
​bc82252	Mathematica.__init__: Implement two modes that work without helper script math-readline
​12a4379	Mark up iffy doctest as optional - mathematica mathematicafrontend
​8492f52	Expect: Optional arg ssh_options for __init__ and set_server_and_command
​b3a33c5	Mathematica: For use_pipe, use ssh -T on server and math-pipe locally

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​596dbc0

Mathematica.__init__: Simplify using 'math -rawterm'

[mkoeppe]

Now here's the solution using math -rawterm. Much simpler and seems to work well. Needs review.

[tscrim]

I am trying this out with Ubuntu 14.04 LTS and Mathematica 11.1.0, and I am running into some problems with mathematica_console(). The first is that I am not seeing my input in the console:

sage: mathematica_console()
Mathematica 11.1.0 Kernel for Linux x86 (64-bit)
Copyright 1988-2017 Wolfram Research, Inc.

In[1]:= 
Out[1]= 4

The second is once I quit Sage, it no longer restores the echo of my terminal (i.e., I have to manually type in stty echo before it is back to normal). Similarly if I immediately go into, e.g., gap_console(). Although within Sage I still get the echo, but I am guessing that has more to do with iPython than with stty. I also removed the additional argument to command, but that had no effect. Instead, what solved it was passing readline=False to mathematica_console, but I don't understand enough to figure out what is going on. At the very least, this should not be the default behavior.

[jhpalmieri]

The input issue seems to have been caused by #20173. This is what is meant in the description here by "puts the terminal in a bad state".

[tscrim]

I see. I thought this ticket was suppose to fix that? Or was #20173 supposed to be reverted as part of this ticket?

[mkoeppe]

Dima, did you install the changed scripts that come with this ticket, using make?

[tscrim]

Replying to mkoeppe:

Dima, did you install the changed scripts that come with this ticket, using make?

I'm not Dima, but I wish I was. :D I'm assuming you were talking to me.

I didn't think I would need to re-run make since it was a modification to a script in bin. After doing a make and trying again it worked.

I agree that it works fairly well. There are some issues with hitting <tab> (as I am too well-trained in tab-completion to not do it) and then backspacing, but that can be a follow-up ticket if it even is on our end.

[dimpase]

Oops, sorry, I overlooked the activity on this ticket... But Travis is impersonating me just fine here :-)

[mkoeppe]

Apologies, I misread who was the commenter in comment 37. Travis, thanks for reviewing!

In a follow-up ticket, one could change the default of mathematica_console to readline=False, because all modern mathematica versions support line editing already. I didn't want to do this change in the same ticket.