Opened 5 years ago

Closed 4 years ago

#23112 closed defect (fixed)

Repair math-readline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script

Reported by: jdemeyer Owned by:
Priority: major Milestone: sage-8.1
Component: interfaces Keywords:
Cc: jhpalmieri, mkoeppe, dimpase, mmarco, vbraun, jakobkroeker, etn40ff, yzh, slabbe, dunfield, charpent, tscrim Merged in:
Authors: Jeroen Demeyer, Matthias Koeppe Reviewers: Dima Pasechnik, Travis Scrimshaw
Report Upstream: N/A Work issues:
Branch: 596dbc0 (Commits, GitHub, GitLab) Commit: 596dbc0289c943613699b620fe17638fa35d1af8
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

The script src/bin/math-readline (from #16703), which is used by the Mathematica pexpect interface and by mathematica_console, puts the terminal in a bad state (due to #20173). There is also this problem reported by John Palmieri:

If I run

   $ sage --sh
   $ math-readline

then 19 out of 20 times, it fails immediately, but every now and then, maybe when the computer is more heavily loaded, it does something else: it says

   /bin/sh: math: command not found

(as with the immediate failures) and then it waits for input.

The patch on this ticket:

  • makes math-readline useful again for interactive use (this also repairs echoing of mathematica_console).
  • Changes the Mathematica pexpect interface so that it does not rely on a helper script any more.

Change History (45)

comment:1 Changed 5 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from math-readline script is broken if there is no Mathematica installed to math-readline script is broken

comment:2 Changed 5 years ago by jhpalmieri

  • Cc jhpalmieri added

comment:3 Changed 4 years ago by jdemeyer

  • Cc mkoeppe added

comment:4 Changed 4 years ago by jdemeyer

  • Branch set to u/jdemeyer/math_readline_script_is_broken

comment:5 Changed 4 years ago by jdemeyer

  • Authors set to Jeroen Demeyer
  • Commit set to cfa1c551ae754f59a48886f9b390fcc095d8e83b
  • Status changed from new to needs_review

I don't actually have Mathematica installed, so I'm just shooting in the dark here. Please test!


New commits:

b2ac334Run plain Mathematica in pexpect interface
cfa1c55Fix math-readline script (reverting Trac #20173)

comment:6 Changed 4 years ago by jdemeyer

By the way, I cannot reproduce nor explain John's problem. So I'm not claiming to fix that, only the terminal issue.

comment:7 Changed 4 years ago by jhpalmieri

This seems to solve one of my problems, which is that the issue with math-readline was causing a doctest to time out, and since the doctest now uses math instead of math-readline, the doctest seems to work better. It's hard to know for sure since it was an intermittent problem, but with the current branch and a heavily loaded system, I had no timeouts in many attempts for src/doctest/external.py. When I reverted the change so as to run Mathematica using math-readline, I immediately got a timeout.

Summarizing: since I don't have mathematica installed, I really only care about the timeout in the doctest. The math-readline script may still be broken, but I don't care. Now people with mathematica need to test this.

comment:8 Changed 4 years ago by mkoeppe

  • Status changed from needs_review to needs_work

Changing mathematica pexpect from math-readline to math breaks the pexpect interface -- it undoes the work that was done in #16703.

comment:9 follow-up: Changed 4 years ago by jdemeyer

Neither 8 nor #16703 explains how the Mathematica breaks, what goes wrong?

It's really hard to fix a problem when you don't even know what the problem is.

comment:10 in reply to: ↑ 9 ; follow-up: Changed 4 years ago by mkoeppe

Replying to jdemeyer:

Neither 8 nor #16703 explains how the Mathematica breaks, what goes wrong?

It's really hard to fix a problem when you don't even know what the problem is.

With math, mathematica('1+2') hangs indefinitely. IIRC, the reason is that there are no prompts, and consequently pexpect hangs.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 4 years ago by jdemeyer

Replying to mkoeppe:

the reason is that there are no prompts

If so, how would the math-readline script fix that? That script is not writing prompts.

I feel like the patch at #16703 is a bad work-around for the Mathematica issue that #16703 fixes.

Anyway, there is not much that I can do with Mathematica being closed source...

comment:12 in reply to: ↑ 11 ; follow-up: Changed 4 years ago by mkoeppe

Replying to jdemeyer:

Replying to mkoeppe:

the reason is that there are no prompts

If so, how would the math-readline script fix that? That script is not writing prompts.

Without investigating further, I think that Mathematica is trying to be helpful and does not write prompts in "non-interactive" situations. I do not know the precise mechanism. The math-readline script seems to trick Mathematica into interactive mode.

I feel like the patch at #16703 is a bad work-around for the Mathematica issue that #16703 fixes.

Anyway, there is not much that I can do with Mathematica being closed source...

I agree with both.

comment:13 in reply to: ↑ 12 ; follow-up: Changed 4 years ago by jdemeyer

Replying to mkoeppe:

The math-readline script seems to trick Mathematica into interactive mode.

The strange thing is that pexpect is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between a pexpect session or, say, an xterm or SSH session.

comment:14 Changed 4 years ago by mkoeppe

  • Authors changed from Jeroen Demeyer to Jeroen Demeyer, Matthias Koeppe
  • Cc dimpase mmarco vbraun jakobkroeker etn40ff added
  • Component changed from scripts to interfaces
  • Description modified (diff)
  • Summary changed from math-readline script is broken to Repair math-readline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script

comment:15 Changed 4 years ago by mkoeppe

  • Branch changed from u/jdemeyer/math_readline_script_is_broken to u/mkoeppe/math_readline_script_is_broken

comment:16 in reply to: ↑ 13 Changed 4 years ago by mkoeppe

  • Cc yzh added
  • Commit changed from cfa1c551ae754f59a48886f9b390fcc095d8e83b to 04f69a2243fb37ebb23e46673484ecb29bb71249
  • Description modified (diff)
  • Status changed from needs_work to needs_review

Replying to jdemeyer:

Replying to mkoeppe:

The math-readline script seems to trick Mathematica into interactive mode.

The strange thing is that pexpect is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between a pexpect session or, say, an xterm or SSH session.

I have investigated further and have implemented two solutions, which can be selected by use_pipe=False or True. In either mode, the math-readline script is no longer used by the Mathematica pexpect interface (except for mathematica_console). See comments in mathematica.py for more info.

Eliminating the use of a helper script also makes it more convenient to work with a remote Mathematica server and is also slightly faster. I have tested with Mathematica 11 on Mac OS X, both locally and with server=localhost.

This could use some testing on different platforms and with different Mathematica versions.


New commits:

04f69a2Mathematica.__init__: Implement two modes that work without helper script math-readline

comment:17 follow-up: Changed 4 years ago by mkoeppe

  • Cc slabbe dunfield added

comment:18 Changed 4 years ago by dimpase

testing on Linux with Mathematica 9.0, I get a timeout:

sage -tp --long --optional=database_gap,gap_packages,mpir,python2,mathematica,sage src/sage/interfaces/mathematica.py 
...
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 901 ##
0
sage: from sage.repl.rich_output import get_display_manager ## line 917 ##
sage: dm = get_display_manager() ## line 918 ##
sage: P = mathematica('Plot[Sin[x],{x,-2Pi,4Pi}]')   # optional - mathematica ## line 919 ##
sage: P._rich_repr_(dm)                              # optional - mathematica ## line 920 ##

**********************************************************************
----------------------------------------------------------------------
sage -t --long --warn-long 73.2 src/sage/interfaces/mathematica.py  # Timed out
----------------------------------------------------------------------
Total time for all tests: 1800.0 seconds

this might have to do with the fact that I do it in screen, as apparently it gets stuck while trying to display stuff.

comment:19 Changed 4 years ago by dimpase

On the other hand, if I run tests as in comment:18 in a "normal" terminal, they work just fine. Thus I don't know whether or not this is a problem---I'm inclined to say that it's OK. Or perhaps one should tag this test like:

sage: P._rich_repr_(dm)    # optional - mathematica,goodconsole

and add a comment saying that this is to prevent a hang if running in screen.

comment:20 follow-up: Changed 4 years ago by mkoeppe

This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:

sage: P._rich_repr_(dm)
OutputImagePng container

comment:21 in reply to: ↑ 20 Changed 4 years ago by dimpase

Replying to mkoeppe:

This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:

sage: P._rich_repr_(dm)
OutputImagePng container

In a screen session, this just hangs. (P.show() hands too)

In terminal, it works---there is a "flashing" movement going through the display, as if it quickly pops something out and closes it. And P.show() displays the graph. Don't know, it might be something to do with X11 in general, or X11 specifics on this particular host.

comment:22 follow-up: Changed 4 years ago by mkoeppe

Is the DISPLAY variable set in your screen session? Could you check if this problem goes away if you unset DISPLAY first? In my opinion, the doctests should never do anything using X11.

comment:23 in reply to: ↑ 22 Changed 4 years ago by dimpase

Replying to mkoeppe:

Is the DISPLAY variable set in your screen session? Could you check if this problem goes away if you unset DISPLAY first?

same behaviour.

In my opinion, the doctests should never do anything using X11.

I don't really know why this happens, but it has to do with Mathematica interaction, apparently. If I instead do

P=plot(sin(x))
P._rich_repr_(dm)

then it does work, outputting OutputImagePng container, as expected.

comment:25 in reply to: ↑ 17 Changed 4 years ago by slabbe

Replying to mkoeppe:

Cc slabbe dunfield added

I do not have access to a mathematica licence anymore since I finished my postdoc in Liège some months ago.

comment:26 Changed 4 years ago by dimpase

I would be happy give it positive review after a change to this iffy doctest.

comment:27 Changed 4 years ago by git

  • Commit changed from 04f69a2243fb37ebb23e46673484ecb29bb71249 to b313a76b1892e908fb34b20b321d47dc1188a154

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d021b5eRun plain Mathematica in pexpect interface
f7ab81dFix math-readline script (reverting Trac #20173)
86234eaMathematica.__init__: Implement two modes that work without helper script math-readline
b313a76Mark up iffy doctest as optional - mathematica mathematicafrontend

comment:28 Changed 4 years ago by mkoeppe

OK, I've invented an optional tag for this. Can't test whether more doctests in this file need this markup as I don't have a Mathematica license on Linux. (Rebased on 8.0)

comment:29 Changed 4 years ago by dimpase

  • Milestone changed from sage-8.0 to sage-8.1
  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

OK, great, thanks!

comment:30 Changed 4 years ago by vbraun

  • Status changed from positive_review to needs_work

has_mathematica() now times out when it is not installed

sage -t --long src/sage/doctest/external.py
    Timed out
**********************************************************************
Tests run before process (pid=10167) timed out:
sage: from sage.doctest.external import has_internet ## line 38 ##
sage: has_internet() # random ## line 39 ##
False
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 41 ##
0
sage: from sage.doctest.external import has_latex ## line 55 ##
sage: has_latex() # random ## line 56 ##
Error: PDFLaTeX does not seem to be installed.  Download it from
ctan.org and try again.
True
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 58 ##
0
sage: from sage.doctest.external import has_magma ## line 77 ##
sage: has_magma() # random ## line 78 ##
False
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 80 ##
0
sage: from sage.doctest.external import has_matlab ## line 94 ##
sage: has_matlab() # random ## line 95 ##
False
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 97 ##
0
sage: from sage.doctest.external import has_mathematica ## line 111 ##
sage: has_mathematica() # random ## line 112 ##
------------------------------------------------------------------------

comment:31 Changed 4 years ago by git

  • Commit changed from b313a76b1892e908fb34b20b321d47dc1188a154 to 40228c0885a728bc0cedb92f4c746cde72711446

Branch pushed to git repo; I updated commit sha1. New commits:

e1ddf30Expect: Optional arg ssh_options for __init__ and set_server_and_command
40228c0Mathematica: For use_pipe, use ssh -T on server and math-pipe locally

comment:32 Changed 4 years ago by mkoeppe

Here is a solution.

But there may be a much simpler solution for this whole problem addressed in #16703, #20173 that may have been overlooked - namely to use math -rawterm (http://reference.wolfram.com/language/tutorial/UsingATextBasedInterface.html.en). (Not tried yet.)

comment:33 Changed 4 years ago by git

  • Commit changed from 40228c0885a728bc0cedb92f4c746cde72711446 to b3a33c5c6639afadadde80ef762b890b53988710

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

6980e18Run plain Mathematica in pexpect interface
2e8f304Fix math-readline script (reverting Trac #20173)
bc82252Mathematica.__init__: Implement two modes that work without helper script math-readline
12a4379Mark up iffy doctest as optional - mathematica mathematicafrontend
8492f52Expect: Optional arg ssh_options for __init__ and set_server_and_command
b3a33c5Mathematica: For use_pipe, use ssh -T on server and math-pipe locally

comment:34 Changed 4 years ago by git

  • Commit changed from b3a33c5c6639afadadde80ef762b890b53988710 to 596dbc0289c943613699b620fe17638fa35d1af8

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

596dbc0Mathematica.__init__: Simplify using 'math -rawterm'

comment:35 Changed 4 years ago by mkoeppe

  • Status changed from needs_work to needs_review

Now here's the solution using math -rawterm. Much simpler and seems to work well. Needs review.

comment:36 Changed 4 years ago by mkoeppe

  • Cc charpent added

comment:37 Changed 4 years ago by tscrim

  • Reviewers changed from Dima Pasechnik to Dima Pasechnik, Travis Scrimshaw
  • Status changed from needs_review to needs_work

I am trying this out with Ubuntu 14.04 LTS and Mathematica 11.1.0, and I am running into some problems with mathematica_console(). The first is that I am not seeing my input in the console:

sage: mathematica_console()
Mathematica 11.1.0 Kernel for Linux x86 (64-bit)
Copyright 1988-2017 Wolfram Research, Inc.

In[1]:= 
Out[1]= 4

The second is once I quit Sage, it no longer restores the echo of my terminal (i.e., I have to manually type in stty echo before it is back to normal). Similarly if I immediately go into, e.g., gap_console(). Although within Sage I still get the echo, but I am guessing that has more to do with iPython than with stty. I also removed the additional argument to command, but that had no effect. Instead, what solved it was passing readline=False to mathematica_console, but I don't understand enough to figure out what is going on. At the very least, this should not be the default behavior.

comment:38 Changed 4 years ago by tscrim

  • Cc tscrim added

comment:39 Changed 4 years ago by jhpalmieri

The input issue seems to have been caused by #20173. This is what is meant in the description here by "puts the terminal in a bad state".

comment:40 Changed 4 years ago by tscrim

I see. I thought this ticket was suppose to fix that? Or was #20173 supposed to be reverted as part of this ticket?

Edit - No, it has been removed; I just didn't look at my diff correctly.

Last edited 4 years ago by tscrim (previous) (diff)

comment:41 follow-up: Changed 4 years ago by mkoeppe

Dima, did you install the changed scripts that come with this ticket, using make?

comment:42 in reply to: ↑ 41 Changed 4 years ago by tscrim

  • Status changed from needs_work to positive_review

Replying to mkoeppe:

Dima, did you install the changed scripts that come with this ticket, using make?

I'm not Dima, but I wish I was. :D I'm assuming you were talking to me.

I didn't think I would need to re-run make since it was a modification to a script in bin. After doing a make and trying again it worked.

I agree that it works fairly well. There are some issues with hitting <tab> (as I am too well-trained in tab-completion to not do it) and then backspacing, but that can be a follow-up ticket if it even is on our end.

comment:43 Changed 4 years ago by dimpase

Oops, sorry, I overlooked the activity on this ticket... But Travis is impersonating me just fine here :-)

comment:44 Changed 4 years ago by mkoeppe

Apologies, I misread who was the commenter in comment 37. Travis, thanks for reviewing!

In a follow-up ticket, one could change the default of mathematica_console to readline=False, because all modern mathematica versions support line editing already. I didn't want to do this change in the same ticket.

comment:45 Changed 4 years ago by vbraun

  • Branch changed from u/mkoeppe/math_readline_script_is_broken to 596dbc0289c943613699b620fe17638fa35d1af8
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.