Opened 6 years ago
Closed 5 years ago
#23112 closed defect (fixed)
Repair mathreadline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script
Reported by:  Jeroen Demeyer  Owned by:  

Priority:  major  Milestone:  sage8.1 
Component:  interfaces  Keywords:  
Cc:  John Palmieri, Matthias Köppe, Dima Pasechnik, Miguel Marco, Volker Braun, Jakob Kroeker, Salvatore Stella, Yuan Zhou, Sébastien Labbé, Nathan Dunfield, Emmanuel Charpentier, Travis Scrimshaw  Merged in:  
Authors:  Jeroen Demeyer, Matthias Koeppe  Reviewers:  Dima Pasechnik, Travis Scrimshaw 
Report Upstream:  N/A  Work issues:  
Branch:  596dbc0 (Commits, GitHub, GitLab)  Commit:  596dbc0289c943613699b620fe17638fa35d1af8 
Dependencies:  Stopgaps: 
Description (last modified by )
The script src/bin/mathreadline
(from #16703), which is used by the Mathematica
pexpect interface and by mathematica_console
, puts the terminal in a bad state (due to #20173). There is also this problem reported by John Palmieri:
If I run
$ sage sh $ mathreadline
then 19 out of 20 times, it fails immediately, but every now and then, maybe when the computer is more heavily loaded, it does something else: it says
/bin/sh: math: command not found
(as with the immediate failures) and then it waits for input.
The patch on this ticket:
 makes
mathreadline
useful again for interactive use (this also repairs echoing ofmathematica_console
).  Changes the
Mathematica
pexpect interface so that it does not rely on a helper script any more.
Change History (45)
comment:1 Changed 6 years ago by
Description:  modified (diff) 

Summary:  mathreadline script is broken if there is no Mathematica installed → mathreadline script is broken 
comment:2 Changed 6 years ago by
Cc:  John Palmieri added 

comment:3 Changed 6 years ago by
Cc:  Matthias Köppe added 

comment:4 Changed 6 years ago by
Branch:  → u/jdemeyer/math_readline_script_is_broken 

comment:5 Changed 6 years ago by
Authors:  → Jeroen Demeyer 

Commit:  → cfa1c551ae754f59a48886f9b390fcc095d8e83b 
Status:  new → needs_review 
comment:6 Changed 6 years ago by
By the way, I cannot reproduce nor explain John's problem. So I'm not claiming to fix that, only the terminal issue.
comment:7 Changed 5 years ago by
This seems to solve one of my problems, which is that the issue with mathreadline
was causing a doctest to time out, and since the doctest now uses math
instead of mathreadline
, the doctest seems to work better. It's hard to know for sure since it was an intermittent problem, but with the current branch and a heavily loaded system, I had no timeouts in many attempts for src/doctest/external.py
. When I reverted the change so as to run Mathematica using mathreadline, I immediately got a timeout.
Summarizing: since I don't have mathematica installed, I really only care about the timeout in the doctest. The mathreadline script may still be broken, but I don't care. Now people with mathematica need to test this.
comment:8 Changed 5 years ago by
Status:  needs_review → needs_work 

Changing mathematica pexpect from mathreadline
to math
breaks the pexpect interface  it undoes the work that was done in #16703.
comment:9 followup: 10 Changed 5 years ago by
comment:10 followup: 11 Changed 5 years ago by
Replying to jdemeyer:
Neither 8 nor #16703 explains how the Mathematica breaks, what goes wrong?
It's really hard to fix a problem when you don't even know what the problem is.
With math
, mathematica('1+2')
hangs indefinitely.
IIRC, the reason is that there are no prompts, and consequently pexpect hangs.
comment:11 followup: 12 Changed 5 years ago by
Replying to mkoeppe:
the reason is that there are no prompts
If so, how would the mathreadline
script fix that? That script is not writing prompts.
I feel like the patch at #16703 is a bad workaround for the Mathematica issue that #16703 fixes.
Anyway, there is not much that I can do with Mathematica being closed source...
comment:12 followup: 13 Changed 5 years ago by
Replying to jdemeyer:
Replying to mkoeppe:
the reason is that there are no prompts
If so, how would the
mathreadline
script fix that? That script is not writing prompts.
Without investigating further, I think that Mathematica is trying to be helpful and does not write prompts in "noninteractive" situations. I do not know the precise mechanism. The mathreadline script seems to trick Mathematica into interactive mode.
I feel like the patch at #16703 is a bad workaround for the Mathematica issue that #16703 fixes.
Anyway, there is not much that I can do with Mathematica being closed source...
I agree with both.
comment:13 followup: 16 Changed 5 years ago by
Replying to mkoeppe:
The mathreadline script seems to trick Mathematica into interactive mode.
The strange thing is that pexpect
is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between a pexpect
session or, say, an xterm
or SSH session.
comment:14 Changed 5 years ago by
Authors:  Jeroen Demeyer → Jeroen Demeyer, Matthias Koeppe 

Cc:  Dima Pasechnik Miguel Marco Volker Braun Jakob Kroeker Salvatore Stella added 
Component:  scripts → interfaces 
Description:  modified (diff) 
Summary:  mathreadline script is broken → Repair mathreadline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script 
comment:15 Changed 5 years ago by
Branch:  u/jdemeyer/math_readline_script_is_broken → u/mkoeppe/math_readline_script_is_broken 

comment:16 Changed 5 years ago by
Cc:  Yuan Zhou added 

Commit:  cfa1c551ae754f59a48886f9b390fcc095d8e83b → 04f69a2243fb37ebb23e46673484ecb29bb71249 
Description:  modified (diff) 
Status:  needs_work → needs_review 
Replying to jdemeyer:
Replying to mkoeppe:
The mathreadline script seems to trick Mathematica into interactive mode.
The strange thing is that
pexpect
is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between apexpect
session or, say, anxterm
or SSH session.
I have investigated further and have implemented two solutions, which can be selected by use_pipe=False
or True
. In either mode, the mathreadline
script is no longer used by the Mathematica
pexpect interface (except for mathematica_console
). See comments in mathematica.py
for more info.
Eliminating the use of a helper script also makes it more convenient to work with a remote Mathematica server and is also slightly faster. I have tested with Mathematica 11 on Mac OS X, both locally and with server=localhost
.
This could use some testing on different platforms and with different Mathematica versions.
New commits:
04f69a2  Mathematica.__init__: Implement two modes that work without helper script mathreadline

comment:17 followup: 25 Changed 5 years ago by
Cc:  Sébastien Labbé Nathan Dunfield added 

comment:18 Changed 5 years ago by
testing on Linux with Mathematica 9.0, I get a timeout:
sage tp long optional=database_gap,gap_packages,mpir,python2,mathematica,sage src/sage/interfaces/mathematica.py ... sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 901 ## 0 sage: from sage.repl.rich_output import get_display_manager ## line 917 ## sage: dm = get_display_manager() ## line 918 ## sage: P = mathematica('Plot[Sin[x],{x,2Pi,4Pi}]') # optional  mathematica ## line 919 ## sage: P._rich_repr_(dm) # optional  mathematica ## line 920 ## **********************************************************************  sage t long warnlong 73.2 src/sage/interfaces/mathematica.py # Timed out  Total time for all tests: 1800.0 seconds
this might have to do with the fact that I do it in screen
, as apparently it gets stuck while trying to display stuff.
comment:19 Changed 5 years ago by
On the other hand, if I run tests as in comment:18 in a "normal" terminal, they work just fine. Thus I don't know whether or not this is a problemI'm inclined to say that it's OK. Or perhaps one should tag this test like:
sage: P._rich_repr_(dm) # optional  mathematica,goodconsole
and add a comment saying that this is to prevent a hang if running in screen
.
comment:20 followup: 21 Changed 5 years ago by
This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:
sage: P._rich_repr_(dm) OutputImagePng container
comment:21 Changed 5 years ago by
Replying to mkoeppe:
This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:
sage: P._rich_repr_(dm) OutputImagePng container
In a screen session, this just hangs. (P.show()
hands too)
In terminal, it worksthere is a "flashing" movement going through the display, as if it quickly pops something out and closes it.
And P.show()
displays the graph. Don't know, it might be something to do with X11 in general, or X11 specifics on this particular host.
comment:22 followup: 23 Changed 5 years ago by
Is the DISPLAY
variable set in your screen session?
Could you check if this problem goes away if you unset DISPLAY
first?
In my opinion, the doctests should never do anything using X11.
comment:23 Changed 5 years ago by
Replying to mkoeppe:
Is the
DISPLAY
variable set in your screen session? Could you check if this problem goes away if you unsetDISPLAY
first?
same behaviour.
In my opinion, the doctests should never do anything using X11.
I don't really know why this happens, but it has to do with Mathematica interaction, apparently. If I instead do
P=plot(sin(x)) P._rich_repr_(dm)
then it does work, outputting OutputImagePng container
, as expected.
comment:24 Changed 5 years ago by
These seem relevant:
 https://mathematica.stackexchange.com/questions/5026/generatingfiguresoverremoteconnectionusingterminal
 https://groups.google.com/forum/#!topic/comp.softsys.math.mathematica/qILC4ZCEvKk
Essentially, these doctests should be marked "on Linux, need DISPLAY"...
comment:25 Changed 5 years ago by
Replying to mkoeppe:
Cc slabbe dunfield added
I do not have access to a mathematica licence anymore since I finished my postdoc in Liège some months ago.
comment:26 Changed 5 years ago by
I would be happy give it positive review after a change to this iffy doctest.
comment:27 Changed 5 years ago by
Commit:  04f69a2243fb37ebb23e46673484ecb29bb71249 → b313a76b1892e908fb34b20b321d47dc1188a154 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
d021b5e  Run plain Mathematica in pexpect interface

f7ab81d  Fix mathreadline script (reverting Trac #20173)

86234ea  Mathematica.__init__: Implement two modes that work without helper script mathreadline

b313a76  Mark up iffy doctest as optional  mathematica mathematicafrontend

comment:28 Changed 5 years ago by
OK, I've invented an optional
tag for this. Can't test whether more doctests in this file need this markup as I don't have a Mathematica license on Linux.
(Rebased on 8.0)
comment:29 Changed 5 years ago by
Milestone:  sage8.0 → sage8.1 

Reviewers:  → Dima Pasechnik 
Status:  needs_review → positive_review 
OK, great, thanks!
comment:30 Changed 5 years ago by
Status:  positive_review → needs_work 

has_mathematica() now times out when it is not installed
sage t long src/sage/doctest/external.py Timed out ********************************************************************** Tests run before process (pid=10167) timed out: sage: from sage.doctest.external import has_internet ## line 38 ## sage: has_internet() # random ## line 39 ## False sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 41 ## 0 sage: from sage.doctest.external import has_latex ## line 55 ## sage: has_latex() # random ## line 56 ## Error: PDFLaTeX does not seem to be installed. Download it from ctan.org and try again. True sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 58 ## 0 sage: from sage.doctest.external import has_magma ## line 77 ## sage: has_magma() # random ## line 78 ## False sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 80 ## 0 sage: from sage.doctest.external import has_matlab ## line 94 ## sage: has_matlab() # random ## line 95 ## False sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 97 ## 0 sage: from sage.doctest.external import has_mathematica ## line 111 ## sage: has_mathematica() # random ## line 112 ## 
comment:31 Changed 5 years ago by
Commit:  b313a76b1892e908fb34b20b321d47dc1188a154 → 40228c0885a728bc0cedb92f4c746cde72711446 

comment:32 Changed 5 years ago by
Here is a solution.
But there may be a much simpler solution for this whole problem addressed in #16703, #20173 that may have been overlooked  namely to use math rawterm
(http://reference.wolfram.com/language/tutorial/UsingATextBasedInterface.html.en).
(Not tried yet.)
comment:33 Changed 5 years ago by
Commit:  40228c0885a728bc0cedb92f4c746cde72711446 → b3a33c5c6639afadadde80ef762b890b53988710 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
6980e18  Run plain Mathematica in pexpect interface

2e8f304  Fix mathreadline script (reverting Trac #20173)

bc82252  Mathematica.__init__: Implement two modes that work without helper script mathreadline

12a4379  Mark up iffy doctest as optional  mathematica mathematicafrontend

8492f52  Expect: Optional arg ssh_options for __init__ and set_server_and_command

b3a33c5  Mathematica: For use_pipe, use ssh T on server and mathpipe locally

comment:34 Changed 5 years ago by
Commit:  b3a33c5c6639afadadde80ef762b890b53988710 → 596dbc0289c943613699b620fe17638fa35d1af8 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
596dbc0  Mathematica.__init__: Simplify using 'math rawterm'

comment:35 Changed 5 years ago by
Status:  needs_work → needs_review 

Now here's the solution using math rawterm
. Much simpler and seems to work well. Needs review.
comment:36 Changed 5 years ago by
Cc:  Emmanuel Charpentier added 

comment:37 Changed 5 years ago by
Reviewers:  Dima Pasechnik → Dima Pasechnik, Travis Scrimshaw 

Status:  needs_review → needs_work 
I am trying this out with Ubuntu 14.04 LTS and Mathematica 11.1.0, and I am running into some problems with mathematica_console()
. The first is that I am not seeing my input in the console:
sage: mathematica_console() Mathematica 11.1.0 Kernel for Linux x86 (64bit) Copyright 19882017 Wolfram Research, Inc. In[1]:= Out[1]= 4
The second is once I quit Sage, it no longer restores the echo of my terminal (i.e., I have to manually type in stty echo
before it is back to normal). Similarly if I immediately go into, e.g., gap_console()
. Although within Sage I still get the echo, but I am guessing that has more to do with iPython than with stty. I also removed the additional argument to command
, but that had no effect. Instead, what solved it was passing readline=False
to mathematica_console
, but I don't understand enough to figure out what is going on. At the very least, this should not be the default behavior.
comment:38 Changed 5 years ago by
Cc:  Travis Scrimshaw added 

comment:39 Changed 5 years ago by
The input issue seems to have been caused by #20173. This is what is meant in the description here by "puts the terminal in a bad state".
comment:40 Changed 5 years ago by
I see. I thought this ticket was suppose to fix that? Or was #20173 supposed to be reverted as part of this ticket?
Edit  No, it has been removed; I just didn't look at my diff correctly.
comment:41 followup: 42 Changed 5 years ago by
Dima, did you install the changed scripts that come with this ticket, using make
?
comment:42 Changed 5 years ago by
Status:  needs_work → positive_review 

Replying to mkoeppe:
Dima, did you install the changed scripts that come with this ticket, using
make
?
I'm not Dima, but I wish I was. :D I'm assuming you were talking to me.
I didn't think I would need to rerun make
since it was a modification to a script in bin
. After doing a make
and trying again it worked.
I agree that it works fairly well. There are some issues with hitting <tab>
(as I am too welltrained in tabcompletion to not do it) and then backspacing, but that can be a followup ticket if it even is on our end.
comment:43 Changed 5 years ago by
Oops, sorry, I overlooked the activity on this ticket... But Travis is impersonating me just fine here :)
comment:44 Changed 5 years ago by
Apologies, I misread who was the commenter in comment 37. Travis, thanks for reviewing!
In a followup ticket, one could change the default of mathematica_console
to readline=False
, because all modern mathematica versions support line editing already. I didn't want to do this change in the same ticket.
comment:45 Changed 5 years ago by
Branch:  u/mkoeppe/math_readline_script_is_broken → 596dbc0289c943613699b620fe17638fa35d1af8 

Resolution:  → fixed 
Status:  positive_review → closed 
I don't actually have Mathematica installed, so I'm just shooting in the dark here. Please test!
New commits:
Run plain Mathematica in pexpect interface
Fix mathreadline script (reverting Trac #20173)