Opened 6 years ago
Closed 5 years ago
#23112 closed defect (fixed)
Repair math-readline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script
Reported by: | jdemeyer | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-8.1 |
Component: | interfaces | Keywords: | |
Cc: | jhpalmieri, mkoeppe, dimpase, mmarco, vbraun, jakobkroeker, etn40ff, yzh, slabbe, dunfield, charpent, tscrim | Merged in: | |
Authors: | Jeroen Demeyer, Matthias Koeppe | Reviewers: | Dima Pasechnik, Travis Scrimshaw |
Report Upstream: | N/A | Work issues: | |
Branch: | 596dbc0 (Commits, GitHub, GitLab) | Commit: | 596dbc0289c943613699b620fe17638fa35d1af8 |
Dependencies: | Stopgaps: |
Description (last modified by )
The script src/bin/math-readline
(from #16703), which is used by the Mathematica
pexpect interface and by mathematica_console
, puts the terminal in a bad state (due to #20173). There is also this problem reported by John Palmieri:
If I run
$ sage --sh $ math-readline
then 19 out of 20 times, it fails immediately, but every now and then, maybe when the computer is more heavily loaded, it does something else: it says
/bin/sh: math: command not found
(as with the immediate failures) and then it waits for input.
The patch on this ticket:
- makes
math-readline
useful again for interactive use (this also repairs echoing ofmathematica_console
). - Changes the
Mathematica
pexpect interface so that it does not rely on a helper script any more.
Change History (45)
comment:1 Changed 6 years ago by
Description: | modified (diff) |
---|---|
Summary: | math-readline script is broken if there is no Mathematica installed → math-readline script is broken |
comment:2 Changed 6 years ago by
Cc: | jhpalmieri added |
---|
comment:3 Changed 6 years ago by
Cc: | mkoeppe added |
---|
comment:4 Changed 6 years ago by
Branch: | → u/jdemeyer/math_readline_script_is_broken |
---|
comment:5 Changed 6 years ago by
Authors: | → Jeroen Demeyer |
---|---|
Commit: | → cfa1c551ae754f59a48886f9b390fcc095d8e83b |
Status: | new → needs_review |
comment:6 Changed 6 years ago by
By the way, I cannot reproduce nor explain John's problem. So I'm not claiming to fix that, only the terminal issue.
comment:7 Changed 6 years ago by
This seems to solve one of my problems, which is that the issue with math-readline
was causing a doctest to time out, and since the doctest now uses math
instead of math-readline
, the doctest seems to work better. It's hard to know for sure since it was an intermittent problem, but with the current branch and a heavily loaded system, I had no timeouts in many attempts for src/doctest/external.py
. When I reverted the change so as to run Mathematica using math-readline, I immediately got a timeout.
Summarizing: since I don't have mathematica installed, I really only care about the timeout in the doctest. The math-readline script may still be broken, but I don't care. Now people with mathematica need to test this.
comment:8 Changed 6 years ago by
Status: | needs_review → needs_work |
---|
Changing mathematica pexpect from math-readline
to math
breaks the pexpect interface -- it undoes the work that was done in #16703.
comment:9 follow-up: 10 Changed 6 years ago by
comment:10 follow-up: 11 Changed 6 years ago by
Replying to jdemeyer:
Neither 8 nor #16703 explains how the Mathematica breaks, what goes wrong?
It's really hard to fix a problem when you don't even know what the problem is.
With math
, mathematica('1+2')
hangs indefinitely.
IIRC, the reason is that there are no prompts, and consequently pexpect hangs.
comment:11 follow-up: 12 Changed 6 years ago by
Replying to mkoeppe:
the reason is that there are no prompts
If so, how would the math-readline
script fix that? That script is not writing prompts.
I feel like the patch at #16703 is a bad work-around for the Mathematica issue that #16703 fixes.
Anyway, there is not much that I can do with Mathematica being closed source...
comment:12 follow-up: 13 Changed 6 years ago by
Replying to jdemeyer:
Replying to mkoeppe:
the reason is that there are no prompts
If so, how would the
math-readline
script fix that? That script is not writing prompts.
Without investigating further, I think that Mathematica is trying to be helpful and does not write prompts in "non-interactive" situations. I do not know the precise mechanism. The math-readline script seems to trick Mathematica into interactive mode.
I feel like the patch at #16703 is a bad work-around for the Mathematica issue that #16703 fixes.
Anyway, there is not much that I can do with Mathematica being closed source...
I agree with both.
comment:13 follow-up: 16 Changed 6 years ago by
Replying to mkoeppe:
The math-readline script seems to trick Mathematica into interactive mode.
The strange thing is that pexpect
is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between a pexpect
session or, say, an xterm
or SSH session.
comment:14 Changed 6 years ago by
Authors: | Jeroen Demeyer → Jeroen Demeyer, Matthias Koeppe |
---|---|
Cc: | dimpase mmarco vbraun jakobkroeker etn40ff added |
Component: | scripts → interfaces |
Description: | modified (diff) |
Summary: | math-readline script is broken → Repair math-readline script for interactive use, mathematica_console and make Mathematica pexpect work without helper script |
comment:15 Changed 6 years ago by
Branch: | u/jdemeyer/math_readline_script_is_broken → u/mkoeppe/math_readline_script_is_broken |
---|
comment:16 Changed 6 years ago by
Cc: | yzh added |
---|---|
Commit: | cfa1c551ae754f59a48886f9b390fcc095d8e83b → 04f69a2243fb37ebb23e46673484ecb29bb71249 |
Description: | modified (diff) |
Status: | needs_work → needs_review |
Replying to jdemeyer:
Replying to mkoeppe:
The math-readline script seems to trick Mathematica into interactive mode.
The strange thing is that
pexpect
is really supposed to be exactly like an interactive session. As far as the application is concerned, there should be no difference between apexpect
session or, say, anxterm
or SSH session.
I have investigated further and have implemented two solutions, which can be selected by use_pipe=False
or True
. In either mode, the math-readline
script is no longer used by the Mathematica
pexpect interface (except for mathematica_console
). See comments in mathematica.py
for more info.
Eliminating the use of a helper script also makes it more convenient to work with a remote Mathematica server and is also slightly faster. I have tested with Mathematica 11 on Mac OS X, both locally and with server=localhost
.
This could use some testing on different platforms and with different Mathematica versions.
New commits:
04f69a2 | Mathematica.__init__: Implement two modes that work without helper script math-readline
|
comment:17 follow-up: 25 Changed 6 years ago by
Cc: | slabbe dunfield added |
---|
comment:18 Changed 6 years ago by
testing on Linux with Mathematica 9.0, I get a timeout:
sage -tp --long --optional=database_gap,gap_packages,mpir,python2,mathematica,sage src/sage/interfaces/mathematica.py ... sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 901 ## 0 sage: from sage.repl.rich_output import get_display_manager ## line 917 ## sage: dm = get_display_manager() ## line 918 ## sage: P = mathematica('Plot[Sin[x],{x,-2Pi,4Pi}]') # optional - mathematica ## line 919 ## sage: P._rich_repr_(dm) # optional - mathematica ## line 920 ## ********************************************************************** ---------------------------------------------------------------------- sage -t --long --warn-long 73.2 src/sage/interfaces/mathematica.py # Timed out ---------------------------------------------------------------------- Total time for all tests: 1800.0 seconds
this might have to do with the fact that I do it in screen
, as apparently it gets stuck while trying to display stuff.
comment:19 Changed 6 years ago by
On the other hand, if I run tests as in comment:18 in a "normal" terminal, they work just fine. Thus I don't know whether or not this is a problem---I'm inclined to say that it's OK. Or perhaps one should tag this test like:
sage: P._rich_repr_(dm) # optional - mathematica,goodconsole
and add a comment saying that this is to prevent a hang if running in screen
.
comment:20 follow-up: 21 Changed 6 years ago by
This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:
sage: P._rich_repr_(dm) OutputImagePng container
comment:21 Changed 6 years ago by
Replying to mkoeppe:
This is very strange. I can't reproduce this in "screen" here on Mac OS X. What happens if you run these commands interactively in a screen session on your Linux box? Here I see:
sage: P._rich_repr_(dm) OutputImagePng container
In a screen session, this just hangs. (P.show()
hands too)
In terminal, it works---there is a "flashing" movement going through the display, as if it quickly pops something out and closes it.
And P.show()
displays the graph. Don't know, it might be something to do with X11 in general, or X11 specifics on this particular host.
comment:22 follow-up: 23 Changed 6 years ago by
Is the DISPLAY
variable set in your screen session?
Could you check if this problem goes away if you unset DISPLAY
first?
In my opinion, the doctests should never do anything using X11.
comment:23 Changed 6 years ago by
Replying to mkoeppe:
Is the
DISPLAY
variable set in your screen session? Could you check if this problem goes away if you unsetDISPLAY
first?
same behaviour.
In my opinion, the doctests should never do anything using X11.
I don't really know why this happens, but it has to do with Mathematica interaction, apparently. If I instead do
P=plot(sin(x)) P._rich_repr_(dm)
then it does work, outputting OutputImagePng container
, as expected.
comment:24 Changed 6 years ago by
These seem relevant:
- https://mathematica.stackexchange.com/questions/5026/generating-figures-over-remote-connection-using-terminal
- https://groups.google.com/forum/#!topic/comp.soft-sys.math.mathematica/qILC4ZCEvKk
Essentially, these doctests should be marked "on Linux, need DISPLAY"...
comment:25 Changed 6 years ago by
Replying to mkoeppe:
Cc slabbe dunfield added
I do not have access to a mathematica licence anymore since I finished my postdoc in Liège some months ago.
comment:26 Changed 6 years ago by
I would be happy give it positive review after a change to this iffy doctest.
comment:27 Changed 6 years ago by
Commit: | 04f69a2243fb37ebb23e46673484ecb29bb71249 → b313a76b1892e908fb34b20b321d47dc1188a154 |
---|
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
d021b5e | Run plain Mathematica in pexpect interface
|
f7ab81d | Fix math-readline script (reverting Trac #20173)
|
86234ea | Mathematica.__init__: Implement two modes that work without helper script math-readline
|
b313a76 | Mark up iffy doctest as optional - mathematica mathematicafrontend
|
comment:28 Changed 6 years ago by
OK, I've invented an optional
tag for this. Can't test whether more doctests in this file need this markup as I don't have a Mathematica license on Linux.
(Rebased on 8.0)
comment:29 Changed 6 years ago by
Milestone: | sage-8.0 → sage-8.1 |
---|---|
Reviewers: | → Dima Pasechnik |
Status: | needs_review → positive_review |
OK, great, thanks!
comment:30 Changed 6 years ago by
Status: | positive_review → needs_work |
---|
has_mathematica() now times out when it is not installed
sage -t --long src/sage/doctest/external.py Timed out ********************************************************************** Tests run before process (pid=10167) timed out: sage: from sage.doctest.external import has_internet ## line 38 ## sage: has_internet() # random ## line 39 ## False sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 41 ## 0 sage: from sage.doctest.external import has_latex ## line 55 ## sage: has_latex() # random ## line 56 ## Error: PDFLaTeX does not seem to be installed. Download it from ctan.org and try again. True sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 58 ## 0 sage: from sage.doctest.external import has_magma ## line 77 ## sage: has_magma() # random ## line 78 ## False sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 80 ## 0 sage: from sage.doctest.external import has_matlab ## line 94 ## sage: has_matlab() # random ## line 95 ## False sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 97 ## 0 sage: from sage.doctest.external import has_mathematica ## line 111 ## sage: has_mathematica() # random ## line 112 ## ------------------------------------------------------------------------
comment:31 Changed 6 years ago by
Commit: | b313a76b1892e908fb34b20b321d47dc1188a154 → 40228c0885a728bc0cedb92f4c746cde72711446 |
---|
comment:32 Changed 6 years ago by
Here is a solution.
But there may be a much simpler solution for this whole problem addressed in #16703, #20173 that may have been overlooked - namely to use math -rawterm
(http://reference.wolfram.com/language/tutorial/UsingATextBasedInterface.html.en).
(Not tried yet.)
comment:33 Changed 5 years ago by
Commit: | 40228c0885a728bc0cedb92f4c746cde72711446 → b3a33c5c6639afadadde80ef762b890b53988710 |
---|
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
6980e18 | Run plain Mathematica in pexpect interface
|
2e8f304 | Fix math-readline script (reverting Trac #20173)
|
bc82252 | Mathematica.__init__: Implement two modes that work without helper script math-readline
|
12a4379 | Mark up iffy doctest as optional - mathematica mathematicafrontend
|
8492f52 | Expect: Optional arg ssh_options for __init__ and set_server_and_command
|
b3a33c5 | Mathematica: For use_pipe, use ssh -T on server and math-pipe locally
|
comment:34 Changed 5 years ago by
Commit: | b3a33c5c6639afadadde80ef762b890b53988710 → 596dbc0289c943613699b620fe17638fa35d1af8 |
---|
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
596dbc0 | Mathematica.__init__: Simplify using 'math -rawterm'
|
comment:35 Changed 5 years ago by
Status: | needs_work → needs_review |
---|
Now here's the solution using math -rawterm
. Much simpler and seems to work well. Needs review.
comment:36 Changed 5 years ago by
Cc: | charpent added |
---|
comment:37 Changed 5 years ago by
Reviewers: | Dima Pasechnik → Dima Pasechnik, Travis Scrimshaw |
---|---|
Status: | needs_review → needs_work |
I am trying this out with Ubuntu 14.04 LTS and Mathematica 11.1.0, and I am running into some problems with mathematica_console()
. The first is that I am not seeing my input in the console:
sage: mathematica_console() Mathematica 11.1.0 Kernel for Linux x86 (64-bit) Copyright 1988-2017 Wolfram Research, Inc. In[1]:= Out[1]= 4
The second is once I quit Sage, it no longer restores the echo of my terminal (i.e., I have to manually type in stty echo
before it is back to normal). Similarly if I immediately go into, e.g., gap_console()
. Although within Sage I still get the echo, but I am guessing that has more to do with iPython than with stty. I also removed the additional argument to command
, but that had no effect. Instead, what solved it was passing readline=False
to mathematica_console
, but I don't understand enough to figure out what is going on. At the very least, this should not be the default behavior.
comment:38 Changed 5 years ago by
Cc: | tscrim added |
---|
comment:39 Changed 5 years ago by
The input issue seems to have been caused by #20173. This is what is meant in the description here by "puts the terminal in a bad state".
comment:40 Changed 5 years ago by
I see. I thought this ticket was suppose to fix that? Or was #20173 supposed to be reverted as part of this ticket?
comment:41 follow-up: 42 Changed 5 years ago by
Dima, did you install the changed scripts that come with this ticket, using make
?
comment:42 Changed 5 years ago by
Status: | needs_work → positive_review |
---|
Replying to mkoeppe:
Dima, did you install the changed scripts that come with this ticket, using
make
?
I'm not Dima, but I wish I was. :D I'm assuming you were talking to me.
I didn't think I would need to re-run make
since it was a modification to a script in bin
. After doing a make
and trying again it worked.
I agree that it works fairly well. There are some issues with hitting <tab>
(as I am too well-trained in tab-completion to not do it) and then backspacing, but that can be a follow-up ticket if it even is on our end.
comment:43 Changed 5 years ago by
Oops, sorry, I overlooked the activity on this ticket... But Travis is impersonating me just fine here :-)
comment:44 Changed 5 years ago by
Apologies, I misread who was the commenter in comment 37. Travis, thanks for reviewing!
In a follow-up ticket, one could change the default of mathematica_console
to readline=False
, because all modern mathematica versions support line editing already. I didn't want to do this change in the same ticket.
comment:45 Changed 5 years ago by
Branch: | u/mkoeppe/math_readline_script_is_broken → 596dbc0289c943613699b620fe17638fa35d1af8 |
---|---|
Resolution: | → fixed |
Status: | positive_review → closed |
I don't actually have Mathematica installed, so I'm just shooting in the dark here. Please test!
New commits:
Run plain Mathematica in pexpect interface
Fix math-readline script (reverting Trac #20173)