#12221 closed defect (fixed)
Failures in gp pexpect interface with specific length of $DOT_SAGE using a "screen" terminal
Reported by: | jdemeyer | Owned by: | was |
---|---|---|---|
Priority: | blocker | Milestone: | sage-5.0 |
Component: | interfaces | Keywords: | gp pexpect spkg |
Cc: | cremona | Merged in: | sage-5.0.beta0 |
Authors: | Jeroen Demeyer | Reviewers: | Georg S. Weber |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
On Gentoo Linux x86_64 and on sage.math running sage-4.8.alpha6 + #11073 (but without #12263):
Use a "screen" terminal.
When $DOT_SAGE (and therefore, $HOME) contains a specific number of characters in relation to the number of columns on the pseudo-terminal ($COLUMNS), failures happen. I believe this is due to temporary filenames word-wrapping in the pseudo-terminal (pty). The pty seems to inherit properties of the actual terminal, that's why there are failures with screen but not with xterm.
For example, on a 138-column terminal:
( export HOME=/mnt/usb1/scratch/jdemeyer/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; mkdir -p $HOME; ./sage -t devel/sage/sage/schemes ) [...] sage -t "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" *** Warning: new stack size = 1003360 (0.957 Mbytes). *** Warning: new stack size = 1003360 (0.957 Mbytes). ********************************************************************** File "/usr/local/src/sage-4.8.alpha4/devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py", line 526: sage: E.conductor(algorithm="gp") Exception raised: Traceback (most recent call last): File "/usr/local/src/sage-4.8.alpha4/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/local/src/sage-4.8.alpha4/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/local/src/sage-4.8.alpha4/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_12[5]>", line 1, in <module> E.conductor(algorithm="gp")###line 526: sage: E.conductor(algorithm="gp") File "/usr/local/src/sage-4.8.alpha4/local/lib/python/site-packages/sage/schemes/elliptic_curves/ell_rational_field.py", line 552, in conductor self.__conductor_gp = Integer(gp.eval('ellglobalred(ellinit(%s,0))[1]'%list(self.a_invariants()))) File "integer.pyx", line 681, in sage.rings.integer.Integer.__init__ (sage/rings/integer.c:6786) TypeError: unable to convert x (=read("/tmp/123456789_123456789_123456789_123456789_123456789_123456789_123456789_123456789/.sage/temp/arcanis/5630//interface//tmp5647") ) to an integer ********************************************************************** File "/usr/local/src/sage-4.8.alpha4/devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py", line 1430: sage: E.simon_two_descent() Exception raised: Traceback (most recent call last): File "/usr/local/src/sage-4.8.alpha4/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/usr/local/src/sage-4.8.alpha4/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/usr/local/src/sage-4.8.alpha4/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_28[4]>", line 1, in <module> E.simon_two_descent()###line 1430: sage: E.simon_two_descent() File "/usr/local/src/sage-4.8.alpha4/local/lib/python/site-packages/sage/schemes/elliptic_curves/ell_rational_field.py", line 1493, in simon_two_descent maxprob=maxprob, limbigprime=limbigprime) File "/usr/local/src/sage-4.8.alpha4/local/lib/python/site-packages/sage/schemes/elliptic_curves/gp_simon.py", line 117, in simon_two_descent ans = sage_eval(v, {'Mod': _gp_mod, 'y': K.gen(0)}) File "/usr/local/src/sage-4.8.alpha4/local/lib/python/site-packages/sage/misc/sage_eval.py", line 199, in sage_eval return eval(source, sage.all.__dict__, locals) File "<string>", line 0 ^ SyntaxError: unexpected EOF while parsing ********************************************************************** [...] The following tests failed: sage -t devel/sage/sage/schemes/elliptic_curves/lseries_ell.py # 1 doctests failed sage -t devel/sage/sage/schemes/elliptic_curves/gp_simon.py # 3 doctests failed sage -t devel/sage/sage/schemes/elliptic_curves/BSD.py # 3 doctests failed sage -t devel/sage/sage/schemes/elliptic_curves/sha_tate.py # 4 doctests failed sage -t devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 14 doctests failed sage -t devel/sage/sage/schemes/elliptic_curves/ell_number_field.py # 11 doctests failed
How to reproduce this more artificially:
apply 12221_debug.patch. This will send all gp pexpect commands less than 200 characters directly to gp, without using a temporary file. The problem is the filename length of the temporary files used in the read() command. This depends on $DOT_SAGE
, on the hostname, on process IDs.
Download test12221.sage and run (from within a "screen" session). This will execute a read() command with increasing filename lengths:
$ ./sage test12221.sage
I see
TERM = screen 18 char filename: ok 19 char filename: ok 20 char filename: ok [...] 68 char filename: ok 69 char filename: ok 70 char filename: fail [...]
apply 12221_pexpect_unset_TERM.patch and the spkg http://boxen.math.washington.edu/home/jdemeyer/spkg/pexpect-2.0.p5.spkg
Attachments (4)
Change History (45)
comment:1 Changed 9 years ago by
- Description modified (diff)
- Type changed from PLEASE CHANGE to defect
comment:2 Changed 9 years ago by
comment:3 Changed 9 years ago by
- Description modified (diff)
comment:4 Changed 9 years ago by
- Description modified (diff)
comment:5 Changed 9 years ago by
It might really be caused by #11704 then. Need to do some more tests.
comment:6 Changed 9 years ago by
I should clarify: I couldn't reproduce this, either before applying the patch from #11704 or after.
comment:7 follow-up: ↓ 8 Changed 9 years ago by
comment:8 in reply to: ↑ 7 Changed 9 years ago by
Replying to jdemeyer:
It might really be caused by #11704 + #11073 then. Need to do some more tests.
That combination doesn't cause me problems on sage.math: I have a copy of sage in /scratch/palmieri/11073/sage-4.8.a4-11073
which has the patches from #11073 and its dependencies, including #11704. Tests pass when I use the command in the ticket description. (I produced this version of Sage with the file http://sage.math.washington.edu/home/palmieri/misc/11073/sage-4.8.a4-11073.tar.)
comment:9 Changed 9 years ago by
- Description modified (diff)
comment:10 Changed 9 years ago by
Now I also have problems reproducing it. It must be some strange interaction between several tickets.
comment:11 Changed 9 years ago by
Maybe it's a race condition of some sort being triggered by #11704?
comment:12 Changed 9 years ago by
- Milestone changed from sage-4.8 to sage-duplicate/invalid/wontfix
I can't seem to reproduce this anymore...
comment:13 Changed 9 years ago by
- Milestone changed from sage-duplicate/invalid/wontfix to sage-5.0
Reproducing it again sometimes when merging #11073... must be some race condition, which makes it an even more annoying bug.
comment:14 Changed 9 years ago by
- Description modified (diff)
comment:15 Changed 9 years ago by
These errors happen much less frequently (but they still do happen) with python-2.7. At least that's my feeling.
comment:16 Changed 9 years ago by
- Summary changed from Failures in gp interface with long $HOME to Failures in gp pexpect interface with #11073
comment:17 Changed 9 years ago by
Important data point: the failures seem to happen only (or at least mostly) during the first test run. Running "make ptest" from a fresh install causes a lot of failures. Running "make ptest" a second time gives no (or much less) failures.
comment:18 Changed 9 years ago by
This observation about the "first run" could be because of a cold disk cache causing some race condition.
comment:19 Changed 9 years ago by
- Description modified (diff)
comment:20 Changed 9 years ago by
- Description modified (diff)
comment:21 Changed 9 years ago by
- Description modified (diff)
comment:22 Changed 9 years ago by
- Description modified (diff)
- Summary changed from Failures in gp pexpect interface with #11073 to Failures in gp pexpect interface with specific length of $DOT_SAGE
comment:23 Changed 9 years ago by
- Description modified (diff)
- Summary changed from Failures in gp pexpect interface with specific length of $DOT_SAGE to Failures in gp pexpect interface with specific length of $DOT_SAGE using a "screen" terminal
Changed 9 years ago by
comment:24 Changed 9 years ago by
- Description modified (diff)
comment:25 Changed 9 years ago by
- Description modified (diff)
comment:26 Changed 9 years ago by
- Description modified (diff)
comment:27 Changed 9 years ago by
- Description modified (diff)
- Status changed from new to needs_review
Very experimental fix seems to work on first sight, certainly needs more testing.
comment:28 Changed 9 years ago by
- Keywords spkg added
comment:29 Changed 9 years ago by
For the record, I still can't reproduce it on Fedora 16:
[vbraun@volker-laptop-two hg]$ hg qapplied 12221_debug.patch [vbraun@volker-laptop-two hg]$ export TERM=screen [vbraun@volker-laptop-two hg]$ sage /tmp/test12221.sage TERM = screen 18 char filename: ok 19 char filename: ok [... 117 char filename: ok
comment:30 follow-up: ↓ 32 Changed 9 years ago by
I haven't looked at this at all, but it's probably safer to set TERM to `dumb' than unsetting it. (As also evidenced by the termcap crash in #12282)
comment:31 Changed 9 years ago by
Volker: you need to actually use "screen", not just set TERM=screen.
And then the obvious: is your sage
executable the same as ./sage
and did you run ./sage -b
?
comment:32 in reply to: ↑ 30 Changed 9 years ago by
comment:33 Changed 9 years ago by
- Description modified (diff)
Changed 9 years ago by
comment:34 Changed 9 years ago by
I tried it under screen
as well without problems. I also tried running the loop longer and this works until I hit the 255 byte filename limit in ext4:
269 char filename: ok 270 char filename: ok Traceback (most recent call last): File "/tmp/test12221.py", line 11, in <module> f = open(gpfilename, "w") IOError: [Errno 36] File name too long: '/tmp/tmppzCAbE/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.gp'
comment:35 Changed 9 years ago by
I can reproduce the failing test12221.sage at home in a screen session, but it goes away with TERM=dumb.
On sage.math the test doesn't fail for me, but I do get a lot of escape sequences (1b 4d, followed by a lot of 1b 5b 43, followed by a 1b 5b 4b) when the length hits 70.
comment:36 Changed 9 years ago by
With some terminal capabilities, gp is using escape sequences to "redraw" the current line once you send a newline after exactly the right (or wrong) input line length. This redrawing of the current line includes the prompt characters, and that triggers pexpect to output the next command prematurely, getting things out of sync.
comment:37 Changed 9 years ago by
- Merged in set to sage-5.0.beta0
comment:38 follow-up: ↓ 39 Changed 9 years ago by
- Reviewers set to Georg S. Weber
- Status changed from needs_review to positive_review
Well, I'm not too fond of mixing up the change of "eval_using_file_cutoff" value from 50 to 1024 (in "12221_pexpect_unset_TERM.patch") on the one hand, and the safer handling of the "TERM" envorinment variable on the other hand --- these seem orthogonal action items to me, being worth independent tickets.
However, none of these changes makes the situation worse than before, and the cleaning up of the pexpect spkg alone is worth to have this getting in the main tree as soon as possible.
Cheers, Georg
comment:39 in reply to: ↑ 38 Changed 9 years ago by
Replying to GeorgSWeber:
Well, I'm not too fond of mixing up the change of "eval_using_file_cutoff" value from 50 to 1024 (in "12221_pexpect_unset_TERM.patch") on the one hand, and the safer handling of the "TERM" envorinment variable on the other hand --- these seem orthogonal action items to me, being worth independent tickets.
Fair enough, see #12330.
Changed 9 years ago by
comment:40 Changed 9 years ago by
- Resolution set to fixed
- Status changed from positive_review to closed
comment:41 Changed 9 years ago by
Georg, would you mind reviewing #12330 then?
I can't reproduce this, either on sage.math or an OS X box. What platform were you using?