Opened 5 years ago
Closed 5 years ago
#22718 closed defect (fixed)
Possible file descriptor leak in sage.lfunctions.dokchitser
Reported by: | embray | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | sage-8.0 |
Component: | porting: Cygwin | Keywords: | windows cygwin |
Cc: | Merged in: | ||
Authors: | Reviewers: | ||
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
One of the last remaining (current) test failures on Cygwin I'm investigating is:
sage -t --long --warn-long 159.9 src/sage/modular/abvar/lseries.py ********************************************************************** File "src/sage/modular/abvar/lseries.py", line 215, in sage.modular.abvar.lseries.Lseries_complex.vanishes_at_1 Failed example: L(1, prec=100) Exception raised: Traceback (most recent call last): File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modular.abvar.lseries.Lseries_complex.vanishes_at_1[6]>", line 1, in <module> L(Integer(1), prec=Integer(100)) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/modular/abvar/lseries.py", line 138, in __call__ for i in range(newform.base_ring().degree())] File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/modular/modform/element.py", line 832, in lseries num_coeffs = L.num_coeffs() File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/lfunctions/dokchitser.py", line 272, in num_coeffs return Integer(self.gp().eval('cflength(%s)'%T)) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/lfunctions/dokchitser.py", line 227, in gp g.read('computel.gp') File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 195, in read self.eval(self._read_in_file_command(filename)) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1300, in eval for L in code.split('\n') if L != '']) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/gp.py", line 444, in _eval_line wait_for_prompt=wait_for_prompt) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 908, in _eval_line self._start() File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/gp.py", line 241, in _start Expect._start(self, alt_message, block_during_init) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 501, in _start self._expect.expect(self._prompt) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 321, in expect timeout, searchwindowsize, async) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 345, in expect_list return exp.expect_loop(timeout) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/expect.py", line 99, in expect_loop incoming = spawn.read_nonblocking(spawn.maxread, timeout) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 443, in read_nonblocking r, w, e = self.__select([self.child_fd], [], [], 0) File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 839, in __select return select.select(iwtd, owtd, ewtd, timeout) ValueError: filedescriptor out of range in select() ********************************************************************** 1 item had failures: 1 of 8 in sage.modular.abvar.lseries.Lseries_complex.vanishes_at_1 [53 tests, 1 failure, 25.33 s]
There isn't any one particular line in the doctests for that module that causes the issue individually. It just happens inevitably after evaluating enough L-series instances, which is using the gp pexpect interface to do some things.
This seems to be a problem on any platform; it just runs out of file descriptors faster on Cygwin, where the FD_SETSIZE
for select()
is 64 (see https://cygwin.com/ml/cygwin/2011-03/msg00651.html).
I wonder if this code could be updated to use the cypari interface instead?
Change History (9)
comment:1 Changed 5 years ago by
comment:2 in reply to: ↑ description Changed 5 years ago by
Replying to embray:
I wonder if this code could be updated to use the cypari interface instead?
Using the optional package gp2c
(which compiles GP code to C, analogous to how Cython compiles Python code to C), yes it can. I tried that in #15809 but nobody seemed to care...
Regardless of the scripts, most of the functionality is actually present in PARI itself. So we should get rid of the scripts completely instead of running them in a different way. Me and John Cremona have a plan to look at this, we just need to find an occasion to sit together and do it.
comment:3 follow-up: ↓ 4 Changed 5 years ago by
I see--so if we convert the scripts with gp2c and compile their functionality into the pari interface (also, how does this fit in with cypari2?) then the rest can be done through the pari interface.
Would I be able to help revive #15809 for now, until/unless a better approach is available?
comment:4 in reply to: ↑ 3 Changed 5 years ago by
Replying to embray:
how does this fit in with cypari2?
Yes, that's a good question. I have other things on my plate for the moment, so I'd rather not work on that right now.
comment:5 Changed 5 years ago by
That's fine--I can probably find a workaround as it relates to Cygwin in the meantime.
comment:6 Changed 5 years ago by
Simply switching terminate_async
for a non-async version doesn't help. Running this test suite still fires up too many pexect interfaces at once and doesn't garbage-collect them fast enough to shut them down. I might just update these tests to do some manual forced garbage collection.
That said, I do have one zany idea: I'm trying to look at how Dokchitser
can be modified to work with a single process. It isn't obvious how to do this, since the gp script relies on a bunch of global variables. But what if a template were created from that script, that replaces each variable with a template for an indexed variable name. E.g. gammaV
-> gammaV_{i}
where i
would be replaced with an integer for each Dokchitser
instance? So for each instance a new script would be generated from the template and run within the same gp
interpreter. When a Dokchitser
instance goes out of scope, if it's not the only one remaining (i.e. the interpreter should be kept running) the __del__
would just kill()
all the global variables for that instance to free up memory.
comment:7 Changed 5 years ago by
Went ahead and tried my idea for templating the script and it seems to work. Need to massage the code a bit more, but all the tests pass with this hack, and now only one gp
process is needed for all Dokchitser L-function calculations.
comment:8 Changed 5 years ago by
See #22746 for a workaround.
comment:9 Changed 5 years ago by
- Resolution set to fixed
- Status changed from new to closed
It looks like this is very similar to the issue I reported here (which it seems I never opened an issue for): https://groups.google.com/d/msg/sage-devel/y79rSTdBLQA/LnkO66FCAwAJ
Part of the problem is that each test spins up a
gp
interpreter, and it takes too long to shut down the interpeters between each test case, resulting in file descriptor exhaustion. I don't know why it takes so long to shut down thegp
interpreter--my previous example of this was with Maxima, and the problem was traceable in part to a bug in ECL, since fixed. It's possiblegp
has a similar bug. I think alsoSagePtyProcess.terminate_async
may be too slow on Cygwin.