Possible file descriptor leak in sage.lfunctions.dokchitser

[embray]

One of the last remaining (current) test failures on Cygwin I'm investigating is:

sage -t --long --warn-long 159.9 src/sage/modular/abvar/lseries.py
**********************************************************************
File "src/sage/modular/abvar/lseries.py", line 215, in sage.modular.abvar.lseries.Lseries_complex.vanishes_at_1
Failed example:
    L(1, prec=100)
Exception raised:
    Traceback (most recent call last):
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modular.abvar.lseries.Lseries_complex.vanishes_at_1[6]>", line 1, in <module>
        L(Integer(1), prec=Integer(100))
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/modular/abvar/lseries.py", line 138, in __call__        for i in range(newform.base_ring().degree())]
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/modular/modform/element.py", line 832, in lseries
        num_coeffs = L.num_coeffs()
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/lfunctions/dokchitser.py", line 272, in num_coeffs
        return Integer(self.gp().eval('cflength(%s)'%T))
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/lfunctions/dokchitser.py", line 227, in gp
        g.read('computel.gp')
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 195, in read
        self.eval(self._read_in_file_command(filename))
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1300, in eval
        for L in code.split('\n') if L != ''])
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/gp.py", line 444, in _eval_line
        wait_for_prompt=wait_for_prompt)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 908, in _eval_line
        self._start()
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/gp.py", line 241, in _start
        Expect._start(self, alt_message, block_during_init)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 501, in _start
        self._expect.expect(self._prompt)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 321, in expect
        timeout, searchwindowsize, async)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 345, in expect_list
        return exp.expect_loop(timeout)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/expect.py", line 99, in expect_loop
        incoming = spawn.read_nonblocking(spawn.maxread, timeout)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 443, in read_nonblocking
        r, w, e = self.__select([self.child_fd], [], [], 0)
      File "/home/embray/src/sagemath/sage-cygwin/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 839, in __select
        return select.select(iwtd, owtd, ewtd, timeout)
    ValueError: filedescriptor out of range in select()
**********************************************************************
1 item had failures:
   1 of   8 in sage.modular.abvar.lseries.Lseries_complex.vanishes_at_1
    [53 tests, 1 failure, 25.33 s]

There isn't any one particular line in the doctests for that module that causes the issue individually. It just happens inevitably after evaluating enough L-series instances, which is using the gp pexpect interface to do some things.

This seems to be a problem on any platform; it just runs out of file descriptors faster on Cygwin, where the FD_SETSIZE for select() is 64 (see ​https://cygwin.com/ml/cygwin/2011-03/msg00651.html).

I wonder if this code could be updated to use the cypari interface instead?

[embray]

It looks like this is very similar to the issue I reported here (which it seems I never opened an issue for): ​https://groups.google.com/d/msg/sage-devel/y79rSTdBLQA/LnkO66FCAwAJ

Part of the problem is that each test spins up a gp interpreter, and it takes too long to shut down the interpeters between each test case, resulting in file descriptor exhaustion. I don't know why it takes so long to shut down the gp interpreter--my previous example of this was with Maxima, and the problem was traceable in part to a bug in ECL, ​since fixed. It's possible gp has a similar bug. I think also SagePtyProcess.terminate_async may be too slow on Cygwin.

[jdemeyer]

Replying to embray:

I wonder if this code could be updated to use the cypari interface instead?

Using the optional package gp2c (which compiles GP code to C, analogous to how Cython compiles Python code to C), yes it can. I tried that in #15809 but nobody seemed to care...

Regardless of the scripts, most of the functionality is actually present in PARI itself. So we should get rid of the scripts completely instead of running them in a different way. Me and John Cremona have a plan to look at this, we just need to find an occasion to sit together and do it.

[embray]

I see--so if we convert the scripts with gp2c and compile their functionality into the pari interface (also, how does this fit in with cypari2?) then the rest can be done through the pari interface.

Would I be able to help revive #15809 for now, until/unless a better approach is available?

[jdemeyer]

Replying to embray:

how does this fit in with cypari2?

Yes, that's a good question. I have other things on my plate for the moment, so I'd rather not work on that right now.

[embray]

That's fine--I can probably find a workaround as it relates to Cygwin in the meantime.

[embray]

Simply switching terminate_async for a non-async version doesn't help. Running this test suite still fires up too many pexect interfaces at once and doesn't garbage-collect them fast enough to shut them down. I might just update these tests to do some manual forced garbage collection.

That said, I do have one zany idea: I'm trying to look at how Dokchitser can be modified to work with a single process. It isn't obvious how to do this, since the gp script relies on a bunch of global variables. But what if a template were created from that script, that replaces each variable with a template for an indexed variable name. E.g. gammaV -> gammaV_{i} where i would be replaced with an integer for each Dokchitser instance? So for each instance a new script would be generated from the template and run within the same gp interpreter. When a Dokchitser instance goes out of scope, if it's not the only one remaining (i.e. the interpreter should be kept running) the __del__ would just kill() all the global variables for that instance to free up memory.

[embray]

Went ahead and tried my idea for templating the script and it seems to work. Need to massage the code a bit more, but all the tests pass with this hack, and now only one gp process is needed for all Dokchitser L-function calculations.

[embray]

See #22746 for a workaround.

[embray]

Between #22746 and #22839 this is fixed.