Opened 5 years ago

Last modified 4 years ago

#23713 closed defect

Support SAGE_NUM_THREADS_PARALLEL in parallel map/reduce — at Version 1

Reported by: jdemeyer Owned by:
Priority: blocker Milestone: sage-8.1
Component: doctest framework Keywords:
Cc: hivert Merged in:
Authors: Jeroen Demeyer Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jdemeyer)

The number of files that the parallel map/reduce doctests requires is O(N) where N is the number of processors, while the number of open files allowed is typically fixed. So, on a machine with 192 processors:

sage -t --long --warn-long 91.0 src/sage/parallel/map_reduce.py
**********************************************************************
File "src/sage/parallel/map_reduce.py", line 228, in sage.parallel.map_reduce
Failed example:
    try:
        res = EX.run(timeout=60)
    except AbortError:
        print("Computation Timeout")
    else:
        print("Computation normally finished")
        res
Exception raised:
    Traceback (most recent call last):
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.parallel.map_reduce[33]>", line 2, in <module>
        res = EX.run(timeout=Integer(60))
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1394, in run
        self.setup_workers(max_proc, reduce_locally)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1070, in setup_workers
        for i in range(self._nprocess)]
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1513, in __init__
        self._request = SimpleQueue()  # Faster than Queue
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/queues.py", line 353, in __init__
        self._reader, self._writer = Pipe(duplex=False)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/__init__.py", line 107, in Pipe
        return Pipe(duplex)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/connection.py", line 196, in Pipe
        fd1, fd2 = os.pipe()
    OSError: [Errno 24] Too many open files
**********************************************************************

Apart from this, this doctest should not start too much processes. I suggest to set the default number of processors to the environment variable SAGE_NUM_THREADS_PARALLEL, which is precisely meant for that purpose.

Change History (1)

comment:1 Changed 5 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from Parallel map/reduce doctests fail on machine with lots of CPUs to Support SAGE_NUM_THREADS_PARALLEL in parallel map/reduce
Note: See TracTickets for help on using tickets.