Opened 5 years ago

Last modified 4 years ago

#23713 closed defect

Parallel map/reduce doctests fail on machine with lots of CPUs — at Initial Version

Reported by: jdemeyer Owned by:
Priority: blocker Milestone: sage-8.1
Component: doctest framework Keywords:
Cc: hivert Merged in:
Authors: Jeroen Demeyer Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description

The number of files that the parallel map/reduce doctests requires is O(N) where N is the number of processors, while the number of open files allowed is typically O(1). So, on a machine with 192 processors:

sage -t --long --warn-long 91.0 src/sage/parallel/map_reduce.py
**********************************************************************
File "src/sage/parallel/map_reduce.py", line 228, in sage.parallel.map_reduce
Failed example:
    try:
        res = EX.run(timeout=60)
    except AbortError:
        print("Computation Timeout")
    else:
        print("Computation normally finished")
        res
Exception raised:
    Traceback (most recent call last):
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.parallel.map_reduce[33]>", line 2, in <module>
        res = EX.run(timeout=Integer(60))
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1394, in run
        self.setup_workers(max_proc, reduce_locally)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1070, in setup_workers
        for i in range(self._nprocess)]
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1513, in __init__
        self._request = SimpleQueue()  # Faster than Queue
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/queues.py", line 353, in __init__
        self._reader, self._writer = Pipe(duplex=False)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/__init__.py", line 107, in Pipe
        return Pipe(duplex)
      File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/connection.py", line 196, in Pipe
        fd1, fd2 = os.pipe()
    OSError: [Errno 24] Too many open files
**********************************************************************

A simple solution would be to restrict the number of processors while doctesting.

Change History (0)

Note: See TracTickets for help on using tickets.