Opened 5 years ago
Last modified 4 years ago
#23713 closed defect
Parallel map/reduce doctests fail on machine with lots of CPUs — at Initial Version
Reported by: | jdemeyer | Owned by: | |
---|---|---|---|
Priority: | blocker | Milestone: | sage-8.1 |
Component: | doctest framework | Keywords: | |
Cc: | hivert | Merged in: | |
Authors: | Jeroen Demeyer | Reviewers: | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
The number of files that the parallel map/reduce doctests requires is O(N)
where N
is the number of processors, while the number of open files allowed is typically O(1)
. So, on a machine with 192 processors:
sage -t --long --warn-long 91.0 src/sage/parallel/map_reduce.py ********************************************************************** File "src/sage/parallel/map_reduce.py", line 228, in sage.parallel.map_reduce Failed example: try: res = EX.run(timeout=60) except AbortError: print("Computation Timeout") else: print("Computation normally finished") res Exception raised: Traceback (most recent call last): File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute exec(compiled, globs) File "<doctest sage.parallel.map_reduce[33]>", line 2, in <module> res = EX.run(timeout=Integer(60)) File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1394, in run self.setup_workers(max_proc, reduce_locally) File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1070, in setup_workers for i in range(self._nprocess)] File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1513, in __init__ self._request = SimpleQueue() # Faster than Queue File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/queues.py", line 353, in __init__ self._reader, self._writer = Pipe(duplex=False) File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/__init__.py", line 107, in Pipe return Pipe(duplex) File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/connection.py", line 196, in Pipe fd1, fd2 = os.pipe() OSError: [Errno 24] Too many open files **********************************************************************
A simple solution would be to restrict the number of processors while doctesting.
Note: See
TracTickets for help on using
tickets.