Changes between Version 1 and Version 4 of Ticket #23713


Ignore:
Timestamp:
10/05/17 10:17:46 (5 years ago)
Author:
jdemeyer
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Ticket #23713

    • Property Status changed from new to needs_work
    • Property Component changed from doctest coverage to doctest framework
    • Property Summary changed from Support SAGE_NUM_THREADS_PARALLEL in parallel map/reduce to Support SAGE_NUM_THREADS for parallellism
    • Property Priority changed from critical to blocker
    • Property Branch changed from to u/jdemeyer/support_sage_num_threads_parallel_in_parallel_map_reduce
    • Property Commit changed from to 282c95423512fe2c14481ab83ab29cfa4c64061f
  • Ticket #23713 – Description

    v1 v4  
    1 The number of files that the parallel map/reduce doctests requires is `O(N)` where `N` is the number of processors, while the number of open files allowed is typically fixed. So, on a machine with 192 processors:
    2 {{{
    3 sage -t --long --warn-long 91.0 src/sage/parallel/map_reduce.py
    4 **********************************************************************
    5 File "src/sage/parallel/map_reduce.py", line 228, in sage.parallel.map_reduce
    6 Failed example:
    7     try:
    8         res = EX.run(timeout=60)
    9     except AbortError:
    10         print("Computation Timeout")
    11     else:
    12         print("Computation normally finished")
    13         res
    14 Exception raised:
    15     Traceback (most recent call last):
    16       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
    17         self.compile_and_execute(example, compiler, test.globs)
    18       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
    19         exec(compiled, globs)
    20       File "<doctest sage.parallel.map_reduce[33]>", line 2, in <module>
    21         res = EX.run(timeout=Integer(60))
    22       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1394, in run
    23         self.setup_workers(max_proc, reduce_locally)
    24       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1070, in setup_workers
    25         for i in range(self._nprocess)]
    26       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/site-packages/sage/parallel/map_reduce.py", line 1513, in __init__
    27         self._request = SimpleQueue()  # Faster than Queue
    28       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/queues.py", line 353, in __init__
    29         self._reader, self._writer = Pipe(duplex=False)
    30       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/__init__.py", line 107, in Pipe
    31         return Pipe(duplex)
    32       File "/home/jdemeyer/sage-patchbot/local/lib/python2.7/multiprocessing/connection.py", line 196, in Pipe
    33         fd1, fd2 = os.pipe()
    34     OSError: [Errno 24] Too many open files
    35 **********************************************************************
    36 }}}
     1There are several independent implementations in Sage to determine the number of threads or processors for a parallel computation. Instead, these should be one way to do that and it should support the environment variable `SAGE_NUM_THREADS`, which is precisely meant for that.
    372
    38 Apart from this, this doctest should not start too much processes. I suggest to set the ''default'' number of processors to the environment variable `SAGE_NUM_THREADS_PARALLEL`, which is precisely meant for that purpose.
     3On top of that, doctests should never use too many threads. So we set `SAGE_NUM_THREADS=2` while doctesting, analogous to #23892.
     4
     5This fixes doctests in:
     6- parallel map/reduce
     7- `cbc`
     8- `cryptominisat`