Ticket #9631: trac_9631-fork_decorator.4-5.diff

File trac_9631-fork_decorator.4-5.diff, 11.9 KB (added by leif, 10 years ago)

Diff between v4 (William's) and v5 (Leif's) patch. For reference / review only.

  • sage/parallel/decorate.py

    diff -Nur sage-9631v4-clean/sage/parallel/decorate.py sage-9631v5/sage/parallel/decorate.py
    old new  
    11r"""
    2 Decorate interface for parallel computation
     2Decorate interface for parallel computation.
    33"""
    44
    55import os
     
    1313
    1414def normalize_input(a):
    1515    r"""
    16     Convert a to a pair (args, kwds) using some rules:
     16    Convert ``a`` to a pair ``(args, kwds)`` using some rules:
    1717
    1818    - if already of that form, leave that way.
    1919    - if ``a`` is a tuple make ``(a,{})``
     
    2121    - otherwise make ``((a,),{})``
    2222
    2323    INPUT:
     24
    2425     - ``a`` -- object
    2526
    2627    OUTPUT:
     28
    2729     - ``args`` -- tuple
    2830     - ``kwds`` -- dictionary
    2931
     
    5052
    5153class Parallel:
    5254    r"""
    53     Create parallel decorated function.
     55    Create a ``parallel``-decorated function.
    5456    """
    55     def __init__(self, p_iter = 'fork', ncpus=None, **kwds):
     57    def __init__(self, p_iter='fork', ncpus=None, **kwds):
    5658        """
    5759        EXAMPLES::
    5860
    5961            sage: P = sage.parallel.decorate.Parallel(); P
    6062            <sage.parallel.decorate.Parallel instance at 0x...>
    6163        """
    62         # The default p_iter is currently the reference implementation.
    63         # This may change.
     64        # The default p_iter is currently the 'fork' implementation.
     65        # This has changed.
     66
    6467        self.p_iter = None
    6568
    6669        if isinstance(p_iter, (int, long, Integer)):
     
    8487
    8588    def __call__(self, f):
    8689        r"""
    87         Create a function that wraps ``f`` and that when called with a
    88         list of inputs returns an iterator over pairs ``(x, f(x))`` in
    89         possibly random order. Here ``x`` is replaced by its
    90         normalized form (args, kwds) using normalize_inputs.
     90        Create a function that wraps ``f()`` and that when called with a
     91        list of inputs returns an iterator over pairs ``(x, f(x))``
     92        in possibly random order. Here ``x`` is replaced by its
     93        normalized form ``(args, kwds)`` using ``normalize_inputs()``.
    9194
    9295        INPUT:
     96
    9397         - ``f`` -- Python callable object or function
    9498
    9599        OUTPUT:
     100
    96101         - Decorated version of ``f``
    97102
    98103        EXAMPLES::
     
    121126                return f(*args, **kwds)
    122127        return g
    123128
    124 def parallel(p_iter = 'fork', ncpus=None, **kwds):
     129def parallel(p_iter='fork', ncpus=None, **kwds):
    125130    r"""
    126131    This is a decorator that gives a function a parallel interface,
    127132    allowing it to be called with a list of inputs, whose values will
     
    132137         The parallel subprocesses will not have access to data
    133138         created in pexpect interfaces.  This behavior with respect to
    134139         pexpect interfaces is very important to keep in mind when
    135          setting up certain computations. It's the one big limitation
     140         setting up certain computations.  It's the one big limitation
    136141         of this decorator.
    137142
    138143    INPUT:
     144
    139145     - ``p_iter`` -- parallel iterator function or string:
    140             - ``fork``            -- (default) use a new forked process for each input
    141             - ``multiprocessing`` -- use multiprocessing library
    142             - ``reference``       -- use a fake serial reference implementation
    143      - ``ncpus`` -- integer, number of cpus
    144      - ``timeout`` -- number of seconds until task is killed (only supported by 'fork')
     146            - ``'fork'``            -- (default) use a new forked subprocess for each input
     147            - ``'multiprocessing'`` -- use multiprocessing library
     148            - ``'reference'``       -- use a fake serial reference implementation
     149     - ``ncpus`` -- integer, maximal number of subprocesses to use at the same time
     150     - ``timeout`` -- number of seconds until each subprocess is killed (only supported
     151       by 'fork'; zero means not at all)
    145152
    146153    .. warning::
    147154
    148          If you use anything but 'fork' above, then a whole new
     155         If you use anything but ``'fork'`` above, then a whole new
    149156         subprocess is spawned, so none of your local state (variables,
    150157         certain functions, etc.) is available.
    151158
     159
    152160    EXAMPLES:
    153161
    154     We create a simple decoration for a simple function. The number
    155     of cpus is automatically detected::
     162    We create a simple decoration for a simple function.  The number
     163    of cpus (or cores, or hardware threads) is automatically detected::
    156164
    157165        sage: @parallel
    158166        ... def f(n): return n*n
     
    161169        sage: sorted(list(f([1,2,3])))
    162170        [(((1,), {}), 1), (((2,), {}), 4), (((3,), {}), 9)]
    163171
    164     We use exactly 2 cpus::
     172    We use exactly two cpus::
    165173
    166174        sage: @parallel(2)
    167175        ... def f(n): return n*n
    168176
    169     We create a decorator that uses 3 processes, and times out
     177
     178    We create a decorator that uses three subprocesses, and times out
    170179    individual processes after 10 seconds::
    171180
    172181        sage: @parallel(ncpus=3, timeout=10)
     
    197206
    198207###################################################################
    199208# The @fork decorator -- evaluate a function with no side effects
    200 # in memory, so the only side effects are on disk.
     209# in memory, so the only side effects (if any) are on disk.
    201210#
    202211# We have both a function and a class below, so that the decorator
    203212# can be used with or without options:
     
    211220
    212221class Fork:
    213222    """
    214     A fork decorator class.
     223    A ``fork`` decorator class.
    215224    """
    216225    def __init__(self, timeout=0, verbose=False):
    217226        """
    218227        INPUT:
    219          - ``timeout`` -- (default: 0) kills subrocess after this many
    220            seconds, or if timeout=0, do not kill the subprocess.
    221          - ``verbose`` -- (default: ``False``) whether to print
    222            anything about what the decorator does (e.g., killing
    223            subprocesses)
     228
     229         - ``timeout`` -- (default: 0) kill the subprocess after it has run this
     230           many seconds (wall time), or if ``timeout`` is zero, do not kill it.
     231         - ``verbose`` -- (default: ``False``) whether to print anything about
     232           what the decorator does (e.g., killing the subprocess)
    224233
    225234        EXAMPLES::
    226235       
     
    235244    def __call__(self, f):
    236245        """
    237246        INPUT:
     247
    238248         - ``f`` -- a function
    239249
    240250        OUTPUT:
     251
    241252         - A decorated function.
    242253
    243254        EXAMPLES::
     
    259270    """
    260271    Decorate a function so that when called it runs in a forked
    261272    subprocess.  This means that it won't have any in-memory
    262     side-effects on the parent Sage process.  The pexpect interfaces
     273    side effects on the parent Sage process.  The pexpect interfaces
    263274    are all reset.
    264275   
    265276    INPUT:
     277
    266278      - ``f`` -- a function
    267       - ``timeout`` -- (default: 0) if positive, kills subrocess after
    268         this many seconds
     279      - ``timeout`` -- (default: 0) if positive, kill the subprocess after
     280        this many seconds (wall time)
    269281      - ``verbose`` -- (default: ``False``) whether to print anything
    270         about what the decorator does (e.g., killing subprocesses)
     282        about what the decorator does (e.g., killing the subprocess)
    271283
    272284    .. warning::
    273285
    274         The forked subprocesses will not have access to data created
     286        The forked subprocess will not have access to data created
    275287        in pexpect interfaces.  This behavior with respect to pexpect
    276288        interfaces is very important to keep in mind when setting up
    277         certain computations. It's the one big limitation of this
     289        certain computations.  It's the one big limitation of this
    278290        decorator.
    279291
    280292    EXAMPLES:
    281293
    282     We create a function and run it with the fork decorator.  Note
     294    We create a function and run it with the ``fork`` decorator.  Note
    283295    that it does not have a side effect.  Despite trying to change
    284     the global variable "a" below in g, the variable a does not get
    285     changed.::
     296    the global variable ``a`` below in ``g``, the variable ``a`` does not
     297    get changed::
    286298   
    287299        sage: a = 5
    288300        sage: @fork
     
    295307        sage: a
    296308        5
    297309
    298     We use fork to make sure that the function terminates after 1
     310    We use ``fork`` to make sure that the function terminates after one
    299311    second, no matter what::
    300312   
    301313        sage: @fork(timeout=1, verbose=True)
     
    306318        Killing subprocess ... with input ((10000000,), {'m': 5}) which took too long
    307319        'NO DATA (timed out)'
    308320
    309     We illustrate that pexpect interface state is not affected by
     321    We illustrate that the state of the pexpect interface is not altered by
    310322    forked functions (they get their own new pexpect interfaces!)::
    311323   
    312324        sage: gp.eval('a = 5')
  • sage/parallel/use_fork.py

    diff -Nur sage-9631v4-clean/sage/parallel/use_fork.py sage-9631v5/sage/parallel/use_fork.py
    old new  
    11"""
    2 Parallel Iterator built using the fork system call
     2Parallel iterator built using the ``fork()`` system call.
    33"""
    44
    55################################################################################
     
    1313
    1414class p_iter_fork:
    1515    """
    16     A parallel iterator implemented using fork.
     16    A parallel iterator implemented using ``fork()``.
    1717
    1818    EXAMPLES::
    1919
     
    2828    """
    2929    def __init__(self, ncpus, timeout=0, verbose=False, reset_interfaces=True):
    3030        """
    31         Create fork-based parallel iterator.
     31        Create a ``fork()``-based parallel iterator.
    3232       
    3333        INPUT:
    3434
    3535            - ``ncpus`` -- the maximal number of simultaneous
    36               processes to spawn
    37             - ``timeout`` -- (float, default: 0) time in seconds until
     36              subprocesses to spawn
     37            - ``timeout`` -- (float, default: 0) wall time in seconds until
    3838              a subprocess is automatically killed
    3939            - ``verbose`` -- (default: False) whether to print
    4040              anything about what the iterator does (e.g., killing
    4141              subprocesses)
    4242            - ``reset_interfaces`` -- (default: True) whether to reset
    43               all expect interfaces
     43              all pexpect interfaces
    4444
    4545        EXAMPLES::
    4646
     
    6262
    6363    def __call__(self, f, inputs):
    6464        """
    65         Parallel iterator using fork.
     65        Parallel iterator using ``fork()``.
    6666
    6767        INPUT:
    6868
    69             - `f` -- a Python function that need not be picklable or anything else!
    70             - ``inputs`` -- a list of pickleable pairs (args, kwds), where
    71               args is a tuple and kwds is a dictionary.
     69            - ``f`` -- a Python function that need not be pickleable or anything else!
     70            - ``inputs`` -- a list of pickleable pairs ``(args, kwds)``, where
     71              ``args`` is a tuple and ``kwds`` is a dictionary.
    7272
    7373        OUTPUT:
    7474
     
    178178           
    179179    def _subprocess(self, f, dir, x):
    180180        """
    181         Setup and run evaluation of f(*x[0], **x[1]), storing the
    182         result in the given directory.  This method is called by each
     181        Setup and run evaluation of ``f(*x[0], **x[1])``, storing the
     182        result in the given directory ``dir``.  This method is called by each
    183183        forked subprocess.
    184184
    185185        INPUT:
    186186
    187             - `f` -- a function
     187            - ``f`` -- a function
    188188            - ``dir`` -- name of a directory
    189             - `x` -- 2-tuple, with args and kwds
     189            - ``x`` -- 2-tuple, with args and kwds
    190190
    191         EXAMPLES::
     191        EXAMPLES:
    192192
    193193        We have only this indirect test, since a direct test would terminate the Sage session.
    194194       
     
    211211            import sage.misc.misc
    212212            reload(sage.misc.misc)
    213213
    214             # The expect interfaces (and objects defined in them) are
     214            # The pexpect interfaces (and objects defined in them) are
    215215            # not valid.
    216216            if self.reset_interfaces:
    217217                sage.interfaces.quit.invalidate_all()