Ticket #12016: 12016-sage.patch

File 12016-sage.patch, 37.6 KB (added by jdemeyer, 10 years ago)
  • doc/en/developer/doctesting.rst

    # HG changeset patch
    # User J. H. Palmieri <palmieri@math.washington.edu>
    # Date 1321130698 28800
    # Node ID 3660d82050a03ac94be8047f71a3ed91fe657176
    # Parent  df5620969fbddef0cf191407d046c2ed522d6a49
    Use SAGE_NUM_THREADS in setup.py
    Document SAGE_NUM_THREADS
    
    diff --git a/doc/en/developer/doctesting.rst b/doc/en/developer/doctesting.rst
    a b  
    99multiple threads. After compiling a source version of Sage, doctesting
    1010can be run on the whole Sage library, on all modules under a given
    1111directory, or on a specified module only. For the purposes of this
    12 chapter, suppose we have compiled Sage 4.1.1 from source and the top
     12chapter, suppose we have compiled Sage 4.8 from source and the top
    1313level Sage directory is
    1414
    1515::
    1616
    17     [mvngu@sage sage-4.1.1]$ pwd
    18     /scratch/mvngu/build/sage-4.1.1
     17    [jdemeyer@sage sage-4.8]$ pwd
     18    /scratch/jdemeyer/build/sage-4.8
    1919
    2020See the section :ref:`chapter-testing` for information on Sage's
    2121automated testing process. The general syntax for doctesting is as
    2222follows. To doctest a module in the library of a version of Sage, use
    2323this syntax::
    2424
    25     /path/to/sage-x.y.z/sage -t [-long] /path/to/sage-x.y.z/path/to/module.py[x]
     25    /path/to/sage-x.y.z/sage -t [--long] /path/to/sage-x.y.z/path/to/module.py[x]
    2626
    27 where ``-long`` is an optional argument. The version of ``sage`` used must
     27where ``--long`` is an optional argument. The version of ``sage`` used must
    2828match the version of Sage containing the module we want to doctest. A
    2929Sage module can be either a Python script (with the file extension
    3030".py") or it can be a Cython script, in which case it has the file
     
    3939top level Sage directory of our local Sage installation. Now  we can
    4040start doctesting as demonstrated in the following terminal session::
    4141
    42     [mvngu@sage sage-4.1.1]$ ./sage -t devel/sage-main/sage/games/sudoku.py
     42    [jdemeyer@sage sage-4.8]$ ./sage -t devel/sage-main/sage/games/sudoku.py
    4343    sage -t  "devel/sage-main/sage/games/sudoku.py"
    44              [6.0 s]
     44             [7.3 s]
    4545
    4646    ----------------------------------------------------------------------
    4747    All tests passed!
    48     Total time for all tests: 6.0 seconds
     48    Total time for all tests: 7.3 seconds
    4949
    5050The numbers output by the test show that testing the sudoku module
    5151takes about six seconds, while testing all specified modules took the
    5252same amount of time. In this case, we only tested one module so it is
    5353not surprising that the total testing time is approximately the same
    5454as the time required to test only that one module. Notice that the
    55 syntax is
     55syntax is ::
    5656
    57 ::
    58 
    59     [mvngu@sage sage-4.1.1]$ ./sage -t devel/sage-main/sage/games/sudoku.py
     57    [jdemeyer@sage sage-4.8]$ ./sage -t devel/sage-main/sage/games/sudoku.py
    6058    sage -t  "devel/sage-main/sage/games/sudoku.py"
    61              [5.7 s]
     59             [7.3 s]
    6260
    6361    ----------------------------------------------------------------------
    6462    All tests passed!
    65     Total time for all tests: 5.7 seconds
    66     [mvngu@sage sage-4.1.1]$ ./sage -t "devel/sage-main/sage/games/sudoku.py"
     63    Total time for all tests: 7.3 seconds
     64    [jdemeyer@sage sage-4.8]$ ./sage -t "devel/sage-main/sage/games/sudoku.py"
    6765    sage -t  "devel/sage-main/sage/games/sudoku.py"
    68              [5.4 s]
     66             [7.5 s]
    6967
    7068    ----------------------------------------------------------------------
    7169    All tests passed!
    72     Total time for all tests: 5.4 seconds
     70    Total time for all tests: 7.6 seconds
    7371
    7472but not
    7573
    7674::
    7775
    78     [mvngu@sage sage-4.1.1]$ ./sage -t sage/games/sudoku.py
     76    [jdemeyer@sage sage-4.8]$ ./sage -t sage/games/sudoku.py
    7977    ERROR: File ./sage/games/sudoku.py is missing
    8078    exit code: 1
    8179
     
    8482
    8583    ./sage/games/sudoku.py
    8684    Total time for all tests: 0.0 seconds
    87     [mvngu@sage sage-4.1.1]$ ./sage -t "sage/games/sudoku.py"
     85    [jdemeyer@sage sage-4.8]$ ./sage -t "sage/games/sudoku.py"
    8886    ERROR: File ./sage/games/sudoku.py is missing
    89     exit code: 1
    9087
    9188    ----------------------------------------------------------------------
    9289    The following tests failed:
    9390
    94     ./sage/games/sudoku.py
     91
     92            ./sage/games/sudoku.py # File not found
    9593    Total time for all tests: 0.0 seconds
    9694
    9795We can also first ``cd`` to the directory containing the module
    9896``sudoku.py`` and doctest that module as follows::
    9997
    100     [mvngu@sage sage-4.1.1]$ cd devel/sage-main/sage/games/
    101     [mvngu@sage games]$ ls
    102     all.py    __init__.py         sudoku_backtrack.pyx
    103     hexad.py  sudoku_backtrack.c  sudoku.py
    104     [mvngu@sage games]$ ../../../../sage -t sudoku.py
     98    [jdemeyer@sage sage-4.8]$ cd devel/sage-main/sage/games/
     99    [jdemeyer@sage games]$ ls
     100    __init__.py  hexad.py       sudoku.py           sudoku_backtrack.pyx
     101    all.py       quantumino.py  sudoku_backtrack.c
     102    [jdemeyer@sage games]$ ../../../../sage -t sudoku.py
    105103    sage -t  "devel/sage-main/sage/games/sudoku.py"
    106              [5.1 s]
     104             [7.1 s]
    107105
    108106    ----------------------------------------------------------------------
    109107    All tests passed!
    110     Total time for all tests: 5.1 seconds
     108    Total time for all tests: 7.1 seconds
    111109
    112110In all of the above terminal sessions, we used a local installation of
    113111Sage to test its own modules. Even if we have a system-wide Sage
     
    128126syntax is acceptable because we explicitly specify the Sage
    129127installation in the current ``SAGE_ROOT``::
    130128
    131     [mvngu@sage sage-4.1.1]$ ./sage -t devel/sage-main/sage/games/sudoku.py
    132     sage -t "devel/sage-main/sage/games/sudoku.py"
    133              [5.1 s]
     129    [jdemeyer@sage sage-4.8]$ ./sage -t devel/sage-main/sage/games/sudoku.py
     130    ./sage -t "devel/sage-main/sage/games/sudoku.py"
     131             [6.9 s]
    134132
    135133    ----------------------------------------------------------------------
    136134    All tests passed!
    137     Total time for all tests: 5.1 seconds
    138     [mvngu@sage sage-4.1.1]$ ./sage -t "devel/sage-main/sage/games/sudoku.py"
     135    Total time for all tests: 6.9 seconds
     136    [jdemeyer@sage sage-4.8]$ ./sage -t "devel/sage-main/sage/games/sudoku.py"
    139137    sage -t  "devel/sage-main/sage/games/sudoku.py"
    140             [5.0 s]
     138             [7.7 s]
    141139
    142140    ----------------------------------------------------------------------
    143141    All tests passed!
    144     Total time for all tests: 5.0 seconds
     142    Total time for all tests: 7.7 seconds
    145143
    146 With a regular user account, the following syntax is not recommended
    147 as we are using a system-wide Sage installation (if it exists)::
     144The following syntax is not recommended as we are using a system-wide
     145Sage installation (if it exists):
    148146
    149     [mvngu@sage sage-4.1.1]$ sage -t devel/sage-main/sage/games/sudoku.py
    150     Traceback (most recent call last):
    151       File "/usr/local/sage/local/bin/sage-test", line 49, in
    152         os.makedirs(TMP)
    153       File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    154         mkdir(name, mode)
    155     OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'
    156     [mvngu@sage sage-4.1.1]$ sage -t "devel/sage-main/sage/games/sudoku.py"
    157     Traceback (most recent call last):
    158       File "/usr/local/sage/local/bin/sage-test", line 49, in
    159         os.makedirs(TMP)
    160       File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    161         mkdir(name, mode)
    162     OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'
     147.. skip
    163148
    164 In this case, we received a permission error because the system-wide
    165 Sage installation attempts to write some data to a system-wide
    166 directory using our login privileges. The system-wide directory is a
    167 temporary directory under the system-wide ``SAGE_ROOT``. Most likely a
    168 system-wide Sage installation was performed by a system administrator
    169 using an account with more privileges than a regular user. As a
    170 regular user, we cannot write to directories where we do not have
    171 write permission. The following syntax is also discouraged when we
    172 login as a regular user::
     149::
    173150
    174     [mvngu@sage sage-4.1.1]$ cd
    175     [mvngu@sage ~]$ sage -t devel/sage-main/sage/games/sudoku.py
    176     Traceback (most recent call last):
    177       File "/usr/local/sage/local/bin/sage-test", line 49, in
    178         os.makedirs(TMP)
    179       File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    180         mkdir(name, mode)
    181     OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'
    182     [mvngu@sage ~]$ sage -t "devel/sage-main/sage/games/sudoku.py"
    183     Traceback (most recent call last):
    184       File "/usr/local/sage/local/bin/sage-test", line 49, in
    185         os.makedirs(TMP)
    186       File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    187         mkdir(name, mode)
    188     OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'
     151    [jdemeyer@sage sage-4.8]$ sage -t devel/sage-main/sage/games/sudoku.py
     152    sage -t  "devel/sage-main/sage/games/sudoku.py"
     153    **********************************************************************
     154    File "/home/jdemeyer/sage/sage-4.8/devel/sage-main/sage/games/sudoku.py", line 515:
     155        sage: h.solve(algorithm='backtrack').next()
     156    Exception raised:
     157        Traceback (most recent call last):
     158          File "/usr/local/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
     159            self.run_one_example(test, example, filename, compileflags)
     160          File "/usr/local/sage/local/bin/sagedoctest.py", line 38, in run_one_example
     161            OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
     162          File "/usr/local/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
     163            compileflags, 1) in test.globs
     164          File "<doctest __main__.example_13[4]>", line 1, in <module>
     165            h.solve(algorithm='backtrack').next()###line 515:
     166        sage: h.solve(algorithm='backtrack').next()
     167          File "/home/jdemeyer/.sage/tmp/sudoku.py", line 607, in solve
     168            for soln in gen:
     169          File "/home/jdemeyer/.sage/tmp/sudoku.py", line 719, in backtrack
     170            from sudoku_backtrack import backtrack_all
     171        ImportError: No module named sudoku_backtrack
     172    **********************************************************************
     173    [...more errors...]
     174    2 items had failures:
     175       4 of  15 in __main__.example_13
     176       2 of   8 in __main__.example_14
     177    ***Test Failed*** 6 failures.
     178    For whitespace errors, see the file /home/jdemeyer/.sage//tmp/.doctest_sudoku.py
     179             [21.1 s]
    189180
    190 This is exactly the same as the previous syntax because in both cases
    191 we attempted to doctest modules in a system-wide Sage installation
    192 using privileges of a regular user. If we do not have permission to
    193 read or write somewhere on a system, we cannot read or write
    194 there. As a regular user, we do not usually have privileges to read
    195 the directory ``/root`` nor do we have privileges to write to the root
    196 directory::
     181    ----------------------------------------------------------------------
     182    The following tests failed:
    197183
    198     [mvngu@sage sage-4.1.1]$ ls /root/
    199     ls: cannot open directory /root/: Permission denied
    200     [mvngu@sage sage-4.1.1]$ cd /
    201     [mvngu@sage /]$ touch demo.txt
    202     touch: cannot touch `demo.txt': Permission denied
    203184
     185            sage -t  "devel/sage-main/sage/games/sudoku.py"
     186    Total time for all tests: 21.3 seconds
     187
     188In this case, we received an error because the system-wide Sage
     189installation is a different (older) version than the one we are
     190using for Sage development.  Make sure you always test the files
     191with the correct version of Sage.
    204192
    205193Parallel testing many modules
    206194=============================
     
    215203of our system for parallel testing.
    216204
    217205Let us doctest all modules in a directory, first using a single thread
    218 and then using two threads. For this example, suppose we want to test
     206and then using four threads. For this example, suppose we want to test
    219207all the modules under ``sage/crypto/``. We can use a syntax similar to
    220208that shown above to achieve this::
    221209
    222     [mvngu@sage sage-4.1.1]$ ./sage -t devel/sage-main/sage/crypto/
    223     sage -t  "devel/sage-main/sage/crypto/lfsr.py"
    224              [2.5 s]
    225     sage -t  "devel/sage-main/sage/crypto/cryptosystem.py"
    226              [1.9 s]
     210    [jdemeyer@sage sage-4.8]$ ./sage -t devel/sage-main/sage/crypto/
     211    sage -t  "devel/sage-main/sage/crypto/block_cipher/__init__.py"
     212             [0.1 s]
    227213    sage -t  "devel/sage-main/sage/crypto/block_cipher/miniaes.py"
    228              [2.5 s]
     214             [5.5 s]
    229215    sage -t  "devel/sage-main/sage/crypto/block_cipher/all.py"
    230216             [0.1 s]
    231     sage -t  "devel/sage-main/sage/crypto/block_cipher/__init__.py"
     217    sage -t  "devel/sage-main/sage/crypto/block_cipher/sdes.py"
     218             [4.2 s]
     219    sage -t  "devel/sage-main/sage/crypto/__init__.py"
    232220             [0.1 s]
    233     sage -t  "devel/sage-main/sage/crypto/classical.py"
    234              [2.7 s]
    235     sage -t  "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
    236              [8.7 s]
    237     sage -t "devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py"
    238              [1.9 s]
     221    sage -t  "devel/sage-main/sage/crypto/stream.py"
     222             [3.7 s]
     223    sage -t  "devel/sage-main/sage/crypto/classical_cipher.py"
     224             [5.1 s]
     225    sage -t  "devel/sage-main/sage/crypto/boolean_function.pyx"
     226             [7.3 s]
     227    sage -t  "devel/sage-main/sage/crypto/lattice.py"
     228             [3.7 s]
     229    sage -t  "devel/sage-main/sage/crypto/util.py"
     230             [3.4 s]
     231    sage -t  "devel/sage-main/sage/crypto/cryptosystem.py"
     232             [3.6 s]
     233    sage -t  "devel/sage-main/sage/crypto/all.py"
     234             [0.1 s]
    239235    sage -t  "devel/sage-main/sage/crypto/mq/__init__.py"
    240236             [0.1 s]
    241237    sage -t  "devel/sage-main/sage/crypto/mq/sbox.py"
    242              [2.8 s]
     238             [4.4 s]
     239    sage -t  "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
     240             [12.8 s]
    243241    sage -t  "devel/sage-main/sage/crypto/mq/sr.py"
    244              [4.9 s]
     242             [10.6 s]
     243    sage -t  "devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py"
     244             [3.4 s]
     245    sage -t  "devel/sage-main/sage/crypto/cipher.py"
     246             [3.4 s]
     247    sage -t  "devel/sage-main/sage/crypto/classical.py"
     248             [13.8 s]
     249    sage -t  "devel/sage-main/sage/crypto/public_key/blum_goldwasser.py"
     250             [3.5 s]
     251    sage -t  "devel/sage-main/sage/crypto/public_key/__init__.py"
     252             [0.1 s]
     253    sage -t  "devel/sage-main/sage/crypto/public_key/all.py"
     254             [0.1 s]
    245255    sage -t  "devel/sage-main/sage/crypto/stream_cipher.py"
    246              [1.9 s]
    247     sage -t  "devel/sage-main/sage/crypto/all.py"
    248              [0.1 s]
    249     sage -t  "devel/sage-main/sage/crypto/stream.py"
    250              [1.9 s]
    251     sage -t  "devel/sage-main/sage/crypto/__init__.py"
    252              [0.1 s]
    253     sage -t  "devel/sage-main/sage/crypto/classical_cipher.py"
    254              [1.9 s]
    255     sage -t  "devel/sage-main/sage/crypto/cipher.py"
    256              [1.9 s]
     256             [3.4 s]
     257    sage -t  "devel/sage-main/sage/crypto/lfsr.py"
     258             [3.5 s]
    257259
    258260    ----------------------------------------------------------------------
    259261    All tests passed!
    260     Total time for all tests: 35.7 seconds
     262    Total time for all tests: 96.1 seconds
    261263
    262264Now we do the same thing, but this time we also use the optional
    263 argument ``-long``::
     265argument ``--long``::
    264266
    265     [mvngu@sage sage-4.1.1]$ ./sage -t -long devel/sage-main/sage/crypto/
    266     sage -t -long "devel/sage-main/sage/crypto/lfsr.py"
    267                   [1.9 s]
    268     sage -t -long "devel/sage-main/sage/crypto/cryptosystem.py"
    269                   [2.0 s]
    270     sage -t -long "devel/sage-main/sage/crypto/block_cipher/miniaes.py"
    271                   [2.6 s]
    272     sage -t -long "devel/sage-main/sage/crypto/block_cipher/all.py"
    273                   [0.1 s]
    274     sage -t -long "devel/sage-main/sage/crypto/block_cipher/__init__.py"
    275                   [0.1 s]
    276     sage -t -long "devel/sage-main/sage/crypto/classical.py"
    277                   [2.7 s]
    278     sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
    279                   [8.7 s]
    280     sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py"
    281                   [2.2 s]
    282     sage -t -long "devel/sage-main/sage/crypto/mq/__init__.py"
    283                   [0.1 s]
    284     sage -t -long "devel/sage-main/sage/crypto/mq/sbox.py"
    285                   [2.9 s]
    286     sage -t -long "devel/sage-main/sage/crypto/mq/sr.py"
    287                   [56.6 s]
    288     sage -t -long "devel/sage-main/sage/crypto/stream_cipher.py"
    289                   [2.5 s]
    290     sage -t -long "devel/sage-main/sage/crypto/all.py"
    291                   [0.1 s]
    292     sage -t -long "devel/sage-main/sage/crypto/stream.py"
    293                   [1.9 s]
    294     sage -t -long "devel/sage-main/sage/crypto/__init__.py"
    295                   [0.1 s]
    296     sage -t -long "devel/sage-main/sage/crypto/classical_cipher.py"
    297                   [1.9 s]
    298     sage -t -long "devel/sage-main/sage/crypto/cipher.py"
    299                   [1.9 s]
     267    [jdemeyer@sage sage-4.8]$ ./sage -t --long devel/sage-main/sage/crypto/
     268    sage -t --long "devel/sage-main/sage/crypto/block_cipher/__init__.py"
     269             [0.1 s]
     270    sage -t --long "devel/sage-main/sage/crypto/block_cipher/miniaes.py"
     271             [4.1 s]
     272    sage -t --long "devel/sage-main/sage/crypto/block_cipher/all.py"
     273             [0.1 s]
     274    sage -t --long "devel/sage-main/sage/crypto/block_cipher/sdes.py"
     275             [3.9 s]
     276    sage -t --long "devel/sage-main/sage/crypto/__init__.py"
     277             [0.0 s]
     278    sage -t --long "devel/sage-main/sage/crypto/stream.py"
     279             [3.3 s]
     280    sage -t --long "devel/sage-main/sage/crypto/classical_cipher.py"
     281             [3.9 s]
     282    sage -t --long "devel/sage-main/sage/crypto/boolean_function.pyx"
     283             [7.2 s]
     284    sage -t --long "devel/sage-main/sage/crypto/lattice.py"
     285             [3.4 s]
     286    sage -t --long "devel/sage-main/sage/crypto/util.py"
     287             [3.3 s]
     288    sage -t --long "devel/sage-main/sage/crypto/cryptosystem.py"
     289             [3.4 s]
     290    sage -t --long "devel/sage-main/sage/crypto/all.py"
     291             [0.1 s]
     292    sage -t --long "devel/sage-main/sage/crypto/mq/__init__.py"
     293             [0.1 s]
     294    sage -t --long "devel/sage-main/sage/crypto/mq/sbox.py"
     295             [3.5 s]
     296    sage -t --long "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
     297             [11.8 s]
     298    sage -t --long "devel/sage-main/sage/crypto/mq/sr.py"
     299             [96.8 s]
     300    sage -t --long "devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py"
     301             [2.9 s]
     302    sage -t --long "devel/sage-main/sage/crypto/cipher.py"
     303             [3.2 s]
     304    sage -t --long "devel/sage-main/sage/crypto/classical.py"
     305             [13.6 s]
     306    sage -t --long "devel/sage-main/sage/crypto/public_key/blum_goldwasser.py"
     307             [3.2 s]
     308    sage -t --long "devel/sage-main/sage/crypto/public_key/__init__.py"
     309             [0.1 s]
     310    sage -t --long "devel/sage-main/sage/crypto/public_key/all.py"
     311             [0.1 s]
     312    sage -t --long "devel/sage-main/sage/crypto/stream_cipher.py"
     313             [3.4 s]
     314    sage -t --long "devel/sage-main/sage/crypto/lfsr.py"
     315             [3.0 s]
    300316
    301317    ----------------------------------------------------------------------
    302318    All tests passed!
    303     Total time for all tests: 88.0 seconds
     319    Total time for all tests: 174.3 seconds
    304320
    305321Notice the time difference between the first set of tests and the
    306 second set, which uses the optional argument ``-long``. Many tests in the
     322second set, which uses the optional argument ``--long``. Many tests in the
    307323Sage library are flagged with ``# long time`` because these are known to
    308 take a long time to run through. Without using the optional ``-long``
    309 argument, the module ``sage/crypto/mq/sr.py`` took about five
    310 seconds. With this optional argument, it required 57 seconds to run
     324take a long time to run through. Without using the optional ``--long``
     325argument, the module ``sage/crypto/mq/sr.py`` took about ten
     326seconds. With this optional argument, it required 97 seconds to run
    311327through all tests in that module. Here is a snippet of a function in
    312328the module ``sage/crypto/mq/sr.py`` with a doctest that has been flagged
    313329as taking a long time::
     
    315331    def test_consistency(max_n=2, **kwargs):
    316332        r"""
    317333        Test all combinations of ``r``, ``c``, ``e`` and ``n`` in ``(1,
    318         2)`` for consistency of random encryptions and their polynomial
    319         systems. `\GF{2}` and `\GF{2^e}` systems are tested. This test
    320         takes
     334        2)`` for consistency of random encryptions and their polynomial
     335        systems. `\GF{2}` and `\GF{2^e}` systems are tested. This test takes
    321336        a while.
    322337
    323338        INPUT:
    324339
    325         - ``max_n`` - maximal number of rounds to consider (default: 2)
    326         - ``kwargs`` - are passed to the SR constructor
     340        - ``max_n`` -- maximal number of rounds to consider (default: 2)
     341        - ``kwargs`` -- are passed to the SR constructor
    327342
    328         TESTS::
     343        TESTS:
     344
     345        The following test called with ``max_n`` = 2 requires a LOT of RAM
     346        (much more than 2GB).  Since this might cause the doctest to fail
     347        on machines with "only" 2GB of RAM, we test ``max_n`` = 1, which
     348        has a more reasonable memory usage. ::
    329349
    330350            sage: from sage.crypto.mq.sr import test_consistency
    331             sage: test_consistency(1) # long time -- calling w/ max_n = 2 requires a LOT of RAM (>> 2GB, evidently).  Calling w/ max_n = 1 is far more manageable.
     351            sage: test_consistency(1)  # long time (80s on sage.math, 2011)
    332352            True
    333 
    334         The above doctest used to fail on a machine with "only" 2GB RAM.
    335         Using ``max_n = 1`` appears to be a more reasonable memory usage.
    336353        """
    337354
    338 Now we doctest the same directory in parallel using two threads::
     355Now we doctest the same directory in parallel using 4 threads::
    339356
    340     [mvngu@sage sage-4.1.1]$ ./sage -tp 2 devel/sage-main/sage/crypto/
     357    [jdemeyer@sage sage-4.8]$ ./sage -tp 4 devel/sage-main/sage/crypto/
    341358    Global iterations: 1
    342359    File iterations: 1
    343360    Using cached timings to run longest doctests first.
    344     Doctesting 17 files doing 2 jobs in parallel
    345     sage -t  devel/sage-main/sage/crypto/lfsr.py
    346              [2.7 s]
    347     sage -t  devel/sage-main/sage/crypto/cryptosystem.py
    348              [2.0 s]
    349     sage -t  devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
    350              [9.4 s]
    351     sage -t  devel/sage-main/sage/crypto/mq/sr.py
    352              [5.2 s]
    353     sage -t  devel/sage-main/sage/crypto/classical.py
    354              [2.8 s]
    355     sage -t  devel/sage-main/sage/crypto/mq/sbox.py
    356              [3.2 s]
    357     sage -t  devel/sage-main/sage/crypto/block_cipher/miniaes.py
    358              [2.6 s]
    359     sage -t  devel/sage-main/sage/crypto/stream_cipher.py
    360              [2.0 s]
    361     sage -t  devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py
    362              [2.0 s]
    363     sage -t  devel/sage-main/sage/crypto/classical_cipher.py
    364              [2.1 s]
    365     sage -t  devel/sage-main/sage/crypto/cipher.py
    366              [2.1 s]
     361    Doctesting 24 files doing 4 jobs in parallel
    367362    sage -t  devel/sage-main/sage/crypto/__init__.py
    368363             [0.1 s]
     364    sage -t  devel/sage-main/sage/crypto/lattice.py
     365             [3.3 s]
     366    sage -t  devel/sage-main/sage/crypto/stream.py
     367             [3.5 s]
     368    sage -t  devel/sage-main/sage/crypto/classical_cipher.py
     369             [4.0 s]
     370    sage -t  devel/sage-main/sage/crypto/all.py
     371             [0.1 s]
     372    sage -t  devel/sage-main/sage/crypto/util.py
     373             [3.4 s]
     374    sage -t  devel/sage-main/sage/crypto/cryptosystem.py
     375             [3.4 s]
     376    sage -t  devel/sage-main/sage/crypto/boolean_function.pyx
     377             [6.9 s]
     378    sage -t  devel/sage-main/sage/crypto/cipher.py
     379             [3.3 s]
    369380    sage -t  devel/sage-main/sage/crypto/block_cipher/__init__.py
    370381             [0.1 s]
     382    sage -t  devel/sage-main/sage/crypto/lfsr.py
     383             [3.3 s]
     384    sage -t  devel/sage-main/sage/crypto/stream_cipher.py
     385             [3.4 s]
     386    sage -t  devel/sage-main/sage/crypto/block_cipher/all.py
     387             [0.1 s]
    371388    sage -t  devel/sage-main/sage/crypto/mq/__init__.py
    372389             [0.1 s]
    373     sage -t  devel/sage-main/sage/crypto/block_cipher/all.py
     390    sage -t  devel/sage-main/sage/crypto/block_cipher/miniaes.py
     391             [4.0 s]
     392    sage -t  devel/sage-main/sage/crypto/block_cipher/sdes.py
     393             [3.6 s]
     394    sage -t  devel/sage-main/sage/crypto/mq/sbox.py
     395             [4.0 s]
     396    sage -t  devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py
     397             [3.2 s]
     398    sage -t  devel/sage-main/sage/crypto/public_key/blum_goldwasser.py
     399             [3.4 s]
     400    sage -t  devel/sage-main/sage/crypto/public_key/__init__.py
    374401             [0.1 s]
    375     sage -t  devel/sage-main/sage/crypto/stream.py
    376              [2.0 s]
    377     sage -t  devel/sage-main/sage/crypto/all.py
     402    sage -t  devel/sage-main/sage/crypto/classical.py
     403             [14.3 s]
     404    sage -t  devel/sage-main/sage/crypto/public_key/all.py
    378405             [0.1 s]
     406    sage -t  devel/sage-main/sage/crypto/mq/sr.py
     407             [9.3 s]
     408    sage -t  devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
     409             [12.0 s]
    379410
    380411    ----------------------------------------------------------------------
    381412    All tests passed!
    382413    Timings have been updated.
    383     Total time for all tests: 19.3 seconds
    384 
    385     [mvngu@sage sage-4.1.1]$ ./sage -tp 2 -long devel/sage-main/sage/crypto/
     414    Total time for all tests: 23.7 seconds
     415    [jdemeyer@sage sage-4.8]$ ./sage -tp 4 --long devel/sage-main/sage/crypto/
    386416    Global iterations: 1
    387417    File iterations: 1
    388     No long cached timings exist; will create upon successful finish.
    389     Doctesting 17 files doing 2 jobs in parallel
    390     sage -t -long devel/sage-main/sage/crypto/cryptosystem.py
    391              [2.7 s]
    392     sage -t -long devel/sage-main/sage/crypto/lfsr.py
    393              [2.7 s]
    394     sage -t -long devel/sage-main/sage/crypto/stream_cipher.py
    395              [2.2 s]
    396     sage -t -long devel/sage-main/sage/crypto/all.py
     418    Using long cached timings to run longest doctests first.
     419    Doctesting 24 files doing 4 jobs in parallel
     420    sage -t --long devel/sage-main/sage/crypto/__init__.py
    397421             [0.1 s]
    398     sage -t -long devel/sage-main/sage/crypto/classical.py
    399              [3.0 s]
    400     sage -t -long devel/sage-main/sage/crypto/__init__.py
     422    sage -t --long devel/sage-main/sage/crypto/stream.py
     423             [3.2 s]
     424    sage -t --long devel/sage-main/sage/crypto/lattice.py
     425             [3.3 s]
     426    sage -t --long devel/sage-main/sage/crypto/classical_cipher.py
     427             [4.1 s]
     428    sage -t --long devel/sage-main/sage/crypto/all.py
    401429             [0.1 s]
    402     sage -t -long devel/sage-main/sage/crypto/stream.py
    403              [2.1 s]
    404     sage -t -long devel/sage-main/sage/crypto/classical_cipher.py
    405              [2.1 s]
    406     sage -t -long devel/sage-main/sage/crypto/cipher.py
    407              [2.1 s]
    408     sage -t -long devel/sage-main/sage/crypto/block_cipher/all.py
     430    sage -t --long devel/sage-main/sage/crypto/util.py
     431             [3.1 s]
     432    sage -t --long devel/sage-main/sage/crypto/cryptosystem.py
     433             [3.3 s]
     434    sage -t --long devel/sage-main/sage/crypto/boolean_function.pyx
     435             [7.0 s]
     436    sage -t --long devel/sage-main/sage/crypto/cipher.py
     437             [3.2 s]
     438    sage -t --long devel/sage-main/sage/crypto/block_cipher/__init__.py
    409439             [0.1 s]
    410     sage -t -long devel/sage-main/sage/crypto/block_cipher/__init__.py
     440    sage -t --long devel/sage-main/sage/crypto/stream_cipher.py
     441             [3.2 s]
     442    sage -t --long devel/sage-main/sage/crypto/block_cipher/all.py
    411443             [0.1 s]
    412     sage -t -long devel/sage-main/sage/crypto/block_cipher/miniaes.py
    413              [2.8 s]
    414     sage -t -long devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py
    415              [2.0 s]
    416     sage -t -long devel/sage-main/sage/crypto/mq/__init__.py
     444    sage -t --long devel/sage-main/sage/crypto/lfsr.py
     445             [3.4 s]
     446    sage -t --long devel/sage-main/sage/crypto/mq/__init__.py
    417447             [0.1 s]
    418     sage -t -long devel/sage-main/sage/crypto/mq/sbox.py
     448    sage -t --long devel/sage-main/sage/crypto/block_cipher/miniaes.py
     449             [4.2 s]
     450    sage -t --long devel/sage-main/sage/crypto/block_cipher/sdes.py
     451             [4.0 s]
     452    sage -t --long devel/sage-main/sage/crypto/mq/sbox.py
     453             [3.8 s]
     454    sage -t --long devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py
    419455             [3.1 s]
    420     sage -t -long devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
    421              [9.1 s]
    422     sage -t -long devel/sage-main/sage/crypto/mq/sr.py
    423              [56.0 s]
     456    sage -t --long devel/sage-main/sage/crypto/classical.py
     457             [13.8 s]
     458    sage -t --long devel/sage-main/sage/crypto/public_key/__init__.py
     459             [0.0 s]
     460    sage -t --long devel/sage-main/sage/crypto/public_key/all.py
     461             [0.0 s]
     462    sage -t --long devel/sage-main/sage/crypto/public_key/blum_goldwasser.py
     463             [3.1 s]
     464    sage -t --long devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
     465             [11.3 s]
     466    sage -t --long devel/sage-main/sage/crypto/mq/sr.py
     467             [95.4 s]
    424468
    425469    ----------------------------------------------------------------------
    426470    All tests passed!
    427     Timings have been updated.
    428     Total time for all tests: 71.8 seconds
     471    Total time for all tests: 109.4 seconds
    429472
    430473As the number of threads increases, the total testing time
    431474decreases. To minimize confusion, it is also a good idea to explicitly
     
    444487``SAGE_ROOT/devel/sage-main/``. We can use the syntax described above
    445488to doctest the main library using multiple threads. When doing release
    446489management or patching the main Sage library, a release manager would
    447 parallel test the library using ten or more threads::
     490parallel test the library using 10 threads with the following command::
    448491
    449     [mvngu@sage sage-4.1.1]$ ./sage -tp 10 -long devel/sage-main/
     492    [jdemeyer@sage sage-4.8]$ ./sage -tp 10 -long devel/sage-main/
    450493
    451 Another way is to edit the file ``Makefile`` in the top level Sage
    452 directory so that the variable ``NUM_THREADS`` is set to ``10``::
     494Another way is run ``make ptestlong``, which builds Sage (if necessary),
     495builds the Sage documentation (if necessary), and then runs parallel
     496doctests.  This determines the number of threads by reading the
     497environment variable :envvar:`MAKE`: if it is set to ``make -j12``, then
     498use 12 threads.  If :envvar:`MAKE` is not set, then by default it uses
     499the number of CPU cores (as determined by the Python function
     500``multiprocessing.cpu_count()``) with a minimum of 2 and a maximum of 8.
    453501
    454     # How many threads should be used when doing parallel testing (and
    455     # sometime in the future, parallel building)?
    456     NUM_THREADS=10
     502In any case, this will test the Sage library with multiple threads::
    457503
    458 After saving all changes to ``Makefile``, we can parallel test with the
    459 ``-long`` option using ten threads::
    460 
    461     [mvngu@sage sage-4.1.1]$ make ptestlong
     504    [jdemeyer@sage sage-4.8]$ make ptestlong
    462505
    463506Any of the following commands would also doctest the Sage library or
    464507one of its clones::
     
    497540
    498541* ``make ptest`` --- Similar to the commands ``make test`` and ``make
    499542  check``. However, doctesting is run with the number of threads as
    500   specified by the variable ``NUM_THREADS``. See the file
    501   ``SAGE_ROOT/Makefile`` for further details.
     543  described above for ``make ptestlong``.
    502544
    503545* ``make ptestlong`` --- Similar to the command ``make ptest``, but
    504546  using the optional argument ``-long`` for doctesting.
  • doc/en/installation/source.rst

    diff --git a/doc/en/installation/source.rst b/doc/en/installation/source.rst
    a b  
    572572     when building Sage with ``MAKE='make -jNUM'`` with NUM greater
    573573     than one.
    574574
     575- :envvar:`SAGE_NUM_THREADS` - if this is set to a number, then when
     576  building the documentation, parallel doctesting, or running ``sage
     577  -b``, use this many threads.  If this is not set, then determine the
     578  number of threads using the value of the :envvar:`MAKE` (see above)
     579  or :envvar:`MAKEFLAGS` environment variables.  If none of these
     580  specifies a number of jobs, use 1 thread (except for parallel
     581  testing: there we use a default of the number of CPU cores, with a
     582  maximum of 8 and a minimum of 2).
     583
    575584- :envvar:`SAGE_PARALLEL_SPKG_BUILD` - if this is set to "no", then
    576585  build spkgs serially rather than in parallel.  If this is "no", then
    577586  each spkg may still take advantage of the setting of :envvar:`MAKE`
  • setup.py

    diff --git a/setup.py b/setup.py
    a b  
    195195###### Parallel Cython execution
    196196#############################################
    197197
    198 def execute_list_of_commands_in_serial(command_list):
    199     """
    200     INPUT:
    201         command_list -- a list of commands, each given as a pair
    202            of the form [command, argument].
    203        
    204     OUTPUT:
    205         the given list of commands are all executed in serial
    206     """
    207     process_command_results(f(v) for f,v in command_list)
    208 
    209198def run_command(cmd):
    210199    """
    211200    INPUT:
     
    229218
    230219def execute_list_of_commands_in_parallel(command_list, nthreads):
    231220    """
     221    Execute the given list of commands, possibly in parallel, using
     222    ``nthreads`` threads.  Terminates ``setup.py`` with an exit code
     223    of 1 if an error occurs in any subcommand.
     224
    232225    INPUT:
    233         command_list -- a list of pairs, consisting of a
    234              function to call and its argument
    235         nthreads -- integer; number of threads to use
    236        
    237     OUTPUT:
    238         Executes the given list of commands, possibly in parallel,
    239         using nthreads threads.  Terminates setup.py with an exit code of 1
    240         if an error occurs in any subcommand.
     226
     227    - ``command_list`` -- a list of commands, each given as a pair of
     228       the form ``[function, argument]`` of a function to call and its
     229       argument
     230
     231    - ``nthreads`` -- integer; number of threads to use
    241232
    242233    WARNING: commands are run roughly in order, but of course successive
    243234    commands may be run at the same time.
    244235    """
    245     print "Execute %s commands (using %s threads)"%(len(command_list), min(len(command_list),nthreads))
    246236    from multiprocessing import Pool
    247237    import twisted.persisted.styles #doing this import will allow instancemethods to be pickable
    248238    p = Pool(nthreads)
     
    259249    if error:
    260250        sys.exit(1)
    261251
    262 def number_of_threads():
    263     """
    264     Try to determine the number of threads one can run at once on this
    265     system (e.g., the number of cores).  If successful return that
    266     number.  Otherwise return 0 to indicate failure.
    267 
    268     OUTPUT:
    269         int
    270     """
    271     try:
    272         from multiprocessing import cpu_count
    273         n = cpu_count()
    274         if n>0:
    275             return n
    276     except NotImplementedError: # unsupported OS       
    277         pass
    278 
    279     try:  # solaris fix
    280         n = int(os.popen2("sysctl -n hw.ncpu")[1].read().strip())
    281         if n>0:
    282             return n
    283     except ValueError:
    284         pass
    285 
    286     return 1
    287    
    288252def execute_list_of_commands(command_list):
    289253    """
    290254    INPUT:
    291         command_list -- a list of strings or pairs
     255
     256    - ``command_list`` -- a list of strings or pairs
     257
    292258    OUTPUT:
    293         For each entry in command_list, we attempt to run the command.
    294         If it is a string, we call os.system. If it is a pair [f, v],
    295         we call f(v). On machines with more than 1 cpu the commands
    296         are run in parallel.
     259
     260    For each entry in command_list, we attempt to run the command.
     261    If it is a string, we call ``os.system()``. If it is a pair [f, v],
     262    we call f(v).
     263   
     264    If the environment variable :envvar:`SAGE_NUM_THREADS` is set, use
     265    that many threads.
    297266    """
    298267    t = time.time()
    299     cpu_count = number_of_threads()  # try hard to determine the actual cpu count
    300     assert(cpu_count>=1)
    301     nthreads = 0  # number of threads to use; zero means don't know
    302 
    303     if os.environ.has_key('MAKE'):  # user-supplied number of threads takes precedence
    304         MAKE = os.environ['MAKE']
    305         # from the manpage: If there is more than one -j option, the last one is effective.
    306         pos = MAKE.rfind(' -j')
    307         if pos>=0:
    308             try:
    309                 if MAKE[pos+3] == '=':   # make -j=N is the same as make -jN
    310                     pos += 1
    311                 nthreads = int(MAKE[pos+3:].split()[0])
    312             except IndexError, ValueError:
    313                 # make -j without number means unlimited threads
    314                 nthreads = 2*cpu_count
    315 
    316     if nthreads==0:
    317         nthreads = cpu_count
    318 
    319     if nthreads > 2*cpu_count:  # sanity check
    320         print "Warning: The number of threads ("+str(nthreads)+") seems impossibly large."
    321         nthreads = min(nthreads, cpu_count)
    322         print "I reduced it to "+str(nthreads)+"."
     268    # Determine the number of threads from the environment variable
     269    # SAGE_NUM_THREADS, which is set automatically by sage-env
     270    try:
     271        nthreads = int(os.environ['SAGE_NUM_THREADS'])
     272    except KeyError:
     273        nthreads = 1
    323274
    324275    # normalize the command_list to handle strings correctly
    325276    command_list = [ [run_command, x] if isinstance(x, str) else x for x in command_list ]
    326277
    327     if nthreads > 1:
    328         execute_list_of_commands_in_parallel(command_list, nthreads)
    329     else:
    330         execute_list_of_commands_in_serial(command_list)
    331     print "Time to execute %s commands: %s seconds"%(len(command_list), time.time() - t)
     278    # No need for more threads than there are commands
     279    nthreads = min(len(command_list), nthreads)
     280
     281    def plural(n,noun):
     282        if n == 1:
     283            return "1 %s"%noun
     284        return "%i %ss"%(n,noun)
     285
     286    print "Executing %s (using %s)"%(plural(len(command_list),"command"), plural(nthreads,"thread"))
     287    execute_list_of_commands_in_parallel(command_list, nthreads)
     288    print "Time to execute %s: %s seconds"%(plural(len(command_list),"command"), time.time() - t)
    332289
    333290
    334291########################################################################