Ticket #3600 (new defect)

Opened 5 months ago

Last modified 5 days ago

[with patch, needs review] implement process pool for dsage workers

Reported by: yi Assigned to: yi
Priority: major Milestone: sage-3.2.2
Component: dsage Keywords: dsage, editor_gfurnish
Cc:

Description

This patch gets rid of the old dsage worker behavior. Specifically, it does the following things:

1. Workers no longer poll the server for new jobs. 2. Workers no longer poll sage for when the job finishes. 3. Doctests run much more reliably now, and in much less time (no need for # long time now) 4. The worker, as well as the server use twistd now, this make things like running them under a profile

trivial.

It's a rather big patch so right now I'd like people to apply it and run the doc/unittests. To run the unittests, do 

sage: !trial sage.dsage

Run doctests as normal.

Attachments

dsage_process_pool.patch (168.1 kB) - added by yi on 07/07/2008 06:29:25 PM.
dsage_process_pool_patches.tar.gz (37.4 kB) - added by yi on 07/07/2008 11:23:24 PM.
all the patches split up and whitespace only patches are in the whitespace sub directory
dsage_process_pool_patches.tar.2.gz (37.4 kB) - added by yi on 07/07/2008 11:23:34 PM.
all the patches split up and whitespace only patches are in the whitespace sub directory
ppool_unit_tests.patch (13.7 kB) - added by yi on 07/10/2008 03:48:57 PM.
New version which fixes the reactor left in an unclean state errors.
dsage_ppool.hg (22.0 kB) - added by yi on 07/13/2008 11:40:42 AM.
bundle to be applied against 3.0.4
dsage_ppool.2.hg (22.0 kB) - added by yi on 07/13/2008 11:40:45 AM.
bundle to be applied against 3.0.4

Change History

07/07/2008 06:29:25 PM changed by yi

  • attachment dsage_process_pool.patch added.

07/07/2008 11:23:24 PM changed by yi

  • attachment dsage_process_pool_patches.tar.gz added.

all the patches split up and whitespace only patches are in the whitespace sub directory

07/07/2008 11:23:34 PM changed by yi

  • attachment dsage_process_pool_patches.tar.2.gz added.

all the patches split up and whitespace only patches are in the whitespace sub directory

07/07/2008 11:33:32 PM changed by yi

Tarball here: http://sage.math.washington.edu/home/yqiang/Patches/dsage_process_pool_patches.tar.gz

07/08/2008 11:01:58 AM changed by gfurnish

  • keywords changed from dsage to dsage, editor_gfurnish.

This should be a very high priority to get reviewed -- it gives orders of magnitude performance improvements.

07/10/2008 05:23:11 AM changed by gfurnish

  • summary changed from [with patch, needs review] implement process pool for dsage workers to [with patch, needs work] implement process pool for dsage workers.

On Sage.math against 3.0.4 

sage -t  devel/sage-main/sage/dsage/dist_functions/dist_factor.pysh: line 1:  4406 Segmentation fault      /home/gfurnish/sage-3.0.4-x86_64-Linux/local/bin/python /home/gfurnish/sage-3.0.4-x86_64-Linux/tmp/.doctest_dist_factor.py >/tmp/tmpofG7O8 2>/tmp/tmpIYRdmR

A mysterious error (perphaps a memory error?) occurred, which may have crashed doctest.

sage.dsage.database.tests.test_client
  ClientTestCase
    testGetPublicKey ...                                                   [OK]
    testGetUsername ...                                                    [OK]
    testInitClient ...                                                     [OK]
    testSetConnected ...                                                   [OK]
    testSetEnabled ...                                                     [OK]
                                                 [ERROR]
sage.dsage.database.tests.test_job
  JobTestCase
    testcreate_job ...                                                     [OK]
    testjobCreationTime ...                                                [OK]
    testjobFailures ...                                                    [OK]
    testjobFile ...                                                        [OK]
    testjobFinishTime ...                                                  [OK]
    testjobKilled ...                                                      [OK]
    testjobResult ...                                                      [OK]
    testjobStatus ...                                                      [OK]
    testjobUpdateTime ...                                                  [OK]
    testjob_id ...                                                         [OK]
                                                     [ERROR]
                                                     [ERROR]
                                                     [ERROR]
                                                     [ERROR]
                                                     [ERROR]

===============================================================================
[ERROR]: sage.dsage.database.tests.test_clientdb

Traceback (most recent call last):
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/trial/runner.py", line 546, in loadPackage
    module = modinfo.load()
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 380, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 618, in moduleLoader
    return self._moduleLoader(modname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 426, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 377, in _importAndCheckStack
    return __import__(importName)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/tests/test_clientdb.py", line 25, in <module>
    from sage.dsage.database.db_config import init_db
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/db_config.py", line 5, in <module>
    from sqlalchemy import *
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/__init__.py", line 8, in <module>
    from sqlalchemy.types import \
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/types.py", line 28, in <module>
    from sqlalchemy.util import pickle, Decimal as _python_Decimal
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/util.py", line 21, in <module>
    set_types = set, sets.Set
exceptions.AttributeError: 'module' object has no attribute 'Set'
===============================================================================
[ERROR]: sage.dsage.database.tests.test_jobdb

Traceback (most recent call last):
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/trial/runner.py", line 546, in loadPackage
    module = modinfo.load()
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 380, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 618, in moduleLoader
    return self._moduleLoader(modname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 426, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 377, in _importAndCheckStack
    return __import__(importName)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/tests/test_jobdb.py", line 24, in <module>
    from sage.dsage.database.jobdb import JobDatabaseSQLite, JobDatabaseSA
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/jobdb.py", line 21, in <module>
    from sqlalchemy import *
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/__init__.py", line 8, in <module>
    from sqlalchemy.types import \
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/types.py", line 28, in <module>
    from sqlalchemy.util import pickle, Decimal as _python_Decimal
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/util.py", line 21, in <module>
    set_types = set, sets.Set
exceptions.AttributeError: 'module' object has no attribute 'Set'
===============================================================================
[ERROR]: sage.dsage.database.tests.test_workerdb

Traceback (most recent call last):
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/trial/runner.py", line 546, in loadPackage
    module = modinfo.load()
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 380, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 618, in moduleLoader
    return self._moduleLoader(modname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 426, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 377, in _importAndCheckStack
    return __import__(importName)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/tests/test_workerdb.py", line 7, in <module>
    from sage.dsage.database.db_config import init_db_sa as init_db
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/db_config.py", line 5, in <module>
    from sqlalchemy import *
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/__init__.py", line 8, in <module>
    from sqlalchemy.types import \
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/types.py", line 28, in <module>
    from sqlalchemy.util import pickle, Decimal as _python_Decimal
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/util.py", line 21, in <module>
    set_types = set, sets.Set
exceptions.AttributeError: 'module' object has no attribute 'Set'
===============================================================================
[ERROR]: sage.dsage.server.tests.test_server

Traceback (most recent call last):
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/trial/runner.py", line 546, in loadPackage
    module = modinfo.load()
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 380, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 618, in moduleLoader
    return self._moduleLoader(modname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 426, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 377, in _importAndCheckStack
    return __import__(importName)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/server/tests/test_server.py", line 25, in <module>
    from sage.dsage.database.jobdb import JobDatabaseSA as JobDatabase
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/jobdb.py", line 21, in <module>
    from sqlalchemy import *
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/__init__.py", line 8, in <module>
    from sqlalchemy.types import \
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/types.py", line 28, in <module>
    from sqlalchemy.util import pickle, Decimal as _python_Decimal
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/util.py", line 21, in <module>
    set_types = set, sets.Set
exceptions.AttributeError: 'module' object has no attribute 'Set'
===============================================================================
[ERROR]: sage.dsage.twisted.tests.test_pubkeyauth

Traceback (most recent call last):
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/trial/runner.py", line 546, in loadPackage
    module = modinfo.load()
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 380, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 618, in moduleLoader
    return self._moduleLoader(modname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 426, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 377, in _importAndCheckStack
    return __import__(importName)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/twisted/tests/test_pubkeyauth.py", line 38, in <module>
    from sage.dsage.database.jobdb import JobDatabaseSA as JobDatabase
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/jobdb.py", line 21, in <module>
    from sqlalchemy import *
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/__init__.py", line 8, in <module>
    from sqlalchemy.types import \
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/types.py", line 28, in <module>
    from sqlalchemy.util import pickle, Decimal as _python_Decimal
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/util.py", line 21, in <module>
    set_types = set, sets.Set
exceptions.AttributeError: 'module' object has no attribute 'Set'
===============================================================================
[ERROR]: sage.dsage.twisted.tests.test_remote

Traceback (most recent call last):
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/trial/runner.py", line 546, in loadPackage
    module = modinfo.load()
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 380, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/modules.py", line 618, in moduleLoader
    return self._moduleLoader(modname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 426, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/twisted/python/reflect.py", line 377, in _importAndCheckStack
    return __import__(importName)
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/twisted/tests/test_remote.py", line 35, in <module>
    from sage.dsage.database.jobdb import JobDatabaseSA as JobDatabase
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/sage/dsage/database/jobdb.py", line 21, in <module>
    from sqlalchemy import *
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/__init__.py", line 8, in <module>
    from sqlalchemy.types import \
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/types.py", line 28, in <module>
    from sqlalchemy.util import pickle, Decimal as _python_Decimal
  File "/home/gfurnish/sage-3.0.4-x86_64-Linux/local/lib/python2.5/site-packages/SQLAlchemy-0.4.6-py2.5.egg/sqlalchemy/util.py", line 21, in <module>
    set_types = set, sets.Set
exceptions.AttributeError: 'module' object has no attribute 'Set'
-------------------------------------------------------------------------------
Ran 15 tests in 0.017s

We discussed this patch pretty much in depth, so once it works properly (and I test it on some more architectures), I'll give it a positive review.

07/10/2008 08:25:36 AM changed by yi

The distfactor thing segfaulting is a known issue. I can't reproduce the unittest failures on sage.math, using 3.0.4. Can you try it on some other architecture and see if you get the same thing?

07/10/2008 02:56:31 PM changed by yi

Another note: You will see some errors about the reactor being left in an unclean state. This is OK for now and is expected. The unit tests should still all pass.

07/10/2008 03:48:57 PM changed by yi

  • attachment ppool_unit_tests.patch added.

New version which fixes the reactor left in an unclean state errors.

07/13/2008 11:40:42 AM changed by yi

  • attachment dsage_ppool.hg added.

bundle to be applied against 3.0.4

07/13/2008 11:40:45 AM changed by yi

  • attachment dsage_ppool.2.hg added.

bundle to be applied against 3.0.4

07/13/2008 11:41:58 AM changed by yi

Please apply dsage_pool.hg and run the unit tests and doctests. The doctests say that they are failing on dsage_interface.py and testdoc.py but that seems to be a problem with the doctest framework since all the tests actually pass (on sage.math, running 3.0.4)

11/27/2008 11:04:16 PM changed by was

  • summary changed from [with patch, needs work] implement process pool for dsage workers to [with patch, needs review] implement process pool for dsage workers.