#31103 closed enhancement (fixed)
sage.numerical.backends: Replace use of TestSuite by pytest
Reported by:  Matthias Köppe  Owned by:  

Priority:  major  Milestone:  sage9.4 
Component:  doctest framework  Keywords:  
Cc:  Tobias Diez, Nicolas M. Thiéry, Travis Scrimshaw, Frédéric Chapoton, Yuan Zhou  Merged in:  
Authors:  Tobias Diez, Matthias Koeppe  Reviewers:  Matthias Koeppe, Tobias Diez 
Report Upstream:  N/A  Work issues:  
Branch:  0592226 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  #30901, #31003, #30551  Stopgaps: 
Description (last modified by )
sage.numerical.backends
provides an opportunity to study use of pytest
as a replacement for TestSuite
(proposed in #30738) in a small selfcontained codebase.
GenericBackend
defines an API and a small number of _test_
methods to check that the API is implemented correctly by the backend implementations. We do this at the example of _test_ncols_nonnegative
, leaving the other test methods for followup tickets.
Important design constraint:
 The testing framework must be extensible by userdefined classes (or classes from separately installed Python packages  such as the existing
sage_numerical_backends_coin
).
This is accomplished by introducing a new GenericBackendTests
class which defines the checks that should hold for all backend implementations. Then for each backend we introduce a class that derives from GenericBackendTests
and returns in the backend
test fixture the backend under test.
The pytests are now also executed in sage t
after the doctests, e.g. bin/sageruntests verbose .
prints at the end
===================================================================================================================== test session starts ====================================================================================================================== platform linux  Python 3.8.6, pytest6.2.1, py1.10.0, pluggy0.13.1  /mnt/d/Programming/sage/src/.venv/bin/python3 cachedir: .pytest_cache rootdir: /mnt/d/Programming/sage/src collected 10 items sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_old_testsuite PASSED [ 10%] sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_ncols_nonnegative PASSED [ 20%] sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_ncols_nonnegative PASSED [ 30%] sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_old_testsuite PASSED [ 40%] sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_ncols_nonnegative PASSED [ 50%] sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_old_testsuite PASSED [ 60%] sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_ncols_nonnegative PASSED [ 70%] sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_old_testsuite PASSED [ 80%] sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_ncols_nonnegative PASSED [ 90%] sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_old_testsuite PASSED [100%]
Change History (51)
comment:1 Changed 21 months ago by
comment:2 followup: 6 Changed 21 months ago by
Great point. This brings me to the question whether it would be possible to either
 invoke the existing
TestSuite
as one pytest method or  dynamically generate adapter methods that invoke the existing
_test
methods.
comment:3 Changed 21 months ago by
Branch:  → public/refactoring/pytestnumericalbackends 

Dependencies:  → #30901, #31003 
Description:  modified (diff) 
Milestone:  sagewishlist → sage9.3 
Status:  new → needs_review 
comment:4 Changed 21 months ago by
Commit:  → 719e442e5d3a3eaadab28bc835b68d46a4a2d12d 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
6dd6e5c  Fix compilation

eceefb3  Remove string wrap

d345bff  Fix test

c47c4bf  Correct indent

3fcaf5f  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/multiarchsimple

090e6f1  Simplify code

fa4556a  Remove _get_sage_local

4cdf94d  Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/refactoring/pytestnumericalbackends

5fd49b7  Migrate some tests to pytest

719e442  Run pytest as part of sage t

comment:5 Changed 21 months ago by
Description:  modified (diff) 

comment:6 Changed 21 months ago by
Replying to mkoeppe:
 invoke the existing
TestSuite
as one pytest method or
That's a great idea, making it possible to stepbystep migrate more test methods without the need to do everything at once! I've used it now.
 dynamically generate adapter methods that invoke the existing
_test
methods.
That might also work, but is more work since the _test
methods are instance methods, and for pytest the adapter method would need to be provided with the instance (which in the current design is passed to the TestSuite
). It's probably easier to simply rewrite the test method in pytest.
comment:8 followup: 12 Changed 21 months ago by
Some quick remarks (I haven't had a chance to test it yet):
 I'd suggest to rename
test_old_testsuite
to something more specific liketest_sage_unittest_testsuite
.  In the addition to
src/bin/sageruntests
 this needs to work without error ifpytest
is not installed (currently it is only an optional package  see #31110)
comment:9 Changed 21 months ago by
Authors:  → Tobias Diez 

comment:10 followup: 13 Changed 21 months ago by
Also, perhaps src/bin/sageruntests
should only invoke pytest
if it is testing directories/packages that contain pytest tests (any heuristic for checking this is fine).
comment:11 Changed 21 months ago by
Commit:  719e442e5d3a3eaadab28bc835b68d46a4a2d12d → d1f14f011c23a1a218400af7400edb7e1c613c0b 

Branch pushed to git repo; I updated commit sha1. New commits:
d1f14f0  Incorporate feedback

comment:12 Changed 21 months ago by
Replying to mkoeppe:
Some quick remarks (I haven't had a chance to test it yet):
 I'd suggest to rename
test_old_testsuite
to something more specific liketest_sage_unittest_testsuite
. In the addition to
src/bin/sageruntests
 this needs to work without error ifpytest
is not installed (currently it is only an optional package  see #31110)
Thanks! Done.
comment:13 followup: 14 Changed 21 months ago by
Replying to mkoeppe:
Also, perhaps
src/bin/sageruntests
should only invokepytest
if it is testing directories/packages that contain pytest tests (any heuristic for checking this is fine).
Mhh, pytest has its own test discovery bultin, so I think its fine to just pass the path to it and let it look for the files. What advantages do you see of using a custom heuristic beforehand?
comment:14 Changed 21 months ago by
Replying to ghtobiasdiez:
What advantages do you see of using a custom heuristic beforehand?
When pytest is not installed and you run sage t
on all of Sage, it should warn that the pytestdependent tests are not run. (Your latest commit does that.)
But it should probably not warn if a user invokes sage t
on a part of Sage that does not use pytest yet  to be less annoying.
comment:15 followup: 16 Changed 21 months ago by
Shouldn't test_sage_unittest_testsuite
be provided by something more general than GenericBackendTests
? (in other words, should GenericBackendTests
not derive from some other class?)
comment:16 Changed 21 months ago by
Replying to mkoeppe:
Shouldn't
test_sage_unittest_testsuite
be provided by something more general thanGenericBackendTests
? (in other words, shouldGenericBackendTests
not derive from some other class?)
Good idea for later purposes. However, it doesn't remove the test_sage_unittest_testsuite method in GenericBackendTests? since all backend tests ignore test_pickling, so this still has to be done there.
comment:17 Changed 21 months ago by
Commit:  d1f14f011c23a1a218400af7400edb7e1c613c0b → f4a0c4a19b7cb07df3cb95295fc5d3b794e0c7d5 

Branch pushed to git repo; I updated commit sha1. New commits:
f4a0c4a  Create SageObJectTest

comment:18 followup: 20 Changed 21 months ago by
Cc:  Frédéric Chapoton added 

Great. Could you add a docstring for test_sage_unittest_testsuite
to explain that "subclasses can override this method if they need to skip some tests".
We should also clarify whether ..._test.py
should be exempt from our rule that every function needs a docstring and a doctest. If so, probably the patchbot will need some adjustments so it will not complain.
comment:19 Changed 21 months ago by
Commit:  f4a0c4a19b7cb07df3cb95295fc5d3b794e0c7d5 → 3c06234e703737f5ddaeb09caa4161ab26578ba7 

Branch pushed to git repo; I updated commit sha1. New commits:
3c06234  Add docstring

comment:20 Changed 21 months ago by
Replying to mkoeppe:
We should also clarify whether
..._test.py
should be exempt from our rule that every function needs a docstring and a doctest. If so, probably the patchbot will need some adjustments so it will not complain.
That's a good idea. Tests indeed usually don't have docstrings (because they are only supposed to test one thing, which can be described by the name of the test method).
I think we also should fix our convention concerning the naming of the test files. There are three possibilities:
object_test.py
in the same folder as the moduleobject.py
(advantage: group test with object)test_object.py
in the same folder as the moduleobject.py
(advantage: group tests) either of the above but in a new
test
folder (advantage: clear separation of tests and source)
comment:21 Changed 21 months ago by
I have a strong preference for the first option, object_test.py
in the same folder... because somehow it is the closest to our current practice of keeping tests as close as possible to the module.
comment:24 Changed 20 months ago by
Matthias, do you encounter the same cython module issue if you run sageruntests
from this branch (you can exit the normal doctests with Ctrl+C)?
comment:25 Changed 20 months ago by
Yes
(sagesh) mkoeppe@egret:worktreealgebraic2018spring$ pytest src =========================================================================== test session starts =========================================================================== platform darwin  Python 3.9.1, pytest6.2.1, py1.10.0, pluggy0.13.1 rootdir: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src collected 0 items / 8 errors ================================================================================= ERRORS ================================================================================== _____________________________________________________ ERROR collecting sage/numerical/backends/cvxopt_backend_test.py _____________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/cvxopt_backend_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/numerical/backends/cvxopt_backend_test.py:2: in <module> from sage.numerical.backends.generic_backend_test import GenericBackendTests src/sage/numerical/backends/generic_backend_test.py:2: in <module> from .generic_backend import GenericBackend E ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend' ____________________________________________________ ERROR collecting sage/numerical/backends/generic_backend_test.py _____________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/generic_backend_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/numerical/backends/generic_backend_test.py:2: in <module> from .generic_backend import GenericBackend E ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend' ______________________________________________________ ERROR collecting sage/numerical/backends/glpk_backend_test.py ______________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/glpk_backend_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/numerical/backends/glpk_backend_test.py:2: in <module> from sage.numerical.backends.generic_backend_test import GenericBackendTests src/sage/numerical/backends/generic_backend_test.py:2: in <module> from .generic_backend import GenericBackend E ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend' ___________________________________________________ ERROR collecting sage/numerical/backends/glpk_exact_backend_test.py ___________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/glpk_exact_backend_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/numerical/backends/glpk_exact_backend_test.py:2: in <module> from sage.numerical.backends.generic_backend_test import GenericBackendTests src/sage/numerical/backends/generic_backend_test.py:2: in <module> from .generic_backend import GenericBackend E ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend' _________________________________________________ ERROR collecting sage/numerical/backends/interactivelp_backend_test.py __________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/interactivelp_backend_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/numerical/backends/interactivelp_backend_test.py:2: in <module> from sage.numerical.backends.generic_backend_test import GenericBackendTests src/sage/numerical/backends/generic_backend_test.py:2: in <module> from .generic_backend import GenericBackend E ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend' ______________________________________________________ ERROR collecting sage/numerical/backends/ppl_backend_test.py _______________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/ppl_backend_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/numerical/backends/ppl_backend_test.py:2: in <module> from sage.numerical.backends.generic_backend_test import GenericBackendTests src/sage/numerical/backends/generic_backend_test.py:2: in <module> from .generic_backend import GenericBackend E ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend' ___________________________________________________________ ERROR collecting sage/structure/sage_object_test.py ___________________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/structure/sage_object_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/structure/__init__.py:2: in <module> import sage.structure.element E ModuleNotFoundError: No module named 'sage.structure.element' _____________________________________________________________ ERROR collecting sage/tests/deprecation_test.py _____________________________________________________________ ImportError while importing test module '/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/tests/deprecation_test.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/sage/tests/deprecation_test.py:12: in <module> from sage.misc.superseded import deprecated_function_alias src/sage/misc/superseded.py:29: in <module> from sage.misc.lazy_attribute import lazy_attribute E ModuleNotFoundError: No module named 'sage.misc.lazy_attribute' ========================================================================= short test summary info ========================================================================= ERROR src/sage/numerical/backends/cvxopt_backend_test.py ERROR src/sage/numerical/backends/generic_backend_test.py ERROR src/sage/numerical/backends/glpk_backend_test.py ERROR src/sage/numerical/backends/glpk_exact_backend_test.py ERROR src/sage/numerical/backends/interactivelp_backend_test.py ERROR src/sage/numerical/backends/ppl_backend_test.py ERROR src/sage/structure/sage_object_test.py ERROR src/sage/tests/deprecation_test.py
comment:26 Changed 20 months ago by
Running pytest
directly only works if you have an editable install, so that's not a surprise. But does it work if you run sageruntests
which now invokes pytest (using python code). From what I read, this should work.
comment:27 Changed 20 months ago by
Milestone:  sage9.3 → sage9.4 

Status:  needs_review → needs_work 
Sorry, I missed that.
$ ./sage tp src/sage/numerical/backends/ too many failed tests, not using stored timings Running doctests with ID 202101241026167200536b. Using optional=build,ccache,dochtml,homebrew,pip,saclib,sage,sage_spkg Sorting sources by runtime so that slower doctests are run first.... Doctesting 25 files using 8 threads. sage t randomseed=0 src/sage/numerical/backends/interactivelp_backend.pxd [0 tests, 0.00 s] [...]  All tests passed!  Total time for all tests: 4.4 seconds cpu time: 7.3 seconds cumulative wall time: 12.9 seconds =========================================================================== test session starts =========================================================================== platform darwin  Python 3.9.1, pytest6.2.1, py1.10.0, pluggy0.13.1 rootdir: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src, configfile: tox.ini collected 0 items / 6 errors ================================================================================= ERRORS ================================================================================== _____________________________________________________ ERROR collecting sage/numerical/backends/cvxopt_backend_test.py _____________________________________________________ /usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) <frozen importlib._bootstrap>:1030: in _gcd_import ??? <frozen importlib._bootstrap>:1007: in _find_and_load ??? <frozen importlib._bootstrap>:986: in _find_and_load_unlocked ??? <frozen importlib._bootstrap>:680: in _load_unlocked ??? local/lib/python3.9/sitepackages/_pytest/assertion/rewrite.py:170: in exec_module exec(co, module.__dict__) local/lib/python3.9/sitepackages/sage/numerical/backends/cvxopt_backend_test.py:6: in <module> class TestCVXOPTBackend(GenericBackendTests): local/lib/python3.9/sitepackages/sage/numerical/backends/cvxopt_backend_test.py:12: in TestCVXOPTBackend def test_sage_unittest_testsuite(self, sage_object: SageObject): E NameError: name 'SageObject' is not defined ____________________________________________________ ERROR collecting sage/numerical/backends/generic_backend_test.py _____________________________________________________ import file mismatch: imported module 'sage.numerical.backends.generic_backend_test' has this __file__ attribute: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.9/sitepackages/sage/numerical/backends/generic_backend_test.py which is not the same as the test file we want to collect: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/generic_backend_test.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules ______________________________________________________ ERROR collecting sage/numerical/backends/glpk_backend_test.py ______________________________________________________ import file mismatch: imported module 'sage.numerical.backends.glpk_backend_test' has this __file__ attribute: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.9/sitepackages/sage/numerical/backends/glpk_backend_test.py which is not the same as the test file we want to collect: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/glpk_backend_test.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules ___________________________________________________ ERROR collecting sage/numerical/backends/glpk_exact_backend_test.py ___________________________________________________ import file mismatch: imported module 'sage.numerical.backends.glpk_exact_backend_test' has this __file__ attribute: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.9/sitepackages/sage/numerical/backends/glpk_exact_backend_test.py which is not the same as the test file we want to collect: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/glpk_exact_backend_test.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules _________________________________________________ ERROR collecting sage/numerical/backends/interactivelp_backend_test.py __________________________________________________ import file mismatch: imported module 'sage.numerical.backends.interactivelp_backend_test' has this __file__ attribute: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.9/sitepackages/sage/numerical/backends/interactivelp_backend_test.py which is not the same as the test file we want to collect: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/interactivelp_backend_test.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules ______________________________________________________ ERROR collecting sage/numerical/backends/ppl_backend_test.py _______________________________________________________ import file mismatch: imported module 'sage.numerical.backends.ppl_backend_test' has this __file__ attribute: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.9/sitepackages/sage/numerical/backends/ppl_backend_test.py which is not the same as the test file we want to collect: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/sage/numerical/backends/ppl_backend_test.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules ========================================================================= short test summary info ========================================================================= ERROR src/sage/numerical/backends/cvxopt_backend_test.py  NameError: name 'SageObject' is not defined ERROR src/sage/numerical/backends/generic_backend_test.py ERROR src/sage/numerical/backends/glpk_backend_test.py ERROR src/sage/numerical/backends/glpk_exact_backend_test.py ERROR src/sage/numerical/backends/interactivelp_backend_test.py ERROR src/sage/numerical/backends/ppl_backend_test.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ============================================================================ 6 errors in 0.55s ============================================================================
comment:29 Changed 20 months ago by
Any progress here? There's some momentum to add more tests to generic_backend, and it would be great if they could be done in the new framework already
comment:30 followup: 35 Changed 20 months ago by
By default, pytest expects the compiled cython files to be in the same folder (inplace install). To support sage's build system, one needs to modify sys.path
in the conftest file as shown here: https://stackoverflow.com/questions/20971619/ensuringpytestincludestheapplicationdirectoryinsyspath
What is the correct SAGE_XYZ folder to specify there? SAGE_LOCAL?
Moreover, it could be a problem that the test files are present twice (in local and src), which I think leads to the other errors above. Maybe https://docs.pytest.org/en/stable/pythonpath.html helps.
I will have a look at these things, but it will take some time as I have to destroy my nicely working inplace install for this. Feel free to play around with the above linked code.
comment:31 Changed 20 months ago by
In an install without pytest, I am getting:
./sage t src/sage/numerical/backends/ ... Pytest is not installed, skip checking tests that rely on it. Traceback (most recent call last): File "/Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/src/bin/sageruntests", line 196, in <module> sys.exit(exit_code_pytest) NameError: name 'exit_code_pytest' is not defined
comment:32 Changed 20 months ago by
Commit:  3c06234e703737f5ddaeb09caa4161ab26578ba7 → 7935ecf70803e16dbd10f1b7a179f50083c20cbb 

Branch pushed to git repo; I updated commit sha1. New commits:
bf014ba  Enforce that test files need to end on _test.py

38e210e  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/pytest_config

6880e1d  Merge tag '9.3.beta6' into t/31003/public/build/pytest_config

62b1c61  Igonre deprecation_test as well

da37e14  Merge branch 'public/build/pytest_config' of git://trac.sagemath.org/sage into t/31003/public/build/pytest_config

15c48da  Merge branch 't/31003/public/build/pytest_config' into t/31103/public/refactoring/pytestnumericalbackends

7935ecf  src/bin/sageruntests: No error when pytest cannot be imported

comment:33 Changed 20 months ago by
Commit:  7935ecf70803e16dbd10f1b7a179f50083c20cbb → c81c8db12aee63238cedac4a05a526f3d121fe43 

comment:35 Changed 20 months ago by
Replying to ghtobiasdiez:
Maybe https://docs.pytest.org/en/stable/pythonpath.html helps.
Yes, it did!
comment:36 Changed 20 months ago by
Cc:  Yuan Zhou added 

comment:37 Changed 20 months ago by
The only remaining problem that I have is that I need to update the 3 standalone packages providing additional backends (https://github.com/mkoeppe/sagenumericalbackendscoin, ...) to also invoke pytest  and to do make sure this does not cause errors on older Sage versions.
comment:38 Changed 20 months ago by
To keep the work limited, perhaps it's best to keep the old _test_
methods so that the test coverage of the backend packages is not sacrificed. We will delete the _test_
methods later.
comment:39 Changed 20 months ago by
Commit:  c81c8db12aee63238cedac4a05a526f3d121fe43 → 0592226745971878657539a740fa54b065c9d0bb 

Branch pushed to git repo; I updated commit sha1. New commits:
0592226  src/sage/numerical/backends/generic_backend.pyx: Restore _test_ncols_nonnegative, with comment

comment:40 Changed 20 months ago by
Authors:  Tobias Diez → Tobias Diez, Matthias Koeppe 

Reviewers:  → Matthias Koeppe, ... 
Status:  needs_work → needs_review 
comment:41 Changed 20 months ago by
Reviewers:  Matthias Koeppe, ... → Matthias Koeppe, Tobias Diez 

Status:  needs_review → positive_review 
Thanks a lot for having a look at this. Your changes look good to me!
comment:42 Changed 20 months ago by
Dependencies:  #30901, #31003 → #30901, #31003, #30551 

Adding #30551 as a dependency because of from __future__ import annotations
. Or can we avoid it?
comment:43 Changed 16 months ago by
Branch:  public/refactoring/pytestnumericalbackends → 0592226745971878657539a740fa54b065c9d0bb 

Resolution:  → fixed 
Status:  positive_review → closed 
comment:44 Changed 12 months ago by
Commit:  0592226745971878657539a740fa54b065c9d0bb 

This seems to break our workflow at https://github.com/flatsurf/eantic. We are running some SageMath doctests with sageruntests
. The tests all pass and then we see:
collected 0 items ====================================================== no tests ran in 0.00s ======================================================
The problem is that this now returns with exit code 5, i.e., the CI assumes that the tests have failed.
comment:45 Changed 12 months ago by
A workaround is to drop a test_nothing.py
file in the directory where the doctests are run:
def test_nothing():pass
comment:46 Changed 12 months ago by
Help with fixing this properly is welcome. The developer who worked on the pytest stuff has disappeared
comment:47 Changed 12 months ago by
A proper fix would probably be to add an additional if check if the pytest return code is 5 here:
+ if err == 0: + sys.exit(exit_code_pytest) + else: + sys.exit(err)
See also the discussion in https://github.com/pytestdev/pytest/issues/2393.
I sadly don't have the time to implement it though.
comment:48 Changed 11 months ago by
The workaround in https://trac.sagemath.org/ticket/31103#comment:45 does not work if the tested module contains some sage metaclass machinery since pytest is unvoked with importmode importlib
, there is an error because the module is not in sys.modules
and this confuses nested_class.pyx
.
In that case, there is an error and not only an error code 5.
==================================== ERRORS ==================================== ____________________ ERROR collecting pyflatsurf/vector.py _____________________ ../src/pyflatsurf/vector.py:285: in <module> class Vectors(UniqueRepresentation, Parent): sage/misc/nested_class.pyx:318: in sage.misc.nested_class.NestedClassMetaclass.__init__ (build/cythonized/sage/misc/nested_class.c:2314) ??? E KeyError: 'vector'
comment:49 Changed 11 months ago by
Another workaround is to just pretend that pytest is not installed:
$ cat nopytest/pytest.py raise ModuleNotFoundError $ PYTHONPATH=nopytest:$PYTHONPATH sage tp ...
But I actually think we should make this a bit more convenient than having to hack the module search path.
What should happen with the generic tests methods like
_test_new, _test_not_implemented_methods, _test_category, _test_pickling
that sadly make the "small selfcontained" part hard?