Opened 21 months ago

Closed 16 months ago

Last modified 10 months ago

#31103 closed enhancement (fixed)

sage.numerical.backends: Replace use of TestSuite by pytest

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.4
Component: doctest framework Keywords:
Cc: Tobias Diez, Nicolas M. Thiéry, Travis Scrimshaw, Frédéric Chapoton, Yuan Zhou Merged in:
Authors: Tobias Diez, Matthias Koeppe Reviewers: Matthias Koeppe, Tobias Diez
Report Upstream: N/A Work issues:
Branch: 0592226 (Commits, GitHub, GitLab) Commit:
Dependencies: #30901, #31003, #30551 Stopgaps:

Status badges

Description (last modified by Tobias Diez)

sage.numerical.backends provides an opportunity to study use of pytest as a replacement for TestSuite (proposed in #30738) in a small self-contained codebase.

GenericBackend defines an API and a small number of _test_ methods to check that the API is implemented correctly by the backend implementations. We do this at the example of _test_ncols_nonnegative, leaving the other test methods for follow-up tickets.

Important design constraint:

  • The testing framework must be extensible by user-defined classes (or classes from separately installed Python packages - such as the existing sage_numerical_backends_coin).

This is accomplished by introducing a new GenericBackendTests class which defines the checks that should hold for all backend implementations. Then for each backend we introduce a class that derives from GenericBackendTests and returns in the backend test fixture the backend under test.

The pytests are now also executed in sage -t after the doctests, e.g. bin/sage-runtests --verbose . prints at the end

===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /mnt/d/Programming/sage/src/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /mnt/d/Programming/sage/src
collected 10 items                                                                                                                                                                                                                                             

sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_old_testsuite PASSED                                                                                                                                                             [ 10%]
sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_ncols_nonnegative PASSED                                                                                                                                                         [ 20%]
sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_ncols_nonnegative PASSED                                                                                                                                                             [ 30%]
sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_old_testsuite PASSED                                                                                                                                                                 [ 40%]
sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_ncols_nonnegative PASSED                                                                                                                                                  [ 50%]
sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_old_testsuite PASSED                                                                                                                                                      [ 60%]
sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_ncols_nonnegative PASSED                                                                                                                                           [ 70%]
sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_old_testsuite PASSED                                                                                                                                               [ 80%]
sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_ncols_nonnegative PASSED                                                                                                                                                               [ 90%]
sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_old_testsuite PASSED                                                                                                                                                                   [100%]

Change History (51)

comment:1 Changed 21 months ago by Tobias Diez

What should happen with the generic tests methods like _test_new, _test_not_implemented_methods, _test_category, _test_pickling that sadly make the "small self-contained" part hard?

comment:2 Changed 21 months ago by Matthias Köppe

Great point. This brings me to the question whether it would be possible to either

  • invoke the existing TestSuite as one pytest method or
  • dynamically generate adapter methods that invoke the existing _test methods.
Last edited 21 months ago by Matthias Köppe (previous) (diff)

comment:3 Changed 21 months ago by Tobias Diez

Branch: public/refactoring/pytest-numerical-backends
Dependencies: #30901, #31003
Description: modified (diff)
Milestone: sage-wishlistsage-9.3
Status: newneeds_review

comment:4 Changed 21 months ago by git

Commit: 719e442e5d3a3eaadab28bc835b68d46a4a2d12d

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

6dd6e5cFix compilation
eceefb3Remove string wrap
d345bffFix test
c47c4bfCorrect indent
3fcaf5fMerge branch 'develop' of git://github.com/sagemath/sage into public/build/multiarchsimple
090e6f1Simplify code
fa4556aRemove _get_sage_local
4cdf94dMerge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/refactoring/pytest-numerical-backends
5fd49b7Migrate some tests to pytest
719e442Run pytest as part of sage -t

comment:5 Changed 21 months ago by Tobias Diez

Description: modified (diff)

comment:6 in reply to:  2 Changed 21 months ago by Tobias Diez

Replying to mkoeppe:

  • invoke the existing TestSuite as one pytest method or

That's a great idea, making it possible to step-by-step migrate more test methods without the need to do everything at once! I've used it now.

  • dynamically generate adapter methods that invoke the existing _test methods.

That might also work, but is more work since the _test methods are instance methods, and for pytest the adapter method would need to be provided with the instance (which in the current design is passed to the TestSuite). It's probably easier to simply rewrite the test method in pytest.

comment:7 Changed 21 months ago by Matthias Köppe

Thanks a lot!

comment:8 Changed 21 months ago by Matthias Köppe

Some quick remarks (I haven't had a chance to test it yet):

  • I'd suggest to rename test_old_testsuite to something more specific like test_sage_unittest_testsuite.
  • In the addition to src/bin/sage-runtests - this needs to work without error if pytest is not installed (currently it is only an optional package - see #31110)

comment:9 Changed 21 months ago by Matthias Köppe

Authors: Tobias Diez

comment:10 Changed 21 months ago by Matthias Köppe

Also, perhaps src/bin/sage-runtests should only invoke pytest if it is testing directories/packages that contain pytest tests (any heuristic for checking this is fine).

comment:11 Changed 21 months ago by git

Commit: 719e442e5d3a3eaadab28bc835b68d46a4a2d12dd1f14f011c23a1a218400af7400edb7e1c613c0b

Branch pushed to git repo; I updated commit sha1. New commits:

d1f14f0Incorporate feedback

comment:12 in reply to:  8 Changed 21 months ago by Tobias Diez

Replying to mkoeppe:

Some quick remarks (I haven't had a chance to test it yet):

  • I'd suggest to rename test_old_testsuite to something more specific like test_sage_unittest_testsuite.
  • In the addition to src/bin/sage-runtests - this needs to work without error if pytest is not installed (currently it is only an optional package - see #31110)

Thanks! Done.

comment:13 in reply to:  10 ; Changed 21 months ago by Tobias Diez

Replying to mkoeppe:

Also, perhaps src/bin/sage-runtests should only invoke pytest if it is testing directories/packages that contain pytest tests (any heuristic for checking this is fine).

Mhh, pytest has its own test discovery bult-in, so I think its fine to just pass the path to it and let it look for the files. What advantages do you see of using a custom heuristic beforehand?

comment:14 in reply to:  13 Changed 21 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

What advantages do you see of using a custom heuristic beforehand?

When pytest is not installed and you run sage -t on all of Sage, it should warn that the pytest-dependent tests are not run. (Your latest commit does that.)

But it should probably not warn if a user invokes sage -t on a part of Sage that does not use pytest yet -- to be less annoying.

comment:15 Changed 21 months ago by Matthias Köppe

Shouldn't test_sage_unittest_testsuite be provided by something more general than GenericBackendTests? (in other words, should GenericBackendTests not derive from some other class?)

comment:16 in reply to:  15 Changed 21 months ago by Tobias Diez

Replying to mkoeppe:

Shouldn't test_sage_unittest_testsuite be provided by something more general than GenericBackendTests? (in other words, should GenericBackendTests not derive from some other class?)

Good idea for later purposes. However, it doesn't remove the test_sage_unittest_testsuite method in GenericBackendTests? since all backend tests ignore test_pickling, so this still has to be done there.

comment:17 Changed 21 months ago by git

Commit: d1f14f011c23a1a218400af7400edb7e1c613c0bf4a0c4a19b7cb07df3cb95295fc5d3b794e0c7d5

Branch pushed to git repo; I updated commit sha1. New commits:

f4a0c4aCreate SageObJectTest

comment:18 Changed 21 months ago by Matthias Köppe

Cc: Frédéric Chapoton added

Great. Could you add a docstring for test_sage_unittest_testsuite to explain that "subclasses can override this method if they need to skip some tests".

We should also clarify whether ..._test.py should be exempt from our rule that every function needs a docstring and a doctest. If so, probably the patchbot will need some adjustments so it will not complain.

comment:19 Changed 21 months ago by git

Commit: f4a0c4a19b7cb07df3cb95295fc5d3b794e0c7d53c06234e703737f5ddaeb09caa4161ab26578ba7

Branch pushed to git repo; I updated commit sha1. New commits:

3c06234Add docstring

comment:20 in reply to:  18 Changed 21 months ago by Tobias Diez

Replying to mkoeppe:

We should also clarify whether ..._test.py should be exempt from our rule that every function needs a docstring and a doctest. If so, probably the patchbot will need some adjustments so it will not complain.

That's a good idea. Tests indeed usually don't have docstrings (because they are only supposed to test one thing, which can be described by the name of the test method).

I think we also should fix our convention concerning the naming of the test files. There are three possibilities:

  • object_test.py in the same folder as the module object.py (advantage: group test with object)
  • test_object.py in the same folder as the module object.py (advantage: group tests)
  • either of the above but in a new test folder (advantage: clear separation of tests and source)

comment:21 Changed 21 months ago by Matthias Köppe

I have a strong preference for the first option, object_test.py in the same folder... because somehow it is the closest to our current practice of keeping tests as close as possible to the module.

comment:22 Changed 21 months ago by Tobias Diez

Ok, I've added this convention to #31003.

comment:23 Changed 21 months ago by Tobias Diez

This and #31003 still need review.

comment:24 Changed 20 months ago by Tobias Diez

Matthias, do you encounter the same cython module issue if you run sage-runtests from this branch (you can exit the normal doctests with Ctrl+C)?

comment:25 Changed 20 months ago by Matthias Köppe

Yes

(sage-sh) mkoeppe@egret:worktree-algebraic-2018-spring$ pytest src
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.9.1, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src
collected 0 items / 8 errors                                                                                                                                              

================================================================================= ERRORS ==================================================================================
_____________________________________________________ ERROR collecting sage/numerical/backends/cvxopt_backend_test.py _____________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/cvxopt_backend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/numerical/backends/cvxopt_backend_test.py:2: in <module>
    from sage.numerical.backends.generic_backend_test import GenericBackendTests
src/sage/numerical/backends/generic_backend_test.py:2: in <module>
    from .generic_backend import GenericBackend
E   ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend'
____________________________________________________ ERROR collecting sage/numerical/backends/generic_backend_test.py _____________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/generic_backend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/numerical/backends/generic_backend_test.py:2: in <module>
    from .generic_backend import GenericBackend
E   ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend'
______________________________________________________ ERROR collecting sage/numerical/backends/glpk_backend_test.py ______________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/glpk_backend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/numerical/backends/glpk_backend_test.py:2: in <module>
    from sage.numerical.backends.generic_backend_test import GenericBackendTests
src/sage/numerical/backends/generic_backend_test.py:2: in <module>
    from .generic_backend import GenericBackend
E   ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend'
___________________________________________________ ERROR collecting sage/numerical/backends/glpk_exact_backend_test.py ___________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/glpk_exact_backend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/numerical/backends/glpk_exact_backend_test.py:2: in <module>
    from sage.numerical.backends.generic_backend_test import GenericBackendTests
src/sage/numerical/backends/generic_backend_test.py:2: in <module>
    from .generic_backend import GenericBackend
E   ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend'
_________________________________________________ ERROR collecting sage/numerical/backends/interactivelp_backend_test.py __________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/interactivelp_backend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/numerical/backends/interactivelp_backend_test.py:2: in <module>
    from sage.numerical.backends.generic_backend_test import GenericBackendTests
src/sage/numerical/backends/generic_backend_test.py:2: in <module>
    from .generic_backend import GenericBackend
E   ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend'
______________________________________________________ ERROR collecting sage/numerical/backends/ppl_backend_test.py _______________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/ppl_backend_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/numerical/backends/ppl_backend_test.py:2: in <module>
    from sage.numerical.backends.generic_backend_test import GenericBackendTests
src/sage/numerical/backends/generic_backend_test.py:2: in <module>
    from .generic_backend import GenericBackend
E   ModuleNotFoundError: No module named 'sage.numerical.backends.generic_backend'
___________________________________________________________ ERROR collecting sage/structure/sage_object_test.py ___________________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/structure/sage_object_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/structure/__init__.py:2: in <module>
    import sage.structure.element
E   ModuleNotFoundError: No module named 'sage.structure.element'
_____________________________________________________________ ERROR collecting sage/tests/deprecation_test.py _____________________________________________________________
ImportError while importing test module '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/tests/deprecation_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/sage/tests/deprecation_test.py:12: in <module>
    from sage.misc.superseded import deprecated_function_alias
src/sage/misc/superseded.py:29: in <module>
    from sage.misc.lazy_attribute import lazy_attribute
E   ModuleNotFoundError: No module named 'sage.misc.lazy_attribute'
========================================================================= short test summary info =========================================================================
ERROR src/sage/numerical/backends/cvxopt_backend_test.py
ERROR src/sage/numerical/backends/generic_backend_test.py
ERROR src/sage/numerical/backends/glpk_backend_test.py
ERROR src/sage/numerical/backends/glpk_exact_backend_test.py
ERROR src/sage/numerical/backends/interactivelp_backend_test.py
ERROR src/sage/numerical/backends/ppl_backend_test.py
ERROR src/sage/structure/sage_object_test.py
ERROR src/sage/tests/deprecation_test.py

comment:26 Changed 20 months ago by Tobias Diez

Running pytest directly only works if you have an editable install, so that's not a surprise. But does it work if you run sage-runtests which now invokes pytest (using python code). From what I read, this should work.

comment:27 Changed 20 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4
Status: needs_reviewneeds_work

Sorry, I missed that.

$ ./sage -tp src/sage/numerical/backends/
too many failed tests, not using stored timings
Running doctests with ID 2021-01-24-10-26-16-7200536b.
Using --optional=build,ccache,dochtml,homebrew,pip,saclib,sage,sage_spkg
Sorting sources by runtime so that slower doctests are run first....
Doctesting 25 files using 8 threads.
sage -t --random-seed=0 src/sage/numerical/backends/interactivelp_backend.pxd
    [0 tests, 0.00 s]
[...]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 4.4 seconds
    cpu time: 7.3 seconds
    cumulative wall time: 12.9 seconds
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.9.1, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src, configfile: tox.ini
collected 0 items / 6 errors                                                                                                                                              

================================================================================= ERRORS ==================================================================================
_____________________________________________________ ERROR collecting sage/numerical/backends/cvxopt_backend_test.py _____________________________________________________
/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
local/lib/python3.9/site-packages/sage/numerical/backends/cvxopt_backend_test.py:6: in <module>
    class TestCVXOPTBackend(GenericBackendTests):
local/lib/python3.9/site-packages/sage/numerical/backends/cvxopt_backend_test.py:12: in TestCVXOPTBackend
    def test_sage_unittest_testsuite(self, sage_object: SageObject):
E   NameError: name 'SageObject' is not defined
____________________________________________________ ERROR collecting sage/numerical/backends/generic_backend_test.py _____________________________________________________
import file mismatch:
imported module 'sage.numerical.backends.generic_backend_test' has this __file__ attribute:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/numerical/backends/generic_backend_test.py
which is not the same as the test file we want to collect:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/generic_backend_test.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
______________________________________________________ ERROR collecting sage/numerical/backends/glpk_backend_test.py ______________________________________________________
import file mismatch:
imported module 'sage.numerical.backends.glpk_backend_test' has this __file__ attribute:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/numerical/backends/glpk_backend_test.py
which is not the same as the test file we want to collect:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/glpk_backend_test.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
___________________________________________________ ERROR collecting sage/numerical/backends/glpk_exact_backend_test.py ___________________________________________________
import file mismatch:
imported module 'sage.numerical.backends.glpk_exact_backend_test' has this __file__ attribute:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/numerical/backends/glpk_exact_backend_test.py
which is not the same as the test file we want to collect:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/glpk_exact_backend_test.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
_________________________________________________ ERROR collecting sage/numerical/backends/interactivelp_backend_test.py __________________________________________________
import file mismatch:
imported module 'sage.numerical.backends.interactivelp_backend_test' has this __file__ attribute:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/numerical/backends/interactivelp_backend_test.py
which is not the same as the test file we want to collect:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/interactivelp_backend_test.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
______________________________________________________ ERROR collecting sage/numerical/backends/ppl_backend_test.py _______________________________________________________
import file mismatch:
imported module 'sage.numerical.backends.ppl_backend_test' has this __file__ attribute:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/numerical/backends/ppl_backend_test.py
which is not the same as the test file we want to collect:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/numerical/backends/ppl_backend_test.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
========================================================================= short test summary info =========================================================================
ERROR src/sage/numerical/backends/cvxopt_backend_test.py - NameError: name 'SageObject' is not defined
ERROR src/sage/numerical/backends/generic_backend_test.py
ERROR src/sage/numerical/backends/glpk_backend_test.py
ERROR src/sage/numerical/backends/glpk_exact_backend_test.py
ERROR src/sage/numerical/backends/interactivelp_backend_test.py
ERROR src/sage/numerical/backends/ppl_backend_test.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================ 6 errors in 0.55s ============================================================================

comment:28 Changed 20 months ago by Tobias Diez

Ok, thanks for the feedback. I'll have a look.

comment:29 Changed 20 months ago by Matthias Köppe

Any progress here? There's some momentum to add more tests to generic_backend, and it would be great if they could be done in the new framework already

comment:30 Changed 20 months ago by Tobias Diez

By default, pytest expects the compiled cython files to be in the same folder (inplace install). To support sage's build system, one needs to modify sys.path in the conftest file as shown here: https://stackoverflow.com/questions/20971619/ensuring-py-test-includes-the-application-directory-in-sys-path What is the correct SAGE_XYZ folder to specify there? SAGE_LOCAL?

Moreover, it could be a problem that the test files are present twice (in local and src), which I think leads to the other errors above. Maybe https://docs.pytest.org/en/stable/pythonpath.html helps.

I will have a look at these things, but it will take some time as I have to destroy my nicely working inplace install for this. Feel free to play around with the above linked code.

comment:31 Changed 20 months ago by Matthias Köppe

In an install without pytest, I am getting:

./sage -t src/sage/numerical/backends/
...
Pytest is not installed, skip checking tests that rely on it.
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/bin/sage-runtests", line 196, in <module>
    sys.exit(exit_code_pytest)
NameError: name 'exit_code_pytest' is not defined

comment:32 Changed 20 months ago by git

Commit: 3c06234e703737f5ddaeb09caa4161ab26578ba77935ecf70803e16dbd10f1b7a179f50083c20cbb

Branch pushed to git repo; I updated commit sha1. New commits:

bf014baEnforce that test files need to end on _test.py
38e210eMerge branch 'develop' of git://github.com/sagemath/sage into public/build/pytest_config
6880e1dMerge tag '9.3.beta6' into t/31003/public/build/pytest_config
62b1c61Igonre deprecation_test as well
da37e14Merge branch 'public/build/pytest_config' of git://trac.sagemath.org/sage into t/31003/public/build/pytest_config
15c48daMerge branch 't/31003/public/build/pytest_config' into t/31103/public/refactoring/pytest-numerical-backends
7935ecfsrc/bin/sage-runtests: No error when pytest cannot be imported

comment:33 Changed 20 months ago by git

Commit: 7935ecf70803e16dbd10f1b7a179f50083c20cbbc81c8db12aee63238cedac4a05a526f3d121fe43

Branch pushed to git repo; I updated commit sha1. New commits:

cc31289src/bin/sage-runtests: Use pytest --import-mode importlib
c81c8dbsrc/sage/numerical/backends/*_test.py: fix imports

comment:34 Changed 20 months ago by Matthias Köppe

This version works for me

comment:35 in reply to:  30 Changed 20 months ago by Matthias Köppe

comment:36 Changed 20 months ago by Matthias Köppe

Cc: Yuan Zhou added

comment:37 Changed 20 months ago by Matthias Köppe

The only remaining problem that I have is that I need to update the 3 standalone packages providing additional backends (https://github.com/mkoeppe/sage-numerical-backends-coin, ...) to also invoke pytest - and to do make sure this does not cause errors on older Sage versions.

comment:38 Changed 20 months ago by Matthias Köppe

To keep the work limited, perhaps it's best to keep the old _test_ methods so that the test coverage of the backend packages is not sacrificed. We will delete the _test_ methods later.

comment:39 Changed 20 months ago by git

Commit: c81c8db12aee63238cedac4a05a526f3d121fe430592226745971878657539a740fa54b065c9d0bb

Branch pushed to git repo; I updated commit sha1. New commits:

0592226src/sage/numerical/backends/generic_backend.pyx: Restore _test_ncols_nonnegative, with comment

comment:40 Changed 20 months ago by Matthias Köppe

Authors: Tobias DiezTobias Diez, Matthias Koeppe
Reviewers: Matthias Koeppe, ...
Status: needs_workneeds_review

comment:41 Changed 20 months ago by Tobias Diez

Reviewers: Matthias Koeppe, ...Matthias Koeppe, Tobias Diez
Status: needs_reviewpositive_review

Thanks a lot for having a look at this. Your changes look good to me!

comment:42 Changed 20 months ago by Matthias Köppe

Dependencies: #30901, #31003#30901, #31003, #30551

Adding #30551 as a dependency because of from __future__ import annotations. Or can we avoid it?

comment:43 Changed 16 months ago by Volker Braun

Branch: public/refactoring/pytest-numerical-backends0592226745971878657539a740fa54b065c9d0bb
Resolution: fixed
Status: positive_reviewclosed

comment:44 Changed 12 months ago by Julian Rüth

Commit: 0592226745971878657539a740fa54b065c9d0bb

This seems to break our workflow at https://github.com/flatsurf/e-antic. We are running some SageMath doctests with sage-runtests. The tests all pass and then we see:

collected 0 items

====================================================== no tests ran in 0.00s ======================================================

The problem is that this now returns with exit code 5, i.e., the CI assumes that the tests have failed.

comment:45 Changed 12 months ago by Julian Rüth

A workaround is to drop a test_nothing.py file in the directory where the doctests are run:

def test_nothing():pass
Last edited 12 months ago by Julian Rüth (previous) (diff)

comment:46 Changed 12 months ago by Matthias Köppe

Help with fixing this properly is welcome. The developer who worked on the pytest stuff has disappeared

comment:47 Changed 12 months ago by Tobias Diez

A proper fix would probably be to add an additional if check if the pytest return code is 5 here:

+    if err == 0:
+        sys.exit(exit_code_pytest)
+    else:
+        sys.exit(err)

See also the discussion in https://github.com/pytest-dev/pytest/issues/2393.

I sadly don't have the time to implement it though.

comment:48 Changed 11 months ago by Julian Rüth

The workaround in https://trac.sagemath.org/ticket/31103#comment:45 does not work if the tested module contains some sage metaclass machinery since pytest is unvoked with --import-mode importlib, there is an error because the module is not in sys.modules and this confuses nested_class.pyx.

In that case, there is an error and not only an error code 5.

  ==================================== ERRORS ====================================
  ____________________ ERROR collecting pyflatsurf/vector.py _____________________
  ../src/pyflatsurf/vector.py:285: in <module>
      class Vectors(UniqueRepresentation, Parent):
  sage/misc/nested_class.pyx:318: in sage.misc.nested_class.NestedClassMetaclass.__init__ (build/cythonized/sage/misc/nested_class.c:2314)
      ???
  E   KeyError: 'vector'
Last edited 11 months ago by Julian Rüth (previous) (diff)

comment:49 Changed 11 months ago by Julian Rüth

Another workaround is to just pretend that pytest is not installed:

$ cat no-pytest/pytest.py
raise ModuleNotFoundError
$ PYTHONPATH=no-pytest:$PYTHONPATH sage -tp ...

But I actually think we should make this a bit more convenient than having to hack the module search path.

Last edited 11 months ago by Julian Rüth (previous) (diff)

comment:50 Changed 10 months ago by Tobias Diez

I guess the issue with the modules is the same as #31924.

comment:51 Changed 10 months ago by Tobias Diez

And the exit code 5 issue should be fixed with #32892.

Note: See TracTickets for help on using tickets.