Opened 3 years ago

Last modified 3 months ago

#28936 new task

Meta-ticket: Adopt mainstream Python testing/linting infrastructure: tox, pytest, ..., describe in Developer's Guide

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.7
Component: doctest framework Keywords:
Cc: jdemeyer, dimpase, embray, vdelecroix, chapoton, tscrim, jmantysalo, gh-tobiasdiez, gh-kliem, nthiery, tornaria Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

This metaticket proposes to use mainstream Python testing and linting infrastructure for Sage -- by adding configuration files and documentation.

tox:

  • #28987 Add src/tox.ini: As an alternative interface to running the sage doctester.
  • #30467 src/tox.ini: Check patchbot plugin patterns
  • #30410 (Command "sage -tox")
  • #30416 (Make tox a standard package)
  • #30453 Document "sage -tox"
  • #30411 (src/tox.ini: Add environment pyright)
  • #30448 (src/tox.ini: Add validation of rst files and docstrings via a quick docbuild)

pytest:

  • #31003 Add minimal pytest configuration
  • #31103 sage.numerical.backends: Replace use of TestSuite by pytest
  • #30362 Add symplectic structures
  • #32975 Improve doctest interaction with pytest
  • #33549 Rename test_... functions to avoid the naming scheme reserved by pytest
  • #31924 sage -t: Do not run pytest on individual Python files unless they match the pytest file pattern
  • #33572 sage -pytest
  • #33560 pytest: ignore cython files
  • #33561 Rename test_... classes to avoid the naming scheme reserved by pytest
  • #33550 Preparse sage files before passing them to pytest
  • #31123 Document pytest
  • #33521 doctesting with pytest fails on system install of sage
  • #33531 pytest in sage is broken, please make it really optional
  • #31110 Make pytest a standard package
  • #33546 Replace custom Sage doctest discovery by pytest (using pytest_collect_file)
  • #30738 Replace custom Sage unit test discovery by pytest
  • #31104 sage.geometry.polyhedron: Replace use of TestSuite by pytest

Other tickets:

  • #30452 ("sage -startuptime" should accept file names too)
  • #30361 (Add pyright config and linting documentation; dup: #17992)
  • #33232 (sage -t: Write report in JUnitXML format)

Documentation tickets:

  • #29784 Update section "Sage Development Process" in Developer's Guide
  • #29520: Mention .lgtm.yml in the Developer's Guide

Infrastructure tickets:

See also:

  • Meta-ticket #29060: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages
  • Generalize sage.numerical.backends.logging_backend.LoggingBackend to help generate _test methods
  • #28998: Add script-type check_dependencies package that installs tox, mock, pytest etc.

Change History (44)

comment:1 Changed 3 years ago by mkoeppe

  • Cc vdelecroix added

comment:2 Changed 3 years ago by mkoeppe

  • Milestone changed from sage-9.0 to sage-9.1
  • Summary changed from Create tox plugin for running the sage doctester to Create tox / pytest plugins for running the sage doctester

comment:3 in reply to: ↑ description ; follow-up: Changed 3 years ago by embray

Replying to mkoeppe:

tox seems to be the preferred interface for running tests - see for example https://github.com/pypa/setuptools/issues/931 (deprecation of setup.py test).

I don't know what you mean by "preferred interface" here. There isn't really any such thing as a preferred interface. It's just being proposed here as a recommendation over setup.py test. Sage doesn't use setup.py test anyways so there's nothing that needs to be "replaced" with a different interface if/when setup.py goes away.

The main advantage to tox is the ability to run a package's tests across multiple runtime environments (esp. different Python versions). This isn't really possible currently with Sage; it would actually be very difficult to use tox in any sensible manner with Sage.

If someone wants to do this anyways I don't see the harm really, but I'm not sure what problem it would be solving.

comment:4 Changed 3 years ago by mkoeppe

  • Type changed from enhancement to task

A first step: #28987

comment:5 in reply to: ↑ 3 Changed 3 years ago by mkoeppe

Replying to embray:

Replying to mkoeppe: The main advantage to tox is the ability to run a package's tests across multiple runtime environments (esp. different Python versions). This isn't really possible currently with Sage; it would actually be very difficult to use tox in any sensible manner with Sage.

FWIW, I managed to use tox on top of the sage installation, see https://github.com/mkoeppe/sage-numerical-backends-gurobi/blob/master/tox.ini: The trick is to run tox with sitepackages = true and an install_command that installs into the user scheme. With this trick, after all, no new plugin for tox is needed to handle the sage environment.

comment:6 Changed 3 years ago by mkoeppe

  • Cc chapoton added
  • Description modified (diff)
  • Summary changed from Create tox / pytest plugins for running the sage doctester to Meta-ticket: Adopt mainstream Python testing/linting infrastructure

I have broadened the ticket description.

comment:7 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

pushing these forward to 9.2

comment:9 Changed 2 years ago by mkoeppe

  • Cc tscrim jmantysalo added
  • Description modified (diff)
  • Summary changed from Meta-ticket: Adopt mainstream Python testing/linting infrastructure to Meta-ticket: Adopt mainstream Python testing/linting infrastructure, describe in Developer's Guide

comment:10 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 2 years ago by mkoeppe

  • Cc gh-tobiasdiez added
  • Description modified (diff)

comment:12 Changed 2 years ago by mkoeppe

  • Cc gh-kliem added
  • Description modified (diff)

comment:13 Changed 2 years ago by gh-tobiasdiez

  • Description modified (diff)

comment:14 Changed 2 years ago by gh-tobiasdiez

  • Description modified (diff)

comment:15 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:16 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:17 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:18 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:19 Changed 2 years ago by gh-tobiasdiez

  • Description modified (diff)

comment:20 Changed 2 years ago by gh-tobiasdiez

  • Description modified (diff)

comment:21 Changed 23 months ago by mkoeppe

  • Description modified (diff)

comment:22 Changed 23 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:23 Changed 22 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:24 Changed 22 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:25 Changed 20 months ago by mkoeppe

  • Cc nthiery added

comment:26 Changed 20 months ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 20 months ago by mkoeppe

  • Description modified (diff)

comment:28 Changed 20 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Meta-ticket: Adopt mainstream Python testing/linting infrastructure, describe in Developer's Guide to Meta-ticket: Adopt mainstream Python testing/linting infrastructure: tox, pytest, ..., describe in Developer's Guide

comment:29 Changed 15 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Moving to 9.4, as 9.3 has been released.

comment:30 Changed 15 months ago by mkoeppe

  • Description modified (diff)

comment:31 Changed 15 months ago by gh-tobiasdiez

What do you think about using pytest-cov for generating the test coverage report? The results can then be uploded to codecov as part of the build process (github action). This gives a nice overview of the actual test coverage (i.e. code covered by tests) and could replace the sage-coverage scripts in the future.

comment:32 Changed 15 months ago by mkoeppe

+1 on investigating this. Myself I have no experience with the available tools (including the sage-coverage scripts even though I have fixed them up at least once). As usual, it's best to build a prototype and then demonstrate to the sage-devel community that it is useful.

comment:33 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:34 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:35 Changed 5 months ago by mkoeppe

  • Cc tornaria added
  • Description modified (diff)

comment:36 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:38 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:39 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:40 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:41 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:42 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:43 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:44 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-9.7
Note: See TracTickets for help on using tickets.