Opened 5 months ago

Closed 3 months ago

Last modified 3 months ago

#33233 closed enhancement (fixed)

sage -t --baseline-stats-path

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.6
Component: doctest framework Keywords:
Cc: chapoton, vbraun, gh-kliem, slabbe, gh-tobiasdiez, mjo, fbissey, arojas, tmonteil, vdelecroix Merged in:
Authors: Matthias Koeppe Reviewers: François Bissey
Report Upstream: N/A Work issues:
Branch: 180cc67 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

If the new option --baseline-stats-path PATH_TO_JSON_FILE is given, read failure information from there.

Tests that already are marked as failed there will be specially marked as baseline failures in the doctest report.

If there are only baseline failures, no new failures, then sage -t will exit with status code 0.

Follow-ups:

Change History (27)

comment:1 Changed 5 months ago by mkoeppe

  • Branch set to u/mkoeppe/sage__t___baseline_stats_path

comment:2 Changed 5 months ago by git

  • Commit set to 3551e196e0a9f65921320f41825f0bc7e6a6d785

Branch pushed to git repo; I updated commit sha1. New commits:

3551e19src/sage/doctest/control.py: Load base line stats

comment:3 Changed 5 months ago by git

  • Commit changed from 3551e196e0a9f65921320f41825f0bc7e6a6d785 to 09ce9e04c5e511c5dfe852c6d083361e5bd55f98

Branch pushed to git repo; I updated commit sha1. New commits:

09ce9e0src/sage/doctest/reporting.py: No error status for failures already seen in baseline

comment:4 Changed 5 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Status changed from new to needs_review

comment:5 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:6 Changed 5 months ago by mkoeppe

  • Cc gh-kliem added

comment:7 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:8 Changed 5 months ago by mkoeppe

  • Cc slabbe added

comment:9 Changed 5 months ago by mkoeppe

  • Cc gh-tobiasdiez added

comment:10 Changed 5 months ago by git

  • Commit changed from 09ce9e04c5e511c5dfe852c6d083361e5bd55f98 to 180cc672c0601110c3addaaa3d6516d92d5e75f9

Branch pushed to git repo; I updated commit sha1. New commits:

180cc67src/sage/doctest/reporting.py: Fix up fail_msg

comment:11 Changed 5 months ago by mkoeppe

  • Cc mjo fbissey added

comment:12 follow-up: Changed 5 months ago by gh-tobiasdiez

Is this already used in #33222? Because in this run https://github.com/mkoeppe/sage/runs/5060070979?check_suite_focus=true the integration tests fail and there is no "failed in baseline" message.

comment:13 follow-up: Changed 5 months ago by fbissey

This is an interesting idea. We can have a record of known failures in distros, in gentoo failing a test suite can prevent install. But with this, we could reduce it to really unexpected failures.

comment:14 Changed 5 months ago by mjo

I don't really like the idea but it's probably a necessary evil. We're the only project where it can take a month to get trivial bug fixes into the development branch that's used for testing all of the other branches.

And (for example) sage-9.5 was released with a number of broken tests having fixes waiting to be merged.

I wish we could address that problem, but will settle for this one.

comment:15 in reply to: ↑ 12 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

Is this already used in #33222? Because in this run https://github.com/mkoeppe/sage/runs/5060070979?check_suite_focus=true the integration tests fail and there is no "failed in baseline" message.

It's merged there but it doesn't end up being used because of a caching mistake

comment:16 in reply to: ↑ 13 ; follow-up: Changed 5 months ago by mkoeppe

Replying to fbissey:

This is an interesting idea. We can have a record of known failures in distros, in gentoo failing a test suite can prevent install. But with this, we could reduce it to really unexpected failures.

Note that currently the failures are recorded on the granularity of files, but it would be easy to make it more fine-grained to support this use case better.

I could imagine that could be better than carrying distro patches that patch out doctests

comment:17 in reply to: ↑ 16 Changed 5 months ago by fbissey

Replying to mkoeppe:

I could imagine that could be better than carrying distro patches that patch out doctests

Definitely. On the other hand, things have moved considerably in the last couple of years. Most distros doctests failures these days are from some version mis-match of dependencies or some corner case that should be fixed. I had a brief period of zero doctest failures in sage-on-gentoo and I am hoping to get there again in 9.6.

Looking at my own patches, there is very little about doctest fixing/suppressing left, and the one that are, are about sage's packaging system.

comment:18 Changed 5 months ago by mkoeppe

  • Cc arojas added

comment:19 Changed 4 months ago by mkoeppe

  • Cc tmonteil vdelecroix added

comment:20 Changed 4 months ago by fbissey

As it turns out, I will soon be moving from testing sage after install to testing as part of the building process. Testing after install should still be doable but I will be moving my focus to the part of the building process. It automatically does all the version of python I intend to support while testing after install has a number of steps involved.

So, I now have the choice of patching everything to get to zero doctests failures or this ticket. A funny thing is that I may need to patch the sage sources for #33141 when the patch really should belong to their individual packages.

There is also the problem of the regular transient failure(s) that only occur when doctesting in parallel and disappear once tested on their own - a long running and recurring issue, for which I only have this ticket to get out of.

comment:21 Changed 4 months ago by fbissey

I am currently testing this on some real case I "built". I have the following failures

sage -t --long --random-seed=140550856929631684337539526074563197630 sage/env.py  # 2 doctests failed [failed in baseline]
sage -t --long --random-seed=140550856929631684337539526074563197630 sage/interfaces/gap_workspace.py  # 2 doctests failed [failed in baseline]
sage -t --long --random-seed=140550856929631684337539526074563197630 sage/plot/plot3d/tachyon.py  # 1 doctest failed [failed in baseline]
sage -t --long --random-seed=140550856929631684337539526074563197630 sage/tests/startup.py  # 1 doctest failed [failed in baseline]
sage -t --long --random-seed=140550856929631684337539526074563197630 sage_docbuild/__init__.py  # 2 doctests failed
sage -t --long --random-seed=140550856929631684337539526074563197630 sage_setup/find.py  # 9 doctests failed [failed in baseline]

and using this json file to establish the baseline

{"sage.env": {"failed": true},"sage.interfaces.gap_workspace": {"failed": true},"sage.plot.plot3d.tachyon": {"failed": true},"sage.tests.startup": {"failed": true},"sage_docbuild.__init__": {"failed": true},"sage_setup.find": {"failed": true},"sage.tests.cmdline": {"failed": true}}

How do I correctly deal with sage_docbuild/__init__.py since adding "sage_docbuild.__init__": {"failed": true} didn't help? Am I missing something obvious?

comment:22 follow-up: Changed 3 months ago by fbissey

  • Reviewers set to François Bissey
  • Status changed from needs_review to positive_review

OK, answering my own question, it seems that "sage_docbuild": {"failed": true} works in that case. Although, I am fairly sure that it is over-reaching.

But otherwise the ticket does its job, so I will review it positively.

comment:23 Changed 3 months ago by mkoeppe

Thanks for the review!

comment:24 in reply to: ↑ 22 ; follow-up: Changed 3 months ago by mkoeppe

Replying to fbissey:

answering my own question, it seems that "sage_docbuild": {"failed": true} works in that case. Although, I am fairly sure that it is over-reaching.

It may just be what it uses as the name of the module in this case. Pretty sure it does not mark all modules in this package as failed.

comment:25 in reply to: ↑ 24 Changed 3 months ago by fbissey

Replying to mkoeppe:

Replying to fbissey:

answering my own question, it seems that "sage_docbuild": {"failed": true} works in that case. Although, I am fairly sure that it is over-reaching.

It may just be what it uses as the name of the module in this case. Pretty sure it does not mark all modules in this package as failed.

I am not about building a test case to check on that :) But I had come to that conclusion about the naming, you don't import __init__.py files, they get read when you import the parent module. At least that's how I thought of it.

comment:26 Changed 3 months ago by vbraun

  • Branch changed from u/mkoeppe/sage__t___baseline_stats_path to 180cc672c0601110c3addaaa3d6516d92d5e75f9
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:27 Changed 3 months ago by fbissey

  • Commit 180cc672c0601110c3addaaa3d6516d92d5e75f9 deleted

A curiosity that may need addressing

sage -t --long --random-seed=234497371435892995843174411690375861350 /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install/usr/lib/python3.10/site-packages/sage/doctest/forker.py  # 1 doctest failed
sage -t --long --random-seed=234497371435892995843174411690375861350 /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install/usr/lib/python3.10/site-packages/sage/env.py  # 2 doctests failed
sage -t --long --random-seed=234497371435892995843174411690375861350 /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install/usr/lib/python3.10/site-packages/sage/interfaces/gap_workspace.py  # 2 doctests failed
sage -t --long --random-seed=234497371435892995843174411690375861350 /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install/usr/lib/python3.10/site-packages/sage/plot/plot3d/tachyon.py  # 1 doctest failed
sage -t --long --random-seed=234497371435892995843174411690375861350 /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install/usr/lib/python3.10/site-packages/sage/tests/startup.py  # 1 doctest failed
sage -t --long --random-seed=234497371435892995843174411690375861350 /usr/lib/python3.10/site-packages/sage_docbuild/__init__.py  # 2 doctests failed [failed in baseline]
sage -t --long --random-seed=234497371435892995843174411690375861350 /usr/lib/python3.10/site-packages/sage_setup/clean.py  # 4 doctests failed [failed in baseline]
sage -t --long --random-seed=234497371435892995843174411690375861350 /usr/lib/python3.10/site-packages/sage_setup/find.py  # 9 doctests failed [failed in baseline]
sage -t --long --random-seed=234497371435892995843174411690375861350 /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install/usr/lib/python3.10/site-packages/sage/tests/cmdline.py  # 1 doctest failed

I probably will avoid running the test suite that way in the future. But I now can do automated testing in a venv (which is why a lot of stuff is prefixed with /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/install, that's the venv root), I ran the above test with --installed on the venv rather than with --all on the source - which is what I will focus on in the future. Everything in that test should have been caught by the provided baseline file but for some reason the stuff in the venv is not.

Note: See TracTickets for help on using tickets.