#23871 closed defect (duplicate)
Assorted doctest failures due to optional packages
Reported by: | mderickx | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-duplicate/invalid/wontfix |
Component: | misc | Keywords: | |
Cc: | Merged in: | ||
Authors: | Reviewers: | ||
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
The following tests fail on ubuntu 16.04 with all optional packages returned by optional_packages()
except gmp igraph libtheora mpfrcx python_igraph
installed. To be concrete this means:
4ti2 atlas autotools beautifulsoup benzene biopython bliss boost brian buckygen cbc ccache cmake coxeter3 cryptominisat csdp d3js database_cremona_ellcurve database_gap database_jones_numfield database_kohel database_mutation_class database_odlyzko_zeta database_pari database_stein_watkins database_stein_watkins_mini database_symbolic_data deformation dot2tex frobby gambit gap_jupyter gap_packages gdb giacpy_sage git_trac gmpy2 gp2c guppy latte_int libbraiding libhomfly libogg lidia lrslib mcqd meataxe mercurial mpi4py mpir nibabel normaliz nose notedown openssl ore_algebra pandoc_attributes pandocfilters pari_jupyter plantri pybtex pyflakes pynormaliz pyopenssl pysingular python2 pyx qepcad qhull rst2ipynb saclib scons singular_jupyter sip sirocco sqlalchemy tdlib termcap tides topcom trac
Note that on this system all test passed before installing the optional packages.
I also installed the pip package service_modules
using:
sage --pip install service_modules
since this greatly reduced the amount of failures.
Most of the errors were due to a failed installation of database_gap and gap_packages and could be solved using:
sage -f database_gap sage -f gap_packages
Then here are failing doctest due to the documentation not building. This can be solved by
apt-get graphviz
The doctests failures in
sage -t --long src/sage/tests/cmdline.py # 5 doctests failed
can be solved by
apt-get pandoc
This leaves us with:
sage -t --long src/sage/ext/memory.pyx # 1 doctest failed sage -t --long src/sage/rings/integer.pyx # 1 doctest failed
which was caused by gmp being picked up.
Change History (34)
comment:1 Changed 3 years ago by
- Component changed from PLEASE CHANGE to misc
comment:2 Changed 3 years ago by
- Description modified (diff)
comment:3 in reply to: ↑ description Changed 3 years ago by
comment:4 Changed 3 years ago by
Do you also have which package you installed when you first saw a failure?
comment:5 Changed 3 years ago by
- Description modified (diff)
comment:6 Changed 3 years ago by
Ok I added a list of all the optional packages installed. I do not have a list of when I first saw a failure since this was obtained by first installing as many packages as possible and then running all the tests. An automated script for doctesting in between each install would be nice indeed. I'll try to see if I can set that up and run it only on the above files. I created this ticket as an overview so that we at least know what is wrong. I certainly plan on working on making it more refined.
comment:7 Changed 3 years ago by
Would you mind posting the details of at least one failure? Given the number of packages there may be multiple sources but you have to start somewhere.
comment:8 Changed 3 years ago by
There are multiple failures there that should be caused by gap_packages
sage -t --long --warn-long 46.3 src/sage/tests/gap_packages.py sage -t --long --warn-long 46.3 src/sage/coding/guava.py
possibly a couple of others. I would have thought this particular package was well tested. I am wondering if there is a bad interaction with something else.
I would like to know the details of
sage -t --long --warn-long 46.3 src/sage/rings/integer.pyx sage -t --long --warn-long 46.3 src/sage/ext/memory.pyx
that couple is suspicious in that it would imply gmp which you said you didn't install. I wonder if it was pulled by something else.
comment:9 Changed 3 years ago by
Hi Francios,
Thank you for pointing at gmp. I did have gmp installed in the same sage install, but then later did
sage -i mpir
which removed gmp from the installed packages list returned by installed_packages()
. Could this be the cause?
comment:10 Changed 3 years ago by
- Description modified (diff)
comment:11 follow-ups: ↓ 14 ↓ 26 Changed 3 years ago by
Could be something funny happening there. Typically gmp will lead to this
File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper Failed example: 1 << (2^60) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate ... bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module> Integer(1) << (Integer(2)**Integer(60)) File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39280) return (<Integer>x)._shift_helper(y, 1) File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39024) sig_on() RuntimeError: Aborted
and
File "/usr/lib64/python2.7/site-packages/sage/ext/memory.pyx", line 9, in sage.ext.memory Failed example: 2^(2^63-2) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate 1152921504606847008 bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.ext.memory[0]>", line 1, in <module> Integer(2)**(Integer(2)**Integer(63)-Integer(2)) File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:14183) sig_on() RuntimeError: Aborted
comment:12 Changed 3 years ago by
I just did:
sage -f database_gap sage -f gap_packages
and this fixed almost all of the failures. For some reason something went wrong with these two packages.
comment:13 Changed 3 years ago by
- Description modified (diff)
comment:14 in reply to: ↑ 11 ; follow-up: ↓ 15 Changed 3 years ago by
Replying to fbissey:
Could be something funny happening there. Typically gmp will lead to this
File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper ... RuntimeError: Aborted
These two errors were exactly the two errors I got. So I guess that settles this. Is there a ticket for those?
comment:15 in reply to: ↑ 14 Changed 3 years ago by
Replying to mderickx:
Replying to fbissey:
Could be something funny happening there. Typically gmp will lead to this
File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper ... RuntimeError: AbortedThese two errors were exactly the two errors I got. So I guess that settles this. Is there a ticket for those?
Yes, possibly in a won't fix state though. These are differences in error messages between mpir and gmp. We are more or less living with them. They are well know of sage-on-distro people since we all use gmp for "reasons" (including sanity).
comment:16 follow-up: ↓ 18 Changed 3 years ago by
I am not shocked by the outcome of the gap stuff. Although I must say I wouldn't have known for a few of those. Now that you have a single test failing, can we have details?
comment:17 Changed 3 years ago by
- Description modified (diff)
- Priority changed from blocker to major
comment:18 in reply to: ↑ 16 Changed 3 years ago by
Replying to fbissey:
I am not shocked by the outcome of the gap stuff. Although I must say I wouldn't have known for a few of those. Now that you have a single test failing, can we have details?
Sorry the detail is gone by subsequent doctests. A lot of the messages were saying that I should install database_gap or gap_packages, which I already had installed. I want to try to make this entire build again from a clean sage install and see if I can reproduce the errors. This will take some time though.
In the meanwhile a lot of thanks for helping figuring all the errors out. This was a great help in reliably installing a lot of optional packages at the same time.
comment:19 Changed 3 years ago by
On my sage4
patchbot, I also saw the whole bunch of failures that Maarten initially posted on this ticket. So there must be a real issue here. Since there are a lot of MemoryError
in the failed tests, this is almost certainly due to #23748. So there must be some optional package which has high memory usage (possibly even if it's not actually used). This could very well be gap_packages
but then I do not understand why reinstalling gap_packages
would help.
My strategy is now to install those packages one by one and run a few selected doctests every time to see which package is responsible for which failures.
comment:20 Changed 3 years ago by
At least some of the failures are caused by cbc
.
comment:21 Changed 3 years ago by
Is cbc failing as in #22006 or is it something else?
comment:22 Changed 3 years ago by
No, I don't mean #22006. I mean some of the failures that you originally posted in the ticket description (and which are not there anymore).
comment:23 Changed 3 years ago by
- Description modified (diff)
comment:24 in reply to: ↑ description ; follow-up: ↓ 25 Changed 3 years ago by
Replying to mderickx:
This leaves us with:
sage -t --long src/sage/ext/memory.pyx # 1 doctest failed sage -t --long src/sage/rings/integer.pyx # 1 doctest failedwhich was caused by gmp being picked up.
Details please. What do you mean with "caused by gmp being picked up"? The Sage package gmp
is not in your list of optional packages.
comment:25 in reply to: ↑ 24 Changed 3 years ago by
Replying to jdemeyer:
Replying to mderickx:
This leaves us with:
sage -t --long src/sage/ext/memory.pyx # 1 doctest failed sage -t --long src/sage/rings/integer.pyx # 1 doctest failedwhich was caused by gmp being picked up.
Details please. What do you mean with "caused by gmp being picked up"? The Sage package
gmp
is not in your list of optional packages.
comment:26 in reply to: ↑ 11 Changed 3 years ago by
Replying to fbissey:
Could be something funny happening there. Typically gmp will lead to this
File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper Failed example: 1 << (2^60) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate ... bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module> Integer(1) << (Integer(2)**Integer(60)) File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39280) return (<Integer>x)._shift_helper(y, 1) File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39024) sig_on() RuntimeError: Abortedand
File "/usr/lib64/python2.7/site-packages/sage/ext/memory.pyx", line 9, in sage.ext.memory Failed example: 2^(2^63-2) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate 1152921504606847008 bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.ext.memory[0]>", line 1, in <module> Integer(2)**(Integer(2)**Integer(63)-Integer(2)) File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:14183) sig_on() RuntimeError: Aborted
I assume that this is with GMP 6 on Gentoo, right? Because Sage still ships GMP 5.1.3 (see #19706).
comment:27 Changed 3 years ago by
As far as I remember both GMP
5 and 6 will give you that message. While on the subject, the problem that stopped #19706 have now gone so we could finish it off.
comment:28 Changed 3 years ago by
#23713 should deal with cbc
and cryptominisat
comment:29 Changed 3 years ago by
- Milestone changed from sage-8.1 to sage-duplicate/invalid/wontfix
- Status changed from new to needs_review
Thanks, I think this ticket can be closed as duplicate since the issues turned out to be flukes or are fixed by other tickets.
comment:30 Changed 3 years ago by
- Status changed from needs_review to positive_review
comment:31 Changed 3 years ago by
- Resolution set to duplicate
- Status changed from positive_review to closed
comment:32 Changed 3 years ago by
I believe that I have now created a blocker ticket for each of the issues that I could reproduce on a particular x86_64 Linux system.
I could not reproduce the issue with gap_packages
and database_gap
: I have no idea what went wrong there. Nor could I reproduce the issue with pandoc
, but that's probably because pandoc
is actually installed on the system where I tested it.
comment:33 Changed 3 years ago by
I should also add that I tested all optional and pip packages except for gmp
, atlas
and mpi4py
. So I included a few packages that you omitted.
comment:34 Changed 3 years ago by
Hi Jeroen thanks for putting so much energy in these optional packages,
I recreated the pandoc problem and created #23985 for this with a clear traceback and explanation of why it occurs. I think the gap_packages
and database_gap
errors can be regarded as a mistake in installing all these optional packages from my side, because I cannot reproduce it either on the machine where it went wrong.
Replying to mderickx:
Instead of listing the packages which are not installed, can you instead list the packages which are installed?