## #23871 closed defect (duplicate)

# Assorted doctest failures due to optional packages

Reported by: | Maarten Derickx | Owned by: | |
---|---|---|---|

Priority: | major | Milestone: | sage-duplicate/invalid/wontfix |

Component: | misc | Keywords: | |

Cc: | Merged in: | ||

Authors: | Reviewers: | ||

Report Upstream: | N/A | Work issues: | |

Branch: | Commit: | ||

Dependencies: | Stopgaps: |

### Description (last modified by )

The following tests fail on ubuntu 16.04 with all optional packages returned by `optional_packages()`

except `gmp igraph libtheora mpfrcx python_igraph`

installed. To be concrete this means:

4ti2 atlas autotools beautifulsoup benzene biopython bliss boost brian buckygen cbc ccache cmake coxeter3 cryptominisat csdp d3js database_cremona_ellcurve database_gap database_jones_numfield database_kohel database_mutation_class database_odlyzko_zeta database_pari database_stein_watkins database_stein_watkins_mini database_symbolic_data deformation dot2tex frobby gambit gap_jupyter gap_packages gdb giacpy_sage git_trac gmpy2 gp2c guppy latte_int libbraiding libhomfly libogg lidia lrslib mcqd meataxe mercurial mpi4py mpir nibabel normaliz nose notedown openssl ore_algebra pandoc_attributes pandocfilters pari_jupyter plantri pybtex pyflakes pynormaliz pyopenssl pysingular python2 pyx qepcad qhull rst2ipynb saclib scons singular_jupyter sip sirocco sqlalchemy tdlib termcap tides topcom trac

Note that on this system all test passed before installing the optional packages.

I also installed the pip package `service_modules`

using:

sage --pip install service_modules

since this greatly reduced the amount of failures.

Most of the errors were due to a failed installation of database_gap and gap_packages and could be solved using:

sage -f database_gap sage -f gap_packages

Then here are failing doctest due to the documentation not building. This can be solved by

apt-get graphviz

The doctests failures in

sage -t --long src/sage/tests/cmdline.py # 5 doctests failed

can be solved by

apt-get pandoc

This leaves us with:

sage -t --long src/sage/ext/memory.pyx # 1 doctest failed sage -t --long src/sage/rings/integer.pyx # 1 doctest failed

which was caused by gmp being picked up.

### Change History (34)

### comment:1 Changed 5 years ago by

Component: | PLEASE CHANGE → misc |
---|

### comment:2 Changed 5 years ago by

Description: | modified (diff) |
---|

### comment:3 Changed 5 years ago by

### comment:4 Changed 5 years ago by

Do you also have which package you installed when you first saw a failure?

### comment:5 Changed 5 years ago by

Description: | modified (diff) |
---|

### comment:6 Changed 5 years ago by

Ok I added a list of all the optional packages installed. I do not have a list of when I first saw a failure since this was obtained by first installing as many packages as possible and then running all the tests. An automated script for doctesting in between each install would be nice indeed. I'll try to see if I can set that up and run it only on the above files. I created this ticket as an overview so that we at least know what is wrong. I certainly plan on working on making it more refined.

### comment:7 Changed 5 years ago by

Would you mind posting the details of at least one failure? Given the number of packages there may be multiple sources but you have to start somewhere.

### comment:8 Changed 5 years ago by

There are multiple failures there that should be caused by gap_packages

sage -t --long --warn-long 46.3 src/sage/tests/gap_packages.py sage -t --long --warn-long 46.3 src/sage/coding/guava.py

possibly a couple of others. I would have thought this particular package was well tested. I am wondering if there is a bad interaction with something else.

I would like to know the details of

sage -t --long --warn-long 46.3 src/sage/rings/integer.pyx sage -t --long --warn-long 46.3 src/sage/ext/memory.pyx

that couple is suspicious in that it would imply gmp which you said you didn't install. I wonder if it was pulled by something else.

### comment:9 Changed 5 years ago by

Hi Francios,

Thank you for pointing at gmp. I did have gmp installed in the same sage install, but then later did

sage -i mpir

which removed gmp from the installed packages list returned by `installed_packages()`

. Could this be the cause?

### comment:10 Changed 5 years ago by

Description: | modified (diff) |
---|

### comment:11 follow-ups: 14 26 Changed 5 years ago by

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper Failed example: 1 << (2^60) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate ... bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module> Integer(1) << (Integer(2)**Integer(60)) File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39280) return (<Integer>x)._shift_helper(y, 1) File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39024) sig_on() RuntimeError: Aborted

and

File "/usr/lib64/python2.7/site-packages/sage/ext/memory.pyx", line 9, in sage.ext.memory Failed example: 2^(2^63-2) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate 1152921504606847008 bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.ext.memory[0]>", line 1, in <module> Integer(2)**(Integer(2)**Integer(63)-Integer(2)) File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:14183) sig_on() RuntimeError: Aborted

### comment:12 Changed 5 years ago by

I just did:

sage -f database_gap sage -f gap_packages

and this fixed almost all of the failures. For some reason something went wrong with these two packages.

### comment:13 Changed 5 years ago by

Description: | modified (diff) |
---|

### comment:14 follow-up: 15 Changed 5 years ago by

Replying to fbissey:

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper ... RuntimeError: Aborted

These two errors were exactly the two errors I got. So I guess that settles this. Is there a ticket for those?

### comment:15 Changed 5 years ago by

Replying to mderickx:

Replying to fbissey:

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper ... RuntimeError: AbortedThese two errors were exactly the two errors I got. So I guess that settles this. Is there a ticket for those?

Yes, possibly in a won't fix state though. These are differences in error messages between mpir and gmp. We are more or less living with them. They are well know of sage-on-distro people since we all use gmp for "reasons" (including sanity).

### comment:16 follow-up: 18 Changed 5 years ago by

I am not shocked by the outcome of the gap stuff. Although I must say I wouldn't have known for a few of those. Now that you have a single test failing, can we have details?

### comment:17 Changed 5 years ago by

Description: | modified (diff) |
---|---|

Priority: | blocker → major |

### comment:18 Changed 5 years ago by

Replying to fbissey:

I am not shocked by the outcome of the gap stuff. Although I must say I wouldn't have known for a few of those. Now that you have a single test failing, can we have details?

Sorry the detail is gone by subsequent doctests. A lot of the messages were saying that I should install database_gap or gap_packages, which I already had installed. I want to try to make this entire build again from a clean sage install and see if I can reproduce the errors. This will take some time though.

In the meanwhile a lot of thanks for helping figuring all the errors out. This was a great help in reliably installing a lot of optional packages at the same time.

### comment:19 Changed 5 years ago by

On my `sage4`

patchbot, I also saw the whole bunch of failures that Maarten initially posted on this ticket. So there must be a real issue here. Since there are a lot of `MemoryError`

in the failed tests, this is almost certainly due to #23748. So there must be some optional package which has high memory usage (possibly even if it's not actually used). This could very well be `gap_packages`

but then I do not understand why reinstalling `gap_packages`

would help.

My strategy is now to install those packages one by one and run a few selected doctests every time to see which package is responsible for which failures.

### comment:22 Changed 5 years ago by

No, I don't mean #22006. I mean some of the failures that you originally posted in the ticket description (and which are not there anymore).

### comment:23 Changed 5 years ago by

Description: | modified (diff) |
---|

### comment:24 follow-up: 25 Changed 5 years ago by

Replying to mderickx:

This leaves us with:

sage -t --long src/sage/ext/memory.pyx # 1 doctest failed sage -t --long src/sage/rings/integer.pyx # 1 doctest failedwhich was caused by gmp being picked up.

Details please. What do you mean with "caused by gmp being picked up"? The Sage package `gmp`

is not in your list of optional packages.

### comment:25 Changed 5 years ago by

Replying to jdemeyer:

Replying to mderickx:

This leaves us with:

sage -t --long src/sage/ext/memory.pyx # 1 doctest failed sage -t --long src/sage/rings/integer.pyx # 1 doctest failedwhich was caused by gmp being picked up.

Details please. What do you mean with "caused by gmp being picked up"? The Sage package

`gmp`

is not in your list of optional packages.

### comment:26 Changed 5 years ago by

Replying to fbissey:

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper Failed example: 1 << (2^60) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate ... bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module> Integer(1) << (Integer(2)**Integer(60)) File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39280) return (<Integer>x)._shift_helper(y, 1) File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39024) sig_on() RuntimeError: Abortedand

File "/usr/lib64/python2.7/site-packages/sage/ext/memory.pyx", line 9, in sage.ext.memory Failed example: 2^(2^63-2) Expected: Traceback (most recent call last): ... MemoryError: failed to allocate 1152921504606847008 bytes Got: gmp: overflow in mpz type Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute exec(compiled, globs) File "<doctest sage.ext.memory[0]>", line 1, in <module> Integer(2)**(Integer(2)**Integer(63)-Integer(2)) File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:14183) sig_on() RuntimeError: Aborted

I assume that this is with GMP 6 on Gentoo, right? Because Sage still ships GMP 5.1.3 (see #19706).

### comment:27 Changed 5 years ago by

As far as I remember both `GMP`

5 and 6 will give you that message. While on the subject, the problem that stopped #19706 have now gone so we could finish it off.

### comment:29 Changed 5 years ago by

Milestone: | sage-8.1 → sage-duplicate/invalid/wontfix |
---|---|

Status: | new → needs_review |

Thanks, I think this ticket can be closed as duplicate since the issues turned out to be flukes or are fixed by other tickets.

### comment:30 Changed 5 years ago by

Status: | needs_review → positive_review |
---|

### comment:31 Changed 5 years ago by

Resolution: | → duplicate |
---|---|

Status: | positive_review → closed |

### comment:32 Changed 5 years ago by

I believe that I have now created a blocker ticket for each of the issues that I could reproduce on a particular x86_64 Linux system.

I could not reproduce the issue with `gap_packages`

and `database_gap`

: I have no idea what went wrong there. Nor could I reproduce the issue with `pandoc`

, but that's probably because `pandoc`

is actually installed on the system where I tested it.

### comment:33 Changed 5 years ago by

I should also add that I tested *all* optional and pip packages except for `gmp`

, `atlas`

and `mpi4py`

. So I included a few packages that you omitted.

### comment:34 Changed 5 years ago by

Hi Jeroen thanks for putting so much energy in these optional packages,

I recreated the pandoc problem and created #23985 for this with a clear traceback and explanation of why it occurs. I think the `gap_packages`

and `database_gap`

errors can be regarded as a mistake in installing all these optional packages from my side, because I cannot reproduce it either on the machine where it went wrong.

**Note:**See TracTickets for help on using tickets.

Replying to mderickx:

Instead of listing the packages which are not installed, can you instead list the packages which are installed?