Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#23871 closed defect (duplicate)

Assorted doctest failures due to optional packages

Reported by: mderickx Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: misc Keywords:
Cc: Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

The following tests fail on ubuntu 16.04 with all optional packages returned by optional_packages() except gmp igraph libtheora mpfrcx python_igraph installed. To be concrete this means:

4ti2 atlas autotools beautifulsoup benzene biopython bliss boost brian buckygen cbc ccache cmake coxeter3 cryptominisat csdp d3js database_cremona_ellcurve database_gap database_jones_numfield database_kohel database_mutation_class database_odlyzko_zeta database_pari database_stein_watkins database_stein_watkins_mini database_symbolic_data deformation dot2tex frobby gambit gap_jupyter gap_packages gdb giacpy_sage git_trac gmpy2 gp2c guppy latte_int libbraiding libhomfly libogg lidia lrslib mcqd meataxe mercurial mpi4py mpir nibabel normaliz nose notedown openssl ore_algebra pandoc_attributes pandocfilters pari_jupyter plantri pybtex pyflakes pynormaliz pyopenssl pysingular python2 pyx qepcad qhull rst2ipynb saclib scons singular_jupyter sip sirocco sqlalchemy tdlib termcap tides topcom trac

Note that on this system all test passed before installing the optional packages.

I also installed the pip package service_modules using:

sage --pip install service_modules

since this greatly reduced the amount of failures.

Most of the errors were due to a failed installation of database_gap and gap_packages and could be solved using:

sage -f database_gap
sage -f gap_packages

Then here are failing doctest due to the documentation not building. This can be solved by

apt-get graphviz

The doctests failures in

sage -t --long src/sage/tests/cmdline.py  # 5 doctests failed

can be solved by

apt-get pandoc

This leaves us with:

sage -t --long src/sage/ext/memory.pyx  # 1 doctest failed
sage -t --long src/sage/rings/integer.pyx  # 1 doctest failed

which was caused by gmp being picked up.

Change History (34)

comment:1 Changed 2 years ago by mderickx

  • Component changed from PLEASE CHANGE to misc

comment:2 Changed 2 years ago by mderickx

  • Description modified (diff)

comment:3 in reply to: ↑ description Changed 2 years ago by jdemeyer

Replying to mderickx:

The following tests fail on ubuntu 16.04 with all optional packages except gmp igraph libtheora mpfrcx python_igraph

Instead of listing the packages which are not installed, can you instead list the packages which are installed?

comment:4 Changed 2 years ago by tscrim

Do you also have which package you installed when you first saw a failure?

comment:5 Changed 2 years ago by mderickx

  • Description modified (diff)

comment:6 Changed 2 years ago by mderickx

Ok I added a list of all the optional packages installed. I do not have a list of when I first saw a failure since this was obtained by first installing as many packages as possible and then running all the tests. An automated script for doctesting in between each install would be nice indeed. I'll try to see if I can set that up and run it only on the above files. I created this ticket as an overview so that we at least know what is wrong. I certainly plan on working on making it more refined.

Last edited 2 years ago by mderickx (previous) (diff)

comment:7 Changed 2 years ago by fbissey

Would you mind posting the details of at least one failure? Given the number of packages there may be multiple sources but you have to start somewhere.

comment:8 Changed 2 years ago by fbissey

There are multiple failures there that should be caused by gap_packages

sage -t --long --warn-long 46.3 src/sage/tests/gap_packages.py
sage -t --long --warn-long 46.3 src/sage/coding/guava.py

possibly a couple of others. I would have thought this particular package was well tested. I am wondering if there is a bad interaction with something else.

I would like to know the details of

sage -t --long --warn-long 46.3 src/sage/rings/integer.pyx
sage -t --long --warn-long 46.3 src/sage/ext/memory.pyx

that couple is suspicious in that it would imply gmp which you said you didn't install. I wonder if it was pulled by something else.

comment:9 Changed 2 years ago by mderickx

Hi Francios,

Thank you for pointing at gmp. I did have gmp installed in the same sage install, but then later did

sage -i mpir

which removed gmp from the installed packages list returned by installed_packages(). Could this be the cause?

comment:10 Changed 2 years ago by mderickx

  • Description modified (diff)

comment:11 follow-ups: Changed 2 years ago by fbissey

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper
Failed example:
    1 << (2^60)
Expected:
    Traceback (most recent call last):
    ...
    MemoryError: failed to allocate ... bytes   
Got:
    gmp: overflow in mpz type
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module>
        Integer(1) << (Integer(2)**Integer(60))
      File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39280)
        return (<Integer>x)._shift_helper(y, 1)
      File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39024)
        sig_on()
    RuntimeError: Aborted

and

File "/usr/lib64/python2.7/site-packages/sage/ext/memory.pyx", line 9, in sage.ext.memory
Failed example:
    2^(2^63-2)
Expected:
    Traceback (most recent call last):
    ...
    MemoryError: failed to allocate 1152921504606847008 bytes  
Got:
    gmp: overflow in mpz type
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.ext.memory[0]>", line 1, in <module>
        Integer(2)**(Integer(2)**Integer(63)-Integer(2))
      File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:14183)
        sig_on()
    RuntimeError: Aborted

comment:12 Changed 2 years ago by mderickx

I just did:

sage -f database_gap
sage -f gap_packages

and this fixed almost all of the failures. For some reason something went wrong with these two packages.

comment:13 Changed 2 years ago by mderickx

  • Description modified (diff)

comment:14 in reply to: ↑ 11 ; follow-up: Changed 2 years ago by mderickx

Replying to fbissey:

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper
...
    RuntimeError: Aborted

These two errors were exactly the two errors I got. So I guess that settles this. Is there a ticket for those?

comment:15 in reply to: ↑ 14 Changed 2 years ago by fbissey

Replying to mderickx:

Replying to fbissey:

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper
...
    RuntimeError: Aborted

These two errors were exactly the two errors I got. So I guess that settles this. Is there a ticket for those?

Yes, possibly in a won't fix state though. These are differences in error messages between mpir and gmp. We are more or less living with them. They are well know of sage-on-distro people since we all use gmp for "reasons" (including sanity).

comment:16 follow-up: Changed 2 years ago by fbissey

I am not shocked by the outcome of the gap stuff. Although I must say I wouldn't have known for a few of those. Now that you have a single test failing, can we have details?

comment:17 Changed 2 years ago by mderickx

  • Description modified (diff)
  • Priority changed from blocker to major

comment:18 in reply to: ↑ 16 Changed 2 years ago by mderickx

Replying to fbissey:

I am not shocked by the outcome of the gap stuff. Although I must say I wouldn't have known for a few of those. Now that you have a single test failing, can we have details?

Sorry the detail is gone by subsequent doctests. A lot of the messages were saying that I should install database_gap or gap_packages, which I already had installed. I want to try to make this entire build again from a clean sage install and see if I can reproduce the errors. This will take some time though.

In the meanwhile a lot of thanks for helping figuring all the errors out. This was a great help in reliably installing a lot of optional packages at the same time.

comment:19 Changed 2 years ago by jdemeyer

On my sage4 patchbot, I also saw the whole bunch of failures that Maarten initially posted on this ticket. So there must be a real issue here. Since there are a lot of MemoryError in the failed tests, this is almost certainly due to #23748. So there must be some optional package which has high memory usage (possibly even if it's not actually used). This could very well be gap_packages but then I do not understand why reinstalling gap_packages would help.

My strategy is now to install those packages one by one and run a few selected doctests every time to see which package is responsible for which failures.

comment:20 Changed 2 years ago by jdemeyer

At least some of the failures are caused by cbc.

comment:21 Changed 2 years ago by mderickx

Is cbc failing as in #22006 or is it something else?

comment:22 Changed 2 years ago by jdemeyer

No, I don't mean #22006. I mean some of the failures that you originally posted in the ticket description (and which are not there anymore).

comment:23 Changed 2 years ago by jdemeyer

  • Description modified (diff)

comment:24 in reply to: ↑ description ; follow-up: Changed 2 years ago by jdemeyer

Replying to mderickx:

This leaves us with:

sage -t --long src/sage/ext/memory.pyx  # 1 doctest failed
sage -t --long src/sage/rings/integer.pyx  # 1 doctest failed

which was caused by gmp being picked up.

Details please. What do you mean with "caused by gmp being picked up"? The Sage package gmp is not in your list of optional packages.

comment:25 in reply to: ↑ 24 Changed 2 years ago by fbissey

Replying to jdemeyer:

Replying to mderickx:

This leaves us with:

sage -t --long src/sage/ext/memory.pyx  # 1 doctest failed
sage -t --long src/sage/rings/integer.pyx  # 1 doctest failed

which was caused by gmp being picked up.

Details please. What do you mean with "caused by gmp being picked up"? The Sage package gmp is not in your list of optional packages.

See https://trac.sagemath.org/ticket/23871#comment:9

comment:26 in reply to: ↑ 11 Changed 2 years ago by jdemeyer

Replying to fbissey:

Could be something funny happening there. Typically gmp will lead to this

File "/usr/lib64/python2.7/site-packages/sage/rings/integer.pyx", line 6099, in sage.rings.integer.Integer._shift_helper
Failed example:
    1 << (2^60)
Expected:
    Traceback (most recent call last):
    ...
    MemoryError: failed to allocate ... bytes   
Got:
    gmp: overflow in mpz type
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.integer.Integer._shift_helper[8]>", line 1, in <module>
        Integer(1) << (Integer(2)**Integer(60))
      File "sage/rings/integer.pyx", line 6177, in sage.rings.integer.Integer.__lshift__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39280)
        return (<Integer>x)._shift_helper(y, 1)
      File "sage/rings/integer.pyx", line 6138, in sage.rings.integer.Integer._shift_helper (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:39024)
        sig_on()
    RuntimeError: Aborted

and

File "/usr/lib64/python2.7/site-packages/sage/ext/memory.pyx", line 9, in sage.ext.memory
Failed example:
    2^(2^63-2)
Expected:
    Traceback (most recent call last):
    ...
    MemoryError: failed to allocate 1152921504606847008 bytes  
Got:
    gmp: overflow in mpz type
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.ext.memory[0]>", line 1, in <module>
        Integer(2)**(Integer(2)**Integer(63)-Integer(2))
      File "sage/rings/integer.pyx", line 2067, in sage.rings.integer.Integer.__pow__ (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/rings/integer.c:14183)
        sig_on()
    RuntimeError: Aborted

I assume that this is with GMP 6 on Gentoo, right? Because Sage still ships GMP 5.1.3 (see #19706).

comment:27 Changed 2 years ago by fbissey

As far as I remember both GMP 5 and 6 will give you that message. While on the subject, the problem that stopped #19706 have now gone so we could finish it off.

Last edited 2 years ago by fbissey (previous) (diff)

comment:28 Changed 2 years ago by jdemeyer

#23713 should deal with cbc and cryptominisat

comment:29 Changed 2 years ago by mderickx

  • Milestone changed from sage-8.1 to sage-duplicate/invalid/wontfix
  • Status changed from new to needs_review

Thanks, I think this ticket can be closed as duplicate since the issues turned out to be flukes or are fixed by other tickets.

comment:30 Changed 2 years ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:31 Changed 2 years ago by mderickx

  • Resolution set to duplicate
  • Status changed from positive_review to closed

comment:32 Changed 2 years ago by jdemeyer

I believe that I have now created a blocker ticket for each of the issues that I could reproduce on a particular x86_64 Linux system.

I could not reproduce the issue with gap_packages and database_gap: I have no idea what went wrong there. Nor could I reproduce the issue with pandoc, but that's probably because pandoc is actually installed on the system where I tested it.

comment:33 Changed 2 years ago by jdemeyer

I should also add that I tested all optional and pip packages except for gmp, atlas and mpi4py. So I included a few packages that you omitted.

comment:34 Changed 2 years ago by mderickx

Hi Jeroen thanks for putting so much energy in these optional packages,

I recreated the pandoc problem and created #23985 for this with a clear traceback and explanation of why it occurs. I think the gap_packages and database_gap errors can be regarded as a mistake in installing all these optional packages from my side, because I cannot reproduce it either on the machine where it went wrong.

Note: See TracTickets for help on using tickets.