Opened 21 months ago

Closed 18 months ago

Last modified 18 months ago

#31498 closed enhancement (fixed)

upgrade GAP to 4.11.1 or newer

Reported by: Dima Pasechnik Owned by:
Priority: major Milestone: sage-9.4
Component: packages: standard Keywords: upgrade, gap
Cc: Antonio Rojas, Erik Bray, François Bissey, Matthias Köppe, Samuel Lelièvre, Tobias Hansen, Vincent Delecroix Merged in:
Authors: Dima Pasechnik Reviewers: Volker Braun, Travis Scrimshaw
Report Upstream: N/A Work issues:
Branch: 82f3fa7 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Dima Pasechnik)

the current GAP release is 4.11.1. Time to upgrade - previous upgrade was in #29314.

Here we also upgrade libsemigroups, in lockstep with GAP, to 1.1.0.

GAP tarball is now hosted on GitHub; we also need to install more GAP packages to satsify needs of updated GAP package digraph and semigroups.

Finally, we fix a bug in TestSuite? for abelian groups

Change History (32)

comment:1 Changed 21 months ago by Samuel Lelièvre

Cc: Antonio Rojas Erik Bray François Bissey Matthias Köppe Samuel Lelièvre added
Keywords: upgrade gap added

comment:2 Changed 19 months ago by Dima Pasechnik

Authors: Dima Pasechnik
Branch: u/dimpase/packages/gap4111
Commit: 37466dd9a122c4aade0f412313f53c59151e82c5

New commits:

37466ddGAP version bump to 4.11.1

comment:3 Changed 19 months ago by git

Commit: 37466dd9a122c4aade0f412313f53c59151e82c58367f8e8422c2859418e084ff70f05b0f5f7bd97

Branch pushed to git repo; I updated commit sha1. New commits:

8367f8ectbl versions now numbered

comment:4 Changed 19 months ago by git

Commit: 8367f8e8422c2859418e084ff70f05b0f5f7bd97246664d79f4267f2ee9566bd25f26da772a3d4db

Branch pushed to git repo; I updated commit sha1. New commits:

1d113f8update libsemigroup
246664dmore GAP packages are deps

comment:5 Changed 19 months ago by Dima Pasechnik

what remains is to fix this weird error - something is emitting that string of * whcih overrides the test output. Not reproducible at sage: prompt:

File "src/sage/tests/gap_packages.py", line 8, in sage.tests.gap_packages
Failed example:
    test_packages(pkgs, only_failures=True)    # optional - gap_packages
Expected:
    ...
      Status   Package   GAP Output
    +--------+---------+------------+
Got:
      Status   Package   GAP Output
    +--------+---------+------------+
**********************************************************************

but

sage: from sage.tests.gap_packages import all_installed_packages, test_packages                                                                                                                                                                                                   
sage: pkgs = all_installed_packages(ignore_dot_gap=True)                                                                                                                                                                                                                          
sage: test_packages(pkgs, only_failures=True)                                                                                                                                                                                                                                     
  Status   Package   GAP Output
+--------+---------+------------+
sage:   

comment:6 in reply to:  5 Changed 19 months ago by Antonio Rojas

Replying to dimpase:

what remains is to fix this weird error - something is emitting that string of * whcih overrides the test output.

Isn't that just the usual test failure output? It seems to me that the actual issue is the "..." wildcard on the first line - there is no newline before the "Status" line so it doesn't match.

comment:7 Changed 19 months ago by git

Commit: 246664d79f4267f2ee9566bd25f26da772a3d4db717019729bb9e8a713b57fcdeb18420ec838a7f3

Branch pushed to git repo; I updated commit sha1. New commits:

7170197modify the test so that Sage's "table" is not used

comment:8 Changed 19 months ago by Dima Pasechnik

another test to fix:

sage -t --warn-long 50.1 --random-seed=0 src/sage/groups/abelian_gps/abelian_group_gap.py
**********************************************************************
File "src/sage/groups/abelian_gps/abelian_group_gap.py", line 201, in sage.groups.abelian_gps.abelian_group_gap.AbelianGroupElement_polycyclic
Failed example:
    TestSuite(G.an_element()).run()       # optional - gap_packages
Expected nothing
Got:
    Failure in _test_pickling:
    Traceback (most recent call last):
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.8/site-packages/sage/misc/sage_unittest.py", line 297, in run
        test_method(tester=tester)
      File "sage/structure/sage_object.pyx", line 647, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:5023)
        tester.assertEqual(loads(dumps(self)), self)
      File "sage/misc/persist.pyx", line 935, in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7536)
        return unpickler.load()
      File "sage/structure/parent.pyx", line 898, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9335)
        return mor._call_(x)
      File "sage/structure/coerce_maps.pyx", line 161, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4622)
        raise
      File "sage/structure/coerce_maps.pyx", line 156, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4514)
        return C._element_constructor(x)
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.8/site-packages/sage/groups/abelian_gps/abelian_group_gap.py", line 368, in _element_constructor_
        x *= gens_gap[i]**(exp[i] % orders[i])
      File "sage/rings/integer.pyx", line 3413, in sage.rings.integer.Integer.__mod__ (build/cythonized/sage/rings/integer.c:22616)
        raise ZeroDivisionError("Integer modulo by zero")
    ZeroDivisionError: Integer modulo by zero
    ------------------------------------------------------------
    The following tests failed: _test_pickling
**********************************************************************

comment:9 Changed 19 months ago by git

Commit: 717019729bb9e8a713b57fcdeb18420ec838a7f35c885dd62363bbc63e88c2dc2473a783bf2c4035

Branch pushed to git repo; I updated commit sha1. New commits:

5c885dddo not take "mod 0", skip "mod"

comment:10 Changed 19 months ago by Dima Pasechnik

Description: modified (diff)
Status: newneeds_review

comment:11 Changed 19 months ago by Dima Pasechnik

Cc: Tobias Hansen Vincent Delecroix added

please review...

comment:12 Changed 19 months ago by François Bissey

I won't lie, I am not quite ready for this in sage-on-gentoo. I have been under rather heavy stress so far this year and I am not doing as much as I want.

comment:13 Changed 19 months ago by Volker Braun

Status: needs_reviewpositive_review

Tell me about it ;-)

But I think we should have people bang away at this asap and not later in the release cycle.

comment:14 Changed 19 months ago by Dima Pasechnik

Thanks!

comment:15 Changed 19 months ago by Volker Braun

Reviewers: Volker Braun

comment:16 Changed 19 months ago by Volker Braun

Status: positive_reviewneeds_work

There is a small merge conflict with #31489, and a failure

sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py
**********************************************************************
File "src/sage/tests/gap_packages.py", line 9, in sage.tests.gap_packages
Failed example:
    for p in pkgs:
        pkg = p.split('-')[0]
        if not libgap.LoadPackage(pkg):
            raise
Exception raised:
    Traceback (most recent call last):
      File "/home/release/Sage/local/lib64/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/release/Sage/local/lib64/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.tests.gap_packages[3]>", line 4, in <module>
        raise
    RuntimeError: No active exception to reraise
**********************************************************************
1 item had failures:
   1 of   6 in sage.tests.gap_packages
    [12 tests, 1 failure, 0.26 s]
----------------------------------------------------------------------
sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py  # 1 doctest failed
----------------------------------------------------------------------

comment:17 in reply to:  16 Changed 19 months ago by Dima Pasechnik

Replying to vbraun:

There is a small merge conflict with #31489, and a failure

sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py
...

I only see this if I manually uninstall gap_packages. But this meant to be with them installed?

96396e238d4 (Volker Braun     2013-07-22 17:05:24 -0400   2) Test the optional GAP packages

GAP doesn't have a clean way to uninstall packages.

I can rebase over #31489 if it helps.

comment:18 Changed 19 months ago by Dima Pasechnik

these test_packages stuff is a mess. We have two different functions that both claim to Return list of all installed packages., test_packages(), and all_installed_packages() which differ in the output format - the former outputs table (sic!) and the latter does not check whether these packages actually may be loaded (so it's largely meaningless).

May we do it on another ticket?

comment:19 Changed 19 months ago by git

Commit: 5c885dd62363bbc63e88c2dc2473a783bf2c40357ee7b5801bb4e4fe29b22de4acf28b2f8b01ab02

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a9ac506GAP version bump to 4.11.1
f8c87d4ctbl versions now numbered
a2bc010update libsemigroup
40e6105more GAP packages are deps
8d44ad0modify the test so that Sage's "table" is not used
05b434ado not take "mod 0", skip "mod"
7ee7b58better fix for the test

comment:20 Changed 19 months ago by Dima Pasechnik

Status: needs_workneeds_review

I believe the test is fixed now.

comment:21 Changed 19 months ago by Dima Pasechnik

Dependencies: #31489

comment:22 Changed 19 months ago by git

Commit: 7ee7b5801bb4e4fe29b22de4acf28b2f8b01ab02101bd7451120e1a7972433f1c968ad62e5b266dc

Branch pushed to git repo; I updated commit sha1. New commits:

101bd74use the change from #31489

comment:23 Changed 19 months ago by Dima Pasechnik

OK, now it depends on #31489 and should cleanly merge (the only change from #31489 is done in comment:22

Version 0, edited 19 months ago by Dima Pasechnik (next)

comment:24 Changed 19 months ago by Dima Pasechnik

I have not merged the full branch of #31489 here, though.

comment:25 Changed 18 months ago by git

Commit: 101bd7451120e1a7972433f1c968ad62e5b266dc82f3fa7bcc0e73c85690a0886a67c5d8cd2e5c2b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

96df458GAP version bump to 4.11.1
91a695fctbl versions now numbered
79b3bdeupdate libsemigroup
1116073more GAP packages are deps
7469042modify the test so that Sage's "table" is not used
82f3fa7better fix for the test

comment:26 Changed 18 months ago by Dima Pasechnik

rebased on 9.4.beta1

comment:27 Changed 18 months ago by Dima Pasechnik

Dependencies: #31489

comment:28 Changed 18 months ago by Travis Scrimshaw

Reviewers: Volker BraunVolker Braun, Travis Scrimshaw
Status: needs_reviewpositive_review

Let's try to get this into the next beta.

Last edited 18 months ago by Travis Scrimshaw (previous) (diff)

comment:29 Changed 18 months ago by Volker Braun

Branch: u/dimpase/packages/gap411182f3fa7bcc0e73c85690a0886a67c5d8cd2e5c2b
Resolution: fixed
Status: positive_reviewclosed

comment:30 Changed 18 months ago by François Bissey

Commit: 82f3fa7bcc0e73c85690a0886a67c5d8cd2e5c2b

I am a bit dumbfounded by this upgrade in a way. There are strange particulars to it in sage-on-gentoo. I cannot build the doc.

[categorie] /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/categories/finite_coxeter_groups.py:docstring of sage.categories.finite_coxeter_groups.FiniteCoxeterGroups.ParentMethods.permutahedron:50: WARNING: Exception occurred in plotting finite_coxeter_groups-1
[categorie]  from /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/doc/en/reference/categories/sage/categories/finite_coxeter_groups.rst:
[categorie] Traceback (most recent call last):
[categorie]   File "/usr/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 483, in run_code
[categorie]     exec(code, ns)
[categorie]   File "<string>", line 1, in <module>
[categorie]   File "sage/misc/lazy_import.pyx", line 362, in sage.misc.lazy_import.LazyImport.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/lazy_import.c:4041)
[categorie]     return self.get_object()(*args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/combinat/root_system/coxeter_group.py", line 135, in CoxeterGroup
[categorie]     return CoxeterMatrixGroup(cartan_type, base_ring, index_set)
[categorie]   File "sage/misc/classcall_metaclass.pyx", line 322, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/classcall_metaclass.c:1743)
[categorie]     return cls.classcall(cls, *args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/groups/matrix_gps/coxeter_group.py", line 230, in __classcall_private__
[categorie]     return super(CoxeterMatrixGroup, cls).__classcall__(cls,
[categorie]   File "sage/misc/cachefunc.pyx", line 1001, in sage.misc.cachefunc.CachedFunction.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:6087)
[categorie]     w = self.f(*args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/structure/unique_representation.py", line 1007, in __classcall__
[categorie]     instance = typecall(cls, *args, **options)
[categorie]   File "sage/misc/classcall_metaclass.pyx", line 486, in sage.misc.classcall_metaclass.typecall (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/classcall_metaclass.c:2207)
[categorie]     return (<PyTypeObject*>type).tp_call(cls, args, kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/groups/matrix_gps/coxeter_group.py", line 273, in __init__
[categorie]     one = MS.one()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/matrix/matrix_space.py", line 1608, in identity_matrix
[categorie]     A = self.zero_matrix().__copy__()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/matrix/matrix_space.py", line 1766, in zero_matrix
[categorie]     zero = self.base_ring().zero()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/rings/universal_cyclotomic_field.py", line 1397, in zero
[categorie]     return self.element_class(self, libgap.zero())
[categorie]   File "sage/libs/gap/libgap.pyx", line 618, in sage.libs.gap.libgap.Gap.zero (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/libgap.c:6277)
[categorie]     return self(0)
[categorie]   File "sage/structure/parent.pyx", line 898, in sage.structure.parent.Parent.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/parent.c:9342)
[categorie]     return mor._call_(x)
[categorie]   File "sage/structure/coerce_maps.pyx", line 161, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_maps.c:4628)
[categorie]     raise
[categorie]   File "sage/structure/coerce_maps.pyx", line 156, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_maps.c:4520)
[categorie]     return C._element_constructor(x)
[categorie]   File "sage/libs/gap/libgap.pyx", line 298, in sage.libs.gap.libgap.Gap._element_constructor_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/libgap.c:3391)
[categorie]     initialize()
[categorie]   File "sage/libs/gap/util.pyx", line 314, in sage.libs.gap.util.initialize (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:5897)
[categorie]     with atomic_write(workspace) as f:
[categorie]   File "sage/libs/gap/util.pyx", line 316, in sage.libs.gap.util.initialize (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:5852)
[categorie]     gap_eval('SaveWorkspace("{0}")'.format(f.name))
[categorie]   File "sage/libs/gap/util.pyx", line 398, in sage.libs.gap.util.gap_eval (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:6068)
[categorie]     GAP_Enter()
[categorie] sage.libs.gap.util.GAPError: Error, reached the pre-set memory limit
[categorie] (change it with the -o command line option)
[graphs   ] The inventory files are in ../../build_doc/inventory/en/reference/graphs.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/graphs
[functions] building [inventory]: targets for 20 source files that are out of date
[functions] updating environment: [new config] 20 added, 0 changed, 0 removed
[finite_ri] The inventory files are in ../../build_doc/inventory/en/reference/finite_rings.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/finite_rings
[function_] The inventory files are in ../../build_doc/inventory/en/reference/function_fields.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/function_fields
[game_theo] building [inventory]: targets for 6 source files that are out of date
[game_theo] updating environment: [new config] 6 added, 0 changed, 0 removed
[games    ] building [inventory]: targets for 5 source files that are out of date
[games    ] updating environment: [new config] 5 added, 0 changed, 0 removed
[game_theo] The inventory files are in ../../build_doc/inventory/en/reference/game_theory.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/game_theory
[categorie] The inventory files are in ../../build_doc/inventory/en/reference/categories.
Error building the documentation.
Traceback (most recent call last):
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 1813, in main
    builder()
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 345, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 582, in _wrapper
    self._build_everything_except_bibliography(format, *args, **kwds)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 565, in _build_everything_except_bibliography
    build_many(build_ref_doc, non_references)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 297, in build_many
    _build_many(target, args, processes=processes)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/utils.py", line 289, in build_many
    raise worker_exc.original_exception
OSError: /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/categories/finite_coxeter_groups.py:docstring of sage.categories.finite_coxeter_groups.FiniteCoxeterGroups.ParentMethods.permutahedron:50: WARNING: Exception occurred in plotting finite_coxeter_groups-1

In sage-on-gentoo, I do separate gap packages and the source of error is more precisely ctbllib-1.3.1+ (I say + because there is a 1.3.2 now in its upstream and the result is the same with it). If I downgrade ctbllib to 1.2_p2 the documentation builds.

The error message suggests to increase the memory limit. This is set somewhere in sage/interface/gap.py and I pushed it from the current 400MB to 2000MB without improvement.

I am probably missing something but what?

comment:31 Changed 18 months ago by Volker Braun

Whats the output of _get_gap_memory_pool_size_MB?

GAP needs swap due to how they reserve the memory pool.

comment:32 in reply to:  31 Changed 18 months ago by François Bissey

Replying to vbraun:

Whats the output of _get_gap_memory_pool_size_MB?

GAP needs swap due to how they reserve the memory pool.

568MB if I leave it to the default. With the changes for my last attempt, it should be 2001MB. But what puzzles me is the influence of ctbllib. So, I am wondering if the message is in fact unhelpful for the issue at hand.

Note: See TracTickets for help on using tickets.