Opened 9 months ago

Closed 6 months ago

Last modified 6 months ago

#31498 closed enhancement (fixed)

upgrade GAP to 4.11.1 or newer

Reported by: dimpase Owned by:
Priority: major Milestone: sage-9.4
Component: packages: standard Keywords: upgrade, gap
Cc: arojas, embray, fbissey, mkoeppe, slelievre, thansen, vdelecroix Merged in:
Authors: Dima Pasechnik Reviewers: Volker Braun, Travis Scrimshaw
Report Upstream: N/A Work issues:
Branch: 82f3fa7 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by dimpase)

the current GAP release is 4.11.1. Time to upgrade - previous upgrade was in #29314.

Here we also upgrade libsemigroups, in lockstep with GAP, to 1.1.0.

GAP tarball is now hosted on GitHub; we also need to install more GAP packages to satsify needs of updated GAP package digraph and semigroups.

Finally, we fix a bug in TestSuite? for abelian groups

Change History (32)

comment:1 Changed 9 months ago by slelievre

  • Cc arojas embray fbissey mkoeppe slelievre added
  • Keywords upgrade gap added

comment:2 Changed 7 months ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to u/dimpase/packages/gap4111
  • Commit set to 37466dd9a122c4aade0f412313f53c59151e82c5

New commits:

37466ddGAP version bump to 4.11.1

comment:3 Changed 7 months ago by git

  • Commit changed from 37466dd9a122c4aade0f412313f53c59151e82c5 to 8367f8e8422c2859418e084ff70f05b0f5f7bd97

Branch pushed to git repo; I updated commit sha1. New commits:

8367f8ectbl versions now numbered

comment:4 Changed 7 months ago by git

  • Commit changed from 8367f8e8422c2859418e084ff70f05b0f5f7bd97 to 246664d79f4267f2ee9566bd25f26da772a3d4db

Branch pushed to git repo; I updated commit sha1. New commits:

1d113f8update libsemigroup
246664dmore GAP packages are deps

comment:5 follow-up: Changed 7 months ago by dimpase

what remains is to fix this weird error - something is emitting that string of * whcih overrides the test output. Not reproducible at sage: prompt:

File "src/sage/tests/gap_packages.py", line 8, in sage.tests.gap_packages
Failed example:
    test_packages(pkgs, only_failures=True)    # optional - gap_packages
Expected:
    ...
      Status   Package   GAP Output
    +--------+---------+------------+
Got:
      Status   Package   GAP Output
    +--------+---------+------------+
**********************************************************************

but

sage: from sage.tests.gap_packages import all_installed_packages, test_packages                                                                                                                                                                                                   
sage: pkgs = all_installed_packages(ignore_dot_gap=True)                                                                                                                                                                                                                          
sage: test_packages(pkgs, only_failures=True)                                                                                                                                                                                                                                     
  Status   Package   GAP Output
+--------+---------+------------+
sage:   

comment:6 in reply to: ↑ 5 Changed 7 months ago by arojas

Replying to dimpase:

what remains is to fix this weird error - something is emitting that string of * whcih overrides the test output.

Isn't that just the usual test failure output? It seems to me that the actual issue is the "..." wildcard on the first line - there is no newline before the "Status" line so it doesn't match.

comment:7 Changed 7 months ago by git

  • Commit changed from 246664d79f4267f2ee9566bd25f26da772a3d4db to 717019729bb9e8a713b57fcdeb18420ec838a7f3

Branch pushed to git repo; I updated commit sha1. New commits:

7170197modify the test so that Sage's "table" is not used

comment:8 Changed 7 months ago by dimpase

another test to fix:

sage -t --warn-long 50.1 --random-seed=0 src/sage/groups/abelian_gps/abelian_group_gap.py
**********************************************************************
File "src/sage/groups/abelian_gps/abelian_group_gap.py", line 201, in sage.groups.abelian_gps.abelian_group_gap.AbelianGroupElement_polycyclic
Failed example:
    TestSuite(G.an_element()).run()       # optional - gap_packages
Expected nothing
Got:
    Failure in _test_pickling:
    Traceback (most recent call last):
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.8/site-packages/sage/misc/sage_unittest.py", line 297, in run
        test_method(tester=tester)
      File "sage/structure/sage_object.pyx", line 647, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:5023)
        tester.assertEqual(loads(dumps(self)), self)
      File "sage/misc/persist.pyx", line 935, in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7536)
        return unpickler.load()
      File "sage/structure/parent.pyx", line 898, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9335)
        return mor._call_(x)
      File "sage/structure/coerce_maps.pyx", line 161, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4622)
        raise
      File "sage/structure/coerce_maps.pyx", line 156, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4514)
        return C._element_constructor(x)
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.8/site-packages/sage/groups/abelian_gps/abelian_group_gap.py", line 368, in _element_constructor_
        x *= gens_gap[i]**(exp[i] % orders[i])
      File "sage/rings/integer.pyx", line 3413, in sage.rings.integer.Integer.__mod__ (build/cythonized/sage/rings/integer.c:22616)
        raise ZeroDivisionError("Integer modulo by zero")
    ZeroDivisionError: Integer modulo by zero
    ------------------------------------------------------------
    The following tests failed: _test_pickling
**********************************************************************

comment:9 Changed 7 months ago by git

  • Commit changed from 717019729bb9e8a713b57fcdeb18420ec838a7f3 to 5c885dd62363bbc63e88c2dc2473a783bf2c4035

Branch pushed to git repo; I updated commit sha1. New commits:

5c885dddo not take "mod 0", skip "mod"

comment:10 Changed 7 months ago by dimpase

  • Description modified (diff)
  • Status changed from new to needs_review

comment:11 Changed 6 months ago by dimpase

  • Cc thansen vdelecroix added

please review...

comment:12 Changed 6 months ago by fbissey

I won't lie, I am not quite ready for this in sage-on-gentoo. I have been under rather heavy stress so far this year and I am not doing as much as I want.

comment:13 Changed 6 months ago by vbraun

  • Status changed from needs_review to positive_review

Tell me about it ;-)

But I think we should have people bang away at this asap and not later in the release cycle.

comment:14 Changed 6 months ago by dimpase

Thanks!

comment:15 Changed 6 months ago by vbraun

  • Reviewers set to Volker Braun

comment:16 follow-up: Changed 6 months ago by vbraun

  • Status changed from positive_review to needs_work

There is a small merge conflict with #31489, and a failure

sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py
**********************************************************************
File "src/sage/tests/gap_packages.py", line 9, in sage.tests.gap_packages
Failed example:
    for p in pkgs:
        pkg = p.split('-')[0]
        if not libgap.LoadPackage(pkg):
            raise
Exception raised:
    Traceback (most recent call last):
      File "/home/release/Sage/local/lib64/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/release/Sage/local/lib64/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.tests.gap_packages[3]>", line 4, in <module>
        raise
    RuntimeError: No active exception to reraise
**********************************************************************
1 item had failures:
   1 of   6 in sage.tests.gap_packages
    [12 tests, 1 failure, 0.26 s]
----------------------------------------------------------------------
sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py  # 1 doctest failed
----------------------------------------------------------------------

comment:17 in reply to: ↑ 16 Changed 6 months ago by dimpase

Replying to vbraun:

There is a small merge conflict with #31489, and a failure

sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py
...

I only see this if I manually uninstall gap_packages. But this meant to be with them installed?

96396e238d4 (Volker Braun     2013-07-22 17:05:24 -0400   2) Test the optional GAP packages

GAP doesn't have a clean way to uninstall packages.

I can rebase over #31489 if it helps.

comment:18 Changed 6 months ago by dimpase

these test_packages stuff is a mess. We have two different functions that both claim to Return list of all installed packages., test_packages(), and all_installed_packages() which differ in the output format - the former outputs table (sic!) and the latter does not check whether these packages actually may be loaded (so it's largely meaningless).

May we do it on another ticket?

comment:19 Changed 6 months ago by git

  • Commit changed from 5c885dd62363bbc63e88c2dc2473a783bf2c4035 to 7ee7b5801bb4e4fe29b22de4acf28b2f8b01ab02

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a9ac506GAP version bump to 4.11.1
f8c87d4ctbl versions now numbered
a2bc010update libsemigroup
40e6105more GAP packages are deps
8d44ad0modify the test so that Sage's "table" is not used
05b434ado not take "mod 0", skip "mod"
7ee7b58better fix for the test

comment:20 Changed 6 months ago by dimpase

  • Status changed from needs_work to needs_review

I believe the test is fixed now.

comment:21 Changed 6 months ago by dimpase

  • Dependencies set to #31489

comment:22 Changed 6 months ago by git

  • Commit changed from 7ee7b5801bb4e4fe29b22de4acf28b2f8b01ab02 to 101bd7451120e1a7972433f1c968ad62e5b266dc

Branch pushed to git repo; I updated commit sha1. New commits:

101bd74use the change from #31489

comment:23 Changed 6 months ago by dimpase

OK, now it depends on #31489 and should cleanly merge: the only change from #31489 is done in comment:22

Last edited 6 months ago by dimpase (previous) (diff)

comment:24 Changed 6 months ago by dimpase

I have not merged the full branch of #31489 here, though.

comment:25 Changed 6 months ago by git

  • Commit changed from 101bd7451120e1a7972433f1c968ad62e5b266dc to 82f3fa7bcc0e73c85690a0886a67c5d8cd2e5c2b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

96df458GAP version bump to 4.11.1
91a695fctbl versions now numbered
79b3bdeupdate libsemigroup
1116073more GAP packages are deps
7469042modify the test so that Sage's "table" is not used
82f3fa7better fix for the test

comment:26 Changed 6 months ago by dimpase

rebased on 9.4.beta1

comment:27 Changed 6 months ago by dimpase

  • Dependencies #31489 deleted

comment:28 Changed 6 months ago by tscrim

  • Reviewers changed from Volker Braun to Volker Braun, Travis Scrimshaw
  • Status changed from needs_review to positive_review

Let's try to get this into the next beta.

Last edited 6 months ago by tscrim (previous) (diff)

comment:29 Changed 6 months ago by vbraun

  • Branch changed from u/dimpase/packages/gap4111 to 82f3fa7bcc0e73c85690a0886a67c5d8cd2e5c2b
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:30 Changed 6 months ago by fbissey

  • Commit 82f3fa7bcc0e73c85690a0886a67c5d8cd2e5c2b deleted

I am a bit dumbfounded by this upgrade in a way. There are strange particulars to it in sage-on-gentoo. I cannot build the doc.

[categorie] /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/categories/finite_coxeter_groups.py:docstring of sage.categories.finite_coxeter_groups.FiniteCoxeterGroups.ParentMethods.permutahedron:50: WARNING: Exception occurred in plotting finite_coxeter_groups-1
[categorie]  from /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/doc/en/reference/categories/sage/categories/finite_coxeter_groups.rst:
[categorie] Traceback (most recent call last):
[categorie]   File "/usr/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 483, in run_code
[categorie]     exec(code, ns)
[categorie]   File "<string>", line 1, in <module>
[categorie]   File "sage/misc/lazy_import.pyx", line 362, in sage.misc.lazy_import.LazyImport.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/lazy_import.c:4041)
[categorie]     return self.get_object()(*args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/combinat/root_system/coxeter_group.py", line 135, in CoxeterGroup
[categorie]     return CoxeterMatrixGroup(cartan_type, base_ring, index_set)
[categorie]   File "sage/misc/classcall_metaclass.pyx", line 322, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/classcall_metaclass.c:1743)
[categorie]     return cls.classcall(cls, *args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/groups/matrix_gps/coxeter_group.py", line 230, in __classcall_private__
[categorie]     return super(CoxeterMatrixGroup, cls).__classcall__(cls,
[categorie]   File "sage/misc/cachefunc.pyx", line 1001, in sage.misc.cachefunc.CachedFunction.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:6087)
[categorie]     w = self.f(*args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/structure/unique_representation.py", line 1007, in __classcall__
[categorie]     instance = typecall(cls, *args, **options)
[categorie]   File "sage/misc/classcall_metaclass.pyx", line 486, in sage.misc.classcall_metaclass.typecall (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/classcall_metaclass.c:2207)
[categorie]     return (<PyTypeObject*>type).tp_call(cls, args, kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/groups/matrix_gps/coxeter_group.py", line 273, in __init__
[categorie]     one = MS.one()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/matrix/matrix_space.py", line 1608, in identity_matrix
[categorie]     A = self.zero_matrix().__copy__()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/matrix/matrix_space.py", line 1766, in zero_matrix
[categorie]     zero = self.base_ring().zero()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/rings/universal_cyclotomic_field.py", line 1397, in zero
[categorie]     return self.element_class(self, libgap.zero())
[categorie]   File "sage/libs/gap/libgap.pyx", line 618, in sage.libs.gap.libgap.Gap.zero (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/libgap.c:6277)
[categorie]     return self(0)
[categorie]   File "sage/structure/parent.pyx", line 898, in sage.structure.parent.Parent.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/parent.c:9342)
[categorie]     return mor._call_(x)
[categorie]   File "sage/structure/coerce_maps.pyx", line 161, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_maps.c:4628)
[categorie]     raise
[categorie]   File "sage/structure/coerce_maps.pyx", line 156, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_maps.c:4520)
[categorie]     return C._element_constructor(x)
[categorie]   File "sage/libs/gap/libgap.pyx", line 298, in sage.libs.gap.libgap.Gap._element_constructor_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/libgap.c:3391)
[categorie]     initialize()
[categorie]   File "sage/libs/gap/util.pyx", line 314, in sage.libs.gap.util.initialize (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:5897)
[categorie]     with atomic_write(workspace) as f:
[categorie]   File "sage/libs/gap/util.pyx", line 316, in sage.libs.gap.util.initialize (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:5852)
[categorie]     gap_eval('SaveWorkspace("{0}")'.format(f.name))
[categorie]   File "sage/libs/gap/util.pyx", line 398, in sage.libs.gap.util.gap_eval (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:6068)
[categorie]     GAP_Enter()
[categorie] sage.libs.gap.util.GAPError: Error, reached the pre-set memory limit
[categorie] (change it with the -o command line option)
[graphs   ] The inventory files are in ../../build_doc/inventory/en/reference/graphs.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/graphs
[functions] building [inventory]: targets for 20 source files that are out of date
[functions] updating environment: [new config] 20 added, 0 changed, 0 removed
[finite_ri] The inventory files are in ../../build_doc/inventory/en/reference/finite_rings.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/finite_rings
[function_] The inventory files are in ../../build_doc/inventory/en/reference/function_fields.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/function_fields
[game_theo] building [inventory]: targets for 6 source files that are out of date
[game_theo] updating environment: [new config] 6 added, 0 changed, 0 removed
[games    ] building [inventory]: targets for 5 source files that are out of date
[games    ] updating environment: [new config] 5 added, 0 changed, 0 removed
[game_theo] The inventory files are in ../../build_doc/inventory/en/reference/game_theory.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/game_theory
[categorie] The inventory files are in ../../build_doc/inventory/en/reference/categories.
Error building the documentation.
Traceback (most recent call last):
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 1813, in main
    builder()
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 345, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 582, in _wrapper
    self._build_everything_except_bibliography(format, *args, **kwds)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 565, in _build_everything_except_bibliography
    build_many(build_ref_doc, non_references)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 297, in build_many
    _build_many(target, args, processes=processes)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/utils.py", line 289, in build_many
    raise worker_exc.original_exception
OSError: /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/categories/finite_coxeter_groups.py:docstring of sage.categories.finite_coxeter_groups.FiniteCoxeterGroups.ParentMethods.permutahedron:50: WARNING: Exception occurred in plotting finite_coxeter_groups-1

In sage-on-gentoo, I do separate gap packages and the source of error is more precisely ctbllib-1.3.1+ (I say + because there is a 1.3.2 now in its upstream and the result is the same with it). If I downgrade ctbllib to 1.2_p2 the documentation builds.

The error message suggests to increase the memory limit. This is set somewhere in sage/interface/gap.py and I pushed it from the current 400MB to 2000MB without improvement.

I am probably missing something but what?

comment:31 follow-up: Changed 6 months ago by vbraun

Whats the output of _get_gap_memory_pool_size_MB?

GAP needs swap due to how they reserve the memory pool.

comment:32 in reply to: ↑ 31 Changed 6 months ago by fbissey

Replying to vbraun:

Whats the output of _get_gap_memory_pool_size_MB?

GAP needs swap due to how they reserve the memory pool.

568MB if I leave it to the default. With the changes for my last attempt, it should be 2001MB. But what puzzles me is the influence of ctbllib. So, I am wondering if the message is in fact unhelpful for the issue at hand.

Note: See TracTickets for help on using tickets.