upgrade GAP to 4.11.1 or newer

[dimpase]

the current GAP release is 4.11.1. Time to upgrade - previous upgrade was in #29314.

Here we also upgrade libsemigroups, in lockstep with GAP, to 1.1.0.

GAP tarball is now hosted on GitHub; we also need to install more GAP packages to satsify needs of updated GAP package digraph and semigroups.

Finally, we fix a bug in TestSuite? for abelian groups

[dimpase]

New commits:

​37466dd

GAP version bump to 4.11.1

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8367f8e

ctbl versions now numbered

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1d113f8	update libsemigroup
​246664d	more GAP packages are deps

[dimpase]

what remains is to fix this weird error - something is emitting that string of * whcih overrides the test output. Not reproducible at sage: prompt:

File "src/sage/tests/gap_packages.py", line 8, in sage.tests.gap_packages
Failed example:
    test_packages(pkgs, only_failures=True)    # optional - gap_packages
Expected:
    ...
      Status   Package   GAP Output
    +--------+---------+------------+
Got:
      Status   Package   GAP Output
    +--------+---------+------------+
**********************************************************************

but

sage: from sage.tests.gap_packages import all_installed_packages, test_packages                                                                                                                                                                                                   
sage: pkgs = all_installed_packages(ignore_dot_gap=True)                                                                                                                                                                                                                          
sage: test_packages(pkgs, only_failures=True)                                                                                                                                                                                                                                     
  Status   Package   GAP Output
+--------+---------+------------+
sage:   

[arojas]

Replying to dimpase:

what remains is to fix this weird error - something is emitting that string of * whcih overrides the test output.

Isn't that just the usual test failure output? It seems to me that the actual issue is the "..." wildcard on the first line - there is no newline before the "Status" line so it doesn't match.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7170197

modify the test so that Sage's "table" is not used

[dimpase]

another test to fix:

sage -t --warn-long 50.1 --random-seed=0 src/sage/groups/abelian_gps/abelian_group_gap.py
**********************************************************************
File "src/sage/groups/abelian_gps/abelian_group_gap.py", line 201, in sage.groups.abelian_gps.abelian_group_gap.AbelianGroupElement_polycyclic
Failed example:
    TestSuite(G.an_element()).run()       # optional - gap_packages
Expected nothing
Got:
    Failure in _test_pickling:
    Traceback (most recent call last):
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.8/site-packages/sage/misc/sage_unittest.py", line 297, in run
        test_method(tester=tester)
      File "sage/structure/sage_object.pyx", line 647, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:5023)
        tester.assertEqual(loads(dumps(self)), self)
      File "sage/misc/persist.pyx", line 935, in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7536)
        return unpickler.load()
      File "sage/structure/parent.pyx", line 898, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9335)
        return mor._call_(x)
      File "sage/structure/coerce_maps.pyx", line 161, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4622)
        raise
      File "sage/structure/coerce_maps.pyx", line 156, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4514)
        return C._element_constructor(x)
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.8/site-packages/sage/groups/abelian_gps/abelian_group_gap.py", line 368, in _element_constructor_
        x *= gens_gap[i]**(exp[i] % orders[i])
      File "sage/rings/integer.pyx", line 3413, in sage.rings.integer.Integer.__mod__ (build/cythonized/sage/rings/integer.c:22616)
        raise ZeroDivisionError("Integer modulo by zero")
    ZeroDivisionError: Integer modulo by zero
    ------------------------------------------------------------
    The following tests failed: _test_pickling
**********************************************************************

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5c885dd

do not take "mod 0", skip "mod"

[dimpase]

please review...

[fbissey]

I won't lie, I am not quite ready for this in sage-on-gentoo. I have been under rather heavy stress so far this year and I am not doing as much as I want.

[vbraun]

Tell me about it ;-)

But I think we should have people bang away at this asap and not later in the release cycle.

[dimpase]

Thanks!

[vbraun]

There is a small merge conflict with #31489, and a failure

sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py
**********************************************************************
File "src/sage/tests/gap_packages.py", line 9, in sage.tests.gap_packages
Failed example:
    for p in pkgs:
        pkg = p.split('-')[0]
        if not libgap.LoadPackage(pkg):
            raise
Exception raised:
    Traceback (most recent call last):
      File "/home/release/Sage/local/lib64/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/release/Sage/local/lib64/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.tests.gap_packages[3]>", line 4, in <module>
        raise
    RuntimeError: No active exception to reraise
**********************************************************************
1 item had failures:
   1 of   6 in sage.tests.gap_packages
    [12 tests, 1 failure, 0.26 s]
----------------------------------------------------------------------
sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py  # 1 doctest failed
----------------------------------------------------------------------

[dimpase]

Replying to vbraun:

There is a small merge conflict with #31489, and a failure

sage -t --long --warn-long 43.1 --random-seed=0 src/sage/tests/gap_packages.py
...

I only see this if I manually uninstall gap_packages. But this meant to be with them installed?

96396e238d4 (Volker Braun     2013-07-22 17:05:24 -0400   2) Test the optional GAP packages

GAP doesn't have a clean way to uninstall packages.

I can rebase over #31489 if it helps.

[dimpase]

these test_packages stuff is a mess. We have two different functions that both claim to Return list of all installed packages., test_packages(), and all_installed_packages() which differ in the output format - the former outputs table (sic!) and the latter does not check whether these packages actually may be loaded (so it's largely meaningless).

May we do it on another ticket?

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​a9ac506	GAP version bump to 4.11.1
​f8c87d4	ctbl versions now numbered
​a2bc010	update libsemigroup
​40e6105	more GAP packages are deps
​8d44ad0	modify the test so that Sage's "table" is not used
​05b434a	do not take "mod 0", skip "mod"
​7ee7b58	better fix for the test

[dimpase]

I believe the test is fixed now.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​101bd74

use the change from #31489

[dimpase]

OK, now it depends on #31489 and should cleanly merge: the only change from #31489 is done in comment:22

[dimpase]

I have not merged the full branch of #31489 here, though.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​96df458	GAP version bump to 4.11.1
​91a695f	ctbl versions now numbered
​79b3bde	update libsemigroup
​1116073	more GAP packages are deps
​7469042	modify the test so that Sage's "table" is not used
​82f3fa7	better fix for the test

[dimpase]

rebased on 9.4.beta1

[tscrim]

Let's try to get this into the next beta.

[fbissey]

I am a bit dumbfounded by this upgrade in a way. There are strange particulars to it in sage-on-gentoo. I cannot build the doc.

[categorie] /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/categories/finite_coxeter_groups.py:docstring of sage.categories.finite_coxeter_groups.FiniteCoxeterGroups.ParentMethods.permutahedron:50: WARNING: Exception occurred in plotting finite_coxeter_groups-1
[categorie]  from /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/doc/en/reference/categories/sage/categories/finite_coxeter_groups.rst:
[categorie] Traceback (most recent call last):
[categorie]   File "/usr/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 483, in run_code
[categorie]     exec(code, ns)
[categorie]   File "<string>", line 1, in <module>
[categorie]   File "sage/misc/lazy_import.pyx", line 362, in sage.misc.lazy_import.LazyImport.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/lazy_import.c:4041)
[categorie]     return self.get_object()(*args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/combinat/root_system/coxeter_group.py", line 135, in CoxeterGroup
[categorie]     return CoxeterMatrixGroup(cartan_type, base_ring, index_set)
[categorie]   File "sage/misc/classcall_metaclass.pyx", line 322, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/classcall_metaclass.c:1743)
[categorie]     return cls.classcall(cls, *args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/groups/matrix_gps/coxeter_group.py", line 230, in __classcall_private__
[categorie]     return super(CoxeterMatrixGroup, cls).__classcall__(cls,
[categorie]   File "sage/misc/cachefunc.pyx", line 1001, in sage.misc.cachefunc.CachedFunction.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:6087)
[categorie]     w = self.f(*args, **kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/structure/unique_representation.py", line 1007, in __classcall__
[categorie]     instance = typecall(cls, *args, **options)
[categorie]   File "sage/misc/classcall_metaclass.pyx", line 486, in sage.misc.classcall_metaclass.typecall (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/classcall_metaclass.c:2207)
[categorie]     return (<PyTypeObject*>type).tp_call(cls, args, kwds)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/groups/matrix_gps/coxeter_group.py", line 273, in __init__
[categorie]     one = MS.one()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/matrix/matrix_space.py", line 1608, in identity_matrix
[categorie]     A = self.zero_matrix().__copy__()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/matrix/matrix_space.py", line 1766, in zero_matrix
[categorie]     zero = self.base_ring().zero()
[categorie]   File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/misc/cachefunc.c:12918)
[categorie]     self.cache = f(self._instance)
[categorie]   File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/rings/universal_cyclotomic_field.py", line 1397, in zero
[categorie]     return self.element_class(self, libgap.zero())
[categorie]   File "sage/libs/gap/libgap.pyx", line 618, in sage.libs.gap.libgap.Gap.zero (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/libgap.c:6277)
[categorie]     return self(0)
[categorie]   File "sage/structure/parent.pyx", line 898, in sage.structure.parent.Parent.__call__ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/parent.c:9342)
[categorie]     return mor._call_(x)
[categorie]   File "sage/structure/coerce_maps.pyx", line 161, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_maps.c:4628)
[categorie]     raise
[categorie]   File "sage/structure/coerce_maps.pyx", line 156, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_maps.c:4520)
[categorie]     return C._element_constructor(x)
[categorie]   File "sage/libs/gap/libgap.pyx", line 298, in sage.libs.gap.libgap.Gap._element_constructor_ (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/libgap.c:3391)
[categorie]     initialize()
[categorie]   File "sage/libs/gap/util.pyx", line 314, in sage.libs.gap.util.initialize (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:5897)
[categorie]     with atomic_write(workspace) as f:
[categorie]   File "sage/libs/gap/util.pyx", line 316, in sage.libs.gap.util.initialize (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:5852)
[categorie]     gap_eval('SaveWorkspace("{0}")'.format(f.name))
[categorie]   File "sage/libs/gap/util.pyx", line 398, in sage.libs.gap.util.gap_eval (/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/libs/gap/util.c:6068)
[categorie]     GAP_Enter()
[categorie] sage.libs.gap.util.GAPError: Error, reached the pre-set memory limit
[categorie] (change it with the -o command line option)
[graphs   ] The inventory files are in ../../build_doc/inventory/en/reference/graphs.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/graphs
[functions] building [inventory]: targets for 20 source files that are out of date
[functions] updating environment: [new config] 20 added, 0 changed, 0 removed
[finite_ri] The inventory files are in ../../build_doc/inventory/en/reference/finite_rings.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/finite_rings
[function_] The inventory files are in ../../build_doc/inventory/en/reference/function_fields.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/function_fields
[game_theo] building [inventory]: targets for 6 source files that are out of date
[game_theo] updating environment: [new config] 6 added, 0 changed, 0 removed
[games    ] building [inventory]: targets for 5 source files that are out of date
[games    ] updating environment: [new config] 5 added, 0 changed, 0 removed
[game_theo] The inventory files are in ../../build_doc/inventory/en/reference/game_theory.
Build finished. The built documents can be found in /var/tmp/portage/sci-mathematics/sage-9999/work/build_doc/inventory/en/reference/game_theory
[categorie] The inventory files are in ../../build_doc/inventory/en/reference/categories.
Error building the documentation.
Traceback (most recent call last):
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 1813, in main
    builder()
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 345, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 582, in _wrapper
    self._build_everything_except_bibliography(format, *args, **kwds)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 565, in _build_everything_except_bibliography
    build_many(build_ref_doc, non_references)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/__init__.py", line 297, in build_many
    _build_many(target, args, processes=processes)
  File "/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/sage_docbuild/utils.py", line 289, in build_many
    raise worker_exc.original_exception
OSError: /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/categories/finite_coxeter_groups.py:docstring of sage.categories.finite_coxeter_groups.FiniteCoxeterGroups.ParentMethods.permutahedron:50: WARNING: Exception occurred in plotting finite_coxeter_groups-1

In sage-on-gentoo, I do separate gap packages and the source of error is more precisely ctbllib-1.3.1+ (I say + because there is a 1.3.2 now in its upstream and the result is the same with it). If I downgrade ctbllib to 1.2_p2 the documentation builds.

The error message suggests to increase the memory limit. This is set somewhere in sage/interface/gap.py and I pushed it from the current 400MB to 2000MB without improvement.

I am probably missing something but what?

[vbraun]

Whats the output of _get_gap_memory_pool_size_MB?

GAP needs swap due to how they reserve the memory pool.

[fbissey]

Replying to vbraun:

Whats the output of _get_gap_memory_pool_size_MB?

GAP needs swap due to how they reserve the memory pool.

568MB if I leave it to the default. With the changes for my last attempt, it should be 2001MB. But what puzzles me is the influence of ctbllib. So, I am wondering if the message is in fact unhelpful for the issue at hand.