Upgrade to GAP 4.9

[nthiery]

GAP 4.9 will come with a completely rewritten build system that will simplify our packaging. In fact, it may well enable Sage to use a vanilla GAP installation as provided by the distribution. It will also enable building GAP as a library, so that we can get rid of our separate libgap package.

The branch attached to this ticket updates Sage to run on top of ​a branch of GAP by Markus Pfeiffer that adds libgap compilation and ​will be merged soon in the devel version of GAP.

See ​https://github.com/markuspf/gap/issues/2 for the few sticking points that could prevent using a vanilla GAP from the distribution (please edit further if you think about more of them).

What the branch does:

• Remove the libgap spkg

• Update the gap spkg to the new build system and build and install libgap

• Replace gap.shi.patch by a plain gap startup script for Sage

Rationale: GAP used to provide a startup shell script. The GAP devs are in the process of getting rid of it and provide a very minimal one. They recommend to just write our own rather than patching it.

• Update a few doctests w.r.t. changes of output of some GAP functions

• Possibly controversial: The new libgap currently *does not come* with symbol rewriting (foobar -> libGAP_foobar). This avoids messing around with GAP's sources; in particular opening the door for using a stock GAP from the OS distribution. However there always is a risk of name conflict. And indeed, GAP's constants (actually cpp macros) T_INT, T_FLOAT, ... conflict with Python's constants. This is currently worked around by forcing the inclusion of Python's structmember.h before the gap headers.

Something similar was started by Volker at #19915.

• Removes configure.patch: it was patching configure for better GMP detection under Cygwin (#13954). This should not be needed anymore with the new build system and use of --with-gmp. If it is, upstream asked for it to be reported and they will fix it.

• Revert #19726 (not needed anymore)

Status: Most long test pass. Tentatively, the 38 remaining failing tests are due to changes in GAP since 4.8.6: Max mentioned that the library has been cleaned up to always use the same random generation source, and some of the group algorithms were changed as well, which can explain, e.g. change of orders in lists of elements. So those should be nothing to worry about. There is not much point in updating those doctests right away; we may as well wait for a more final version of 4.9 to be out.

TODO:

• Automatic handling of headers (see below for how to do it by hand). GAP's build system will eventually provide a rule to install headers which will make this trivial.

• Use GAP's own make install ​when it will be implemented.

• Check against #19915 to see if any of the changes there should be ported here. Then close as won't fix.

• Update doctests as needed

• ???

Fetching Markus's GAP sources:

git clone git@github.com:markuspf/gap.git $LIBGAP
cd $LIBGAP
git remote add markuspf git@github.com:markuspf/gap.git
git fetch markuspf
git checkout -b markuspf/hpc-merge-libgap
./autogen.sh
./configure
make bootstrap-pkg-minimal

Testing libgap:

./configure --enable-libgap
make -j4 libgap
make test-libgap

Build and install a tardist for Sage, and rebuild the spkg:

make distclean
./autogen.sh
./configure
make manuals
make clean

(cd ..; tar zcvf $SAGE/upstream/$GAP.tar.gz --exclude .git $GAP)

sage --package fix-checksum
sage -f gap                      # -s

Header files:

• Copy GAP's header files, as well as gen/config.h to $SAGE/local/include
• Fix them to adapt the include path: #include <src/...> -> #include <gap/...>
• Replace T_INT by 0 in TNUM_OBJ, around line 414 of objects.h

Run:

sage -b

Basic tests on libgap:

sage: libgap.eval("GAPInfo.Version")
sage: libgap.DihedralGroup(10).CharacterTable()
CharacterTable( <pc group of size 10 with 2 generators> )
sage: libgap.Group(libgap.eval("[(1,2,3),(1,2)]")).Size()
6

Running most relevant tests:

sage -tp 8 sage/groups sage/libs/gap

Current status: all tests pass!

Testing packages with dynamic loading (e.g. IO):

Install IO:

cd $SAGE/local/gap/latest/pkg
wget http://www.gap-system.org/pub/gap/gap4/tar.gz/packages/io-4.4.6.tar.gz
tar xvf /tmp/io-4.4.6.tar.gz
mv io-4.4.6 io
cd io
./configure
make

Test it locally:

cd ../..
./gap -l .
gap> LoadPackage("IO");
true

This does not yet work:

sage: libgap.LoadPackage("IO")
ValueError: libGAP: Error, module '/opt/sage-git/local/gap/latest/pkg/io/bin/x86_64-pc-linux-gnu-gcc-default64/io.so' not found

This should be fixed once GAP's gap binary is built on top of libgap. See: ​https://github.com/markuspf/gap/issues/1.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1c645e0	#22626: remove GAP's symbol prefixing in libgap: libGAP_Foo -> Foo
​1ecc67b	#22626: doctest update w.r.t. minor changes of output in GAP
​e8ebbca	#22626: GMP detection patch for cygwin should not be needed anymore
​fd65b06	#22626: Remove libgap spkg
​9511733	#22626: replace patch for GAP's startup script template in favor of a custom script
​550625e	#22626: remove GAP's symbol prefixing in libgap: libGAP_Foo -> Foo, and workaround GAP <-> Python symbol conflict
​a9c859c	#22626: updated gap spkg w.r.t. GAP's devel version and its new build system; also include compilation and installation of libgap
​3011ac0	Merge branch 'develop' into t/22626/upgrade_to_gap_4_9

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​234c54b

#22626: revert #19726 as it won't be needed for gap 4.9

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b301e4c	#22626: ReST fix
​431845f	#22626: better work around for the GAP-Python name clash on T_INT, ...

[nthiery]

New commits:

​b301e4c	#22626: ReST fix
​431845f	#22626: better work around for the GAP-Python name clash on T_INT, ...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7c04025

22626: document the work around for the GAP-Python name clash on T_INT, ...

[fbissey]

So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code. Do we have an idea on when we'll have just have gap linking on libgap which would also solve this particular problem?

[nthiery]

Salut François,

Replying to fbissey:

So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code.

Exactly.

Do we have an idea on when we'll have just have gap linking on libgap which would also solve this particular problem?

The GAP developers are well aware of that and planning to implement it. They have not yet done yet just to be more incremental; their new build system is a big PR already :-) I would assume that this will be done before GAP 4.9 which they are planning for a couple months from now.

See: ​https://github.com/markuspf/gap/issues/2

Btw: feel free to comment / expand on the list there. I'll advertise this issue for additional feedback on the sage-packaging mailing list.

[jdemeyer]

Replying to fbissey:

So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code.

For this reason, I wonder why you didn't keep the separation GAP + libGAP in Sage too. If it effectively behaves as two packages, it seems more natural to keep it as two packages in Sage too. Imagine for example that we need to patch GAP but not libGAP or conversely, that would be harder with the current approach. That being said, this is mostly bikeshedding. So, if the current setup works well, there might be no reason to change it.

[dimpase]

Replying to jdemeyer:

Replying to fbissey:

So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code.

For this reason, I wonder why you didn't keep the separation GAP + libGAP in Sage too. If it effectively behaves as two packages, it seems more natural to keep it as two packages in Sage too. Imagine for example that we need to patch GAP but not libGAP or conversely, that would be harder with the current approach. That being said, this is mostly bikeshedding. So, if the current setup works well, there might be no reason to change it.

Two completely separate packages would also mean that gap_packages should come in two different flavours, etc. I'd rather keep it simple and do not multiply these instances (but rather hope that libGAP will support all of GAP packages soon).

[jdemeyer]

Replying to dimpase:

Two completely separate packages would also mean that gap_packages should come in two different flavours, etc.

So, you are saying that we need to install gap_packages twice too? Once for GAP and once for libGAP?

[nthiery]

Replying to jdemeyer:

For this reason, I wonder why you didn't keep the separation GAP + libGAP in Sage too.

I wanted to experiment with how far we were from having a proper single package, so as to give early feedback to the gap developers on any sticking points. And indeed, linking gap to libgap is basically the only sticking point. Since this point will be most likely resolved by the time this ticket gets merged in, we might as well shoot directly for the "right thing".

[dimpase]

Replying to jdemeyer:

Replying to dimpase:

Two completely separate packages would also mean that gap_packages should come in two different flavours, etc.

So, you are saying that we need to install gap_packages twice too? Once for GAP and once for libGAP?

no, not at all---I mean to say that there exist GAP packages (not part of Sage ATM) that break libGAP now. As some of these packages are very useful, it's important to have this fixed (but not at expense of having 2 separate gap_packages).

[nthiery]

Replying to dimpase:

As some of these packages are very useful, it's important to have this fixed

YES. In fact that is my main motivation to work on the libgap integration: I badly want to use Semigroups that requires IO! We went through the details with Markus and Max, and the shared aim is indeed to have this resolved with the upgrade to 4.9.

[jdemeyer]

Is there any news here?

[slelievre]

GAP 4.9.0, a beta release for GAP 4.9, has been released.

This means we can start working on building Sage with it.

• Release notes: ​https://github.com/gap-system/gap/wiki/GAP-4.9-release-notes

[dimpase]

Do you know if they now provide a replacement for libGAP?

[slelievre]

Cc-ing @alexk and @markuspf.

[nthiery]

Replying to dimpase:

Do you know if they now provide a replacement for libGAP?

In principle yes: the branch I posted here last April was using their stock alpha version. Now the next step is to revive the branch and adapt it if needed.

I can work on this, but probably not at once. Any volunteer to take over welcome.

Markus: do you foresee any major changes in how GAP's lib is to be built that could affect what I was doing last April?

[dimpase]

I just noticed that in 4.9 smallgrps and transgrps are released under a GPL-compatible license. This is great! We really should make them (i.e. database_gap) standard Sage package, and do away with the extra hurdle of having database_gap optional.

[jdemeyer]

If you're willing to wait 2 months, this might be a good task for the Cernay workshop ​https://github.com/OpenDreamKit/OpenDreamKit/issues/251

[nthiery]

Replying to jdemeyer:

If you're willing to wait 2 months, this might be a good task for the Cernay workshop ​https://github.com/OpenDreamKit/OpenDreamKit/issues/251

Yes indeed; it will be helpful to have GAP people under hand. I may just become impatient before that for some research project of mine, in which case I'll have a head start :-)

[nthiery]

Replying to dimpase:

I just noticed that in 4.9 smallgrps and transgrps are released under a GPL-compatible license. This is great! We really should make them (i.e. database_gap) standard Sage package, and do away with the extra hurdle of having database_gap optional.

Yes! I can't wait for all the simplifications this will bring to us!

[jdemeyer]

Replying to fbissey:

So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages".

With the past and future modifications to the Sage build system in mind, I now object more strongly than before to treating two separate packages (GAP + libGAP) as one package.

The reason is that we now try to separate the build and install stages of packages. But that is incompatible with the recipe here of first "build and install GAP" and then "build and install libGAP".

[jdemeyer]

For an example of two Sage packages with the same sources, see gcc and gfortran. IMHO, you should do the same for GAP and libGAP (unless they are merged upstream but my impression is that this has not happened yet).

[fbissey]

Replying to jdemeyer:

Replying to fbissey:

So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages".

With the past and future modifications to the Sage build system in mind, I now object more strongly than before to treating two separate packages (GAP + libGAP) as one package.

The reason is that we now try to separate the build and install stages of packages. But that is incompatible with the recipe here of first "build and install GAP" and then "build and install libGAP".

Hopefully we will be getting something more serious at final release time, like with pari where you build libpari and then the gp executable. That would be the ideal scenario where it is really one package. Have to see how far they have gone with 4.9.

[jdemeyer]

A lot of changes on this ticket (also the changes likely to lead to merge conflicts) are related to the unprefixing of libGAP symbols. To ease development, maybe we should try to do that separately from the upgrade to GAP 4.9.

I was thinking to do something like #19915 but changing only the Sage source code (not libGAP) and undoing the prefixing with macros like #define SomeGapFunction libGAP_SomeGapFunction. What do you think?

[dimpase]

there is also libgap unprefixing in Python/Cython? that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.

[jdemeyer]

Replying to dimpase:

there is also libgap unprefixing in Python/Cython? that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.

Yes, that is what I meant. Creating a separate ticket for unprefixing only at the Cython level by using macros to translate between the unprefixed names in Cython and the prefixed names in libGAP.

[dimpase]

Replying to jdemeyer:

Replying to dimpase:

there is also libgap unprefixing in Python/Cython? that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.

Yes, that is what I meant. Creating a separate ticket for unprefixing only at the Cython level by using macros to translate between the unprefixed names in Cython and the prefixed names in libGAP.

I am not sure I understand. Currently in Cython on can do something like

    from sage.libs.gap.libgap import libgap
    g=libgap.ProjectiveGeneralLinearGroup(3,3)

How will this change under what you propose?

[jdemeyer]

Replying to dimpase:

    from sage.libs.gap.libgap import libgap
    g=libgap.ProjectiveGeneralLinearGroup(3,3)

How will this change under what you propose?

The Python interface won't change at all. I am talking about how sage.libs.gap calls libGAP.

[jdemeyer]

As far as I can tell, GAP 4.9 has not been released. Does anybody know how close we are to an actual release?

[slelievre]

I think GAP 4.9.0 is considered a public beta for GAP 4.9.

The list of releases at

• ​https://www.gap-system.org/Releases/

still displays GAP 4.8.10 as the latest release.

In the list of past releases there, the GAP 4.8, 4.7, 4.6, 4.5 series start at GAP 4.8.2, 4.7.2, 4.6.2, 4.5.4, so probably GAP 4.x.y with those x and lower y were considered beta.

Still, seeing how 4.9.0 works with Sage would be nice.

[jdemeyer]

Where is the GAP source tarball which is supposed to be used here?

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​72b5066	Unprefixed libGAP interface
​b7278a1	Upgrade to GAP 4.9

[dimpase]

Replying to jdemeyer:

Where is the GAP source tarball which is supposed to be used here?

here: ​https://www.gap-system.org/pub/gap/gap-4.9/beta/