Opened 5 years ago
Last modified 4 years ago
#22626 closed enhancement
Upgrade to GAP 4.9 — at Version 55
Reported by: | nthiery | Owned by: | |
---|---|---|---|
Priority: | critical | Milestone: | sage-8.6 |
Component: | interfaces | Keywords: | days85, libgap |
Cc: | alexk, dimpase, embray, fbissey, arojas, gh-sebasguts, jpflori, markuspf, nthiery, slelievre, vbraun, wstein, gh-timokau | Merged in: | |
Authors: | Nicolas M. Thiéry, ... | Reviewers: | |
Report Upstream: | N/A | Work issues: | Wait for gap 4.9 release |
Branch: | u/jdemeyer/upgrade_to_gap_4_9 (Commits, GitHub, GitLab) | Commit: | b7278a120c5db710d1e11297b3dd1411d69d302b |
Dependencies: | #25273 | Stopgaps: |
Description (last modified by )
GAP 4.9 will come with a completely rewritten build system that will simplify our packaging. In fact, it may well enable Sage to use a vanilla GAP installation as provided by the distribution. It will also enable building GAP as a library, so that we can get rid of our separate libgap package.
The branch attached to this ticket updates Sage to run on top of a branch of GAP by Markus Pfeiffer that adds libgap compilation and will be merged soon in the devel version of GAP.
See https://github.com/markuspf/gap/issues/2 for the few sticking points that could prevent using a vanilla GAP from the distribution (please edit further if you think about more of them).
What the branch does:
- Remove the libgap spkg
- Update the gap spkg to the new build system and build and install libgap
- Replace
gap.shi.patch
by a plain gap startup script for Sage
Rationale: GAP used to provide a startup shell script. The GAP devs are in the process of getting rid of it and provide a very minimal one. They recommend to just write our own rather than patching it.
- Update a few doctests w.r.t. changes of output of some GAP functions
- Possibly controversial: The new libgap currently *does not come*
with symbol rewriting (
foobar
->libGAP_foobar
). This avoids messing around with GAP's sources; in particular opening the door for using a stock GAP from the OS distribution. However there always is a risk of name conflict. And indeed, GAP's constants (actually cpp macros) T_INT, T_FLOAT, ... conflict with Python's constants. This is currently worked around by forcing the inclusion of Python'sstructmember.h
before the gap headers.
Something similar was started by Volker at #19915.
- Removes configure.patch: it was patching configure for better GMP detection under Cygwin (#13954). This should not be needed anymore with the new build system and use of --with-gmp. If it is, upstream asked for it to be reported and they will fix it.
- Revert #19726 (not needed anymore)
Status: Most long test pass. Tentatively, the 38 remaining failing tests are due to changes in GAP since 4.8.6: Max mentioned that the library has been cleaned up to always use the same random generation source, and some of the group algorithms were changed as well, which can explain, e.g. change of orders in lists of elements. So those should be nothing to worry about. There is not much point in updating those doctests right away; we may as well wait for a more final version of 4.9 to be out.
TODO:
- Automatic handling of headers (see below for how to do it by hand). GAP's build system will eventually provide a rule to install headers which will make this trivial.
- Use GAP's own
make install
when it will be implemented.
- Check against #19915 to see if any of the changes there should be ported here. Then close as won't fix.
- Update doctests as needed
- ???
Fetching Markus's GAP sources:
git clone git@github.com:markuspf/gap.git $LIBGAP cd $LIBGAP git remote add markuspf git@github.com:markuspf/gap.git git fetch markuspf git checkout -b markuspf/hpc-merge-libgap ./autogen.sh ./configure make bootstrap-pkg-minimal
Testing libgap:
./configure --enable-libgap make -j4 libgap make test-libgap
Build and install a tardist for Sage, and rebuild the spkg:
make distclean ./autogen.sh ./configure make manuals make clean (cd ..; tar zcvf $SAGE/upstream/$GAP.tar.gz --exclude .git $GAP) sage --package fix-checksum sage -f gap # -s
Header files:
- Copy GAP's header files, as well as gen/config.h to $SAGE/local/include
- Fix them to adapt the include path: #include <src/...> -> #include <gap/...>
- Replace T_INT by 0 in TNUM_OBJ, around line 414 of objects.h
Run:
sage -b
Basic tests on libgap:
sage: libgap.eval("GAPInfo.Version") sage: libgap.DihedralGroup(10).CharacterTable() CharacterTable( <pc group of size 10 with 2 generators> ) sage: libgap.Group(libgap.eval("[(1,2,3),(1,2)]")).Size() 6
Running most relevant tests:
sage -tp 8 sage/groups sage/libs/gap
Current status: all tests pass!
Testing packages with dynamic loading (e.g. IO):
Install IO:
cd $SAGE/local/gap/latest/pkg wget http://www.gap-system.org/pub/gap/gap4/tar.gz/packages/io-4.4.6.tar.gz tar xvf /tmp/io-4.4.6.tar.gz mv io-4.4.6 io cd io ./configure make
Test it locally:
cd ../.. ./gap -l . gap> LoadPackage("IO"); true
This does not yet work:
sage: libgap.LoadPackage("IO") ValueError: libGAP: Error, module '/opt/sage-git/local/gap/latest/pkg/io/bin/x86_64-pc-linux-gnu-gcc-default64/io.so' not found
This should be fixed once GAP's gap binary is built on top of libgap. See: https://github.com/markuspf/gap/issues/1.
Change History (56)
comment:1 Changed 5 years ago by
- Description modified (diff)
comment:2 Changed 5 years ago by
- Description modified (diff)
comment:3 Changed 5 years ago by
- Description modified (diff)
comment:4 Changed 5 years ago by
- Description modified (diff)
comment:5 Changed 5 years ago by
- Description modified (diff)
comment:6 Changed 5 years ago by
- Branch set to u/nthiery/upgrade_to_gap_4_9
comment:7 Changed 5 years ago by
- Commit set to 3011ac0908d667c0f245ca21859e336511106b5f
comment:8 Changed 5 years ago by
- Description modified (diff)
comment:9 Changed 5 years ago by
- Description modified (diff)
comment:10 Changed 5 years ago by
- Description modified (diff)
comment:11 Changed 5 years ago by
- Description modified (diff)
comment:12 Changed 5 years ago by
- Description modified (diff)
comment:13 Changed 5 years ago by
- Cc vbraun dimpase wstein added
- Keywords days85 libgap added
- Work issues set to Wait for gap 4.9 release
comment:14 Changed 5 years ago by
- Description modified (diff)
comment:15 Changed 5 years ago by
- Cc fbissey added
comment:16 Changed 5 years ago by
- Commit changed from 3011ac0908d667c0f245ca21859e336511106b5f to 234c54b4b7e0495e343c3ab1b925e2c99f37e391
Branch pushed to git repo; I updated commit sha1. New commits:
234c54b | #22626: revert #19726 as it won't be needed for gap 4.9
|
comment:17 Changed 5 years ago by
- Description modified (diff)
Changed 5 years ago by
comment:18 Changed 5 years ago by
- Description modified (diff)
comment:19 Changed 5 years ago by
- Commit changed from 234c54b4b7e0495e343c3ab1b925e2c99f37e391 to 431845f6da27d20c78b5e7a42b5f47bea866201c
comment:20 Changed 5 years ago by
- Description modified (diff)
comment:21 Changed 5 years ago by
- Commit changed from 431845f6da27d20c78b5e7a42b5f47bea866201c to 7c04025083d61cab671df3302c32f353c4e28313
Branch pushed to git repo; I updated commit sha1. New commits:
7c04025 | 22626: document the work around for the GAP-Python name clash on T_INT, ...
|
comment:22 follow-ups: ↓ 23 ↓ 24 ↓ 40 Changed 5 years ago by
So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code. Do we have an idea on when we'll have just have gap linking on libgap which would also solve this particular problem?
comment:23 in reply to: ↑ 22 Changed 5 years ago by
Salut François,
Replying to fbissey:
So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code.
Exactly.
Do we have an idea on when we'll have just have gap linking on libgap which would also solve this particular problem?
The GAP developers are well aware of that and planning to implement it. They have not yet done yet just to be more incremental; their new build system is a big PR already :-) I would assume that this will be done before GAP 4.9 which they are planning for a couple months from now.
See: https://github.com/markuspf/gap/issues/2
Btw: feel free to comment / expand on the list there. I'll advertise this issue for additional feedback on the sage-packaging mailing list.
comment:24 in reply to: ↑ 22 ; follow-ups: ↓ 25 ↓ 27 Changed 5 years ago by
Replying to fbissey:
So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code.
For this reason, I wonder why you didn't keep the separation GAP + libGAP in Sage too. If it effectively behaves as two packages, it seems more natural to keep it as two packages in Sage too. Imagine for example that we need to patch GAP but not libGAP or conversely, that would be harder with the current approach. That being said, this is mostly bikeshedding. So, if the current setup works well, there might be no reason to change it.
comment:25 in reply to: ↑ 24 ; follow-up: ↓ 26 Changed 5 years ago by
Replying to jdemeyer:
Replying to fbissey:
So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages" with the same source code.
For this reason, I wonder why you didn't keep the separation GAP + libGAP in Sage too. If it effectively behaves as two packages, it seems more natural to keep it as two packages in Sage too. Imagine for example that we need to patch GAP but not libGAP or conversely, that would be harder with the current approach. That being said, this is mostly bikeshedding. So, if the current setup works well, there might be no reason to change it.
Two completely separate packages would also mean that gap_packages
should come in two different flavours, etc. I'd rather keep it simple and do not multiply these instances (but rather hope that libGAP
will support all of GAP packages soon).
comment:26 in reply to: ↑ 25 ; follow-up: ↓ 28 Changed 5 years ago by
Replying to dimpase:
Two completely separate packages would also mean that
gap_packages
should come in two different flavours, etc.
So, you are saying that we need to install gap_packages
twice too? Once for GAP and once for libGAP?
comment:27 in reply to: ↑ 24 Changed 5 years ago by
Replying to jdemeyer:
For this reason, I wonder why you didn't keep the separation GAP + libGAP in Sage too.
I wanted to experiment with how far we were from having a proper single package, so as to give early feedback to the gap developers on any sticking points. And indeed, linking gap to libgap is basically the only sticking point. Since this point will be most likely resolved by the time this ticket gets merged in, we might as well shoot directly for the "right thing".
comment:28 in reply to: ↑ 26 ; follow-up: ↓ 29 Changed 5 years ago by
Replying to jdemeyer:
Replying to dimpase:
Two completely separate packages would also mean that
gap_packages
should come in two different flavours, etc.So, you are saying that we need to install
gap_packages
twice too? Once for GAP and once for libGAP?
no, not at all---I mean to say that there exist GAP packages (not part of Sage ATM) that break libGAP now. As some of these packages are very useful, it's important to have this fixed (but not at expense of having 2 separate gap_packages
).
comment:29 in reply to: ↑ 28 Changed 5 years ago by
Replying to dimpase:
As some of these packages are very useful, it's important to have this fixed
YES. In fact that is my main motivation to work on the libgap integration: I badly want to use Semigroups that requires IO! We went through the details with Markus and Max, and the shared aim is indeed to have this resolved with the upgrade to 4.9.
comment:30 Changed 5 years ago by
- Cc jpflori added
comment:31 Changed 5 years ago by
Is there any news here?
comment:32 Changed 5 years ago by
- Cc slelievre added
GAP 4.9.0, a beta release for GAP 4.9, has been released.
This means we can start working on building Sage with it.
comment:33 follow-up: ↓ 35 Changed 5 years ago by
Do you know if they now provide a replacement for libGAP
?
comment:34 Changed 5 years ago by
- Cc alexk markuspf added
- Description modified (diff)
Cc-ing @alexk and @markuspf.
comment:35 in reply to: ↑ 33 Changed 5 years ago by
Replying to dimpase:
Do you know if they now provide a replacement for
libGAP
?
In principle yes: the branch I posted here last April was using their stock alpha version. Now the next step is to revive the branch and adapt it if needed.
I can work on this, but probably not at once. Any volunteer to take over welcome.
Markus: do you foresee any major changes in how GAP's lib is to be built that could affect what I was doing last April?
comment:36 follow-up: ↓ 39 Changed 4 years ago by
I just noticed that in 4.9 smallgrps and transgrps are released under a GPL-compatible license. This is great! We really should make them (i.e. database_gap) standard Sage package, and do away with the extra hurdle of having database_gap optional.
comment:37 follow-up: ↓ 38 Changed 4 years ago by
If you're willing to wait 2 months, this might be a good task for the Cernay workshop https://github.com/OpenDreamKit/OpenDreamKit/issues/251
comment:38 in reply to: ↑ 37 Changed 4 years ago by
Replying to jdemeyer:
If you're willing to wait 2 months, this might be a good task for the Cernay workshop https://github.com/OpenDreamKit/OpenDreamKit/issues/251
Yes indeed; it will be helpful to have GAP people under hand. I may just become impatient before that for some research project of mine, in which case I'll have a head start :-)
comment:39 in reply to: ↑ 36 Changed 4 years ago by
Replying to dimpase:
I just noticed that in 4.9 smallgrps and transgrps are released under a GPL-compatible license. This is great! We really should make them (i.e. database_gap) standard Sage package, and do away with the extra hurdle of having database_gap optional.
Yes! I can't wait for all the simplifications this will bring to us!
comment:40 in reply to: ↑ 22 ; follow-up: ↓ 42 Changed 4 years ago by
Replying to fbissey:
So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages".
With the past and future modifications to the Sage build system in mind, I now object more strongly than before to treating two separate packages (GAP + libGAP) as one package.
The reason is that we now try to separate the build and install stages of packages. But that is incompatible with the recipe here of first "build and install GAP" and then "build and install libGAP".
comment:41 Changed 4 years ago by
For an example of two Sage packages with the same sources, see gcc
and gfortran
. IMHO, you should do the same for GAP and libGAP (unless they are merged upstream but my impression is that this has not happened yet).
comment:42 in reply to: ↑ 40 Changed 4 years ago by
Replying to jdemeyer:
Replying to fbissey:
So if I understand the mechanics involved, at the moment you have to configure and install gap. Then clean and configure and install for libgap. So it is still behaving like "two packages".
With the past and future modifications to the Sage build system in mind, I now object more strongly than before to treating two separate packages (GAP + libGAP) as one package.
The reason is that we now try to separate the build and install stages of packages. But that is incompatible with the recipe here of first "build and install GAP" and then "build and install libGAP".
Hopefully we will be getting something more serious at final release time, like with pari where you build libpari and then the gp executable. That would be the ideal scenario where it is really one package. Have to see how far they have gone with 4.9.
comment:43 Changed 4 years ago by
A lot of changes on this ticket (also the changes likely to lead to merge conflicts) are related to the unprefixing of libGAP symbols. To ease development, maybe we should try to do that separately from the upgrade to GAP 4.9.
I was thinking to do something like #19915 but changing only the Sage source code (not libGAP) and undoing the prefixing with macros like #define SomeGapFunction libGAP_SomeGapFunction
. What do you think?
comment:44 follow-up: ↓ 45 Changed 4 years ago by
there is also libgap unprefixing in Python/Cython? that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.
comment:45 in reply to: ↑ 44 ; follow-up: ↓ 46 Changed 4 years ago by
Replying to dimpase:
there is also libgap unprefixing in Python/Cython? that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.
Yes, that is what I meant. Creating a separate ticket for unprefixing only at the Cython level by using macros to translate between the unprefixed names in Cython and the prefixed names in libGAP.
comment:46 in reply to: ↑ 45 ; follow-up: ↓ 47 Changed 4 years ago by
Replying to jdemeyer:
Replying to dimpase:
there is also libgap unprefixing in Python/Cython? that has to be done. Perhaps separating these from C level unprefixing would help making it smoother.
Yes, that is what I meant. Creating a separate ticket for unprefixing only at the Cython level by using macros to translate between the unprefixed names in Cython and the prefixed names in libGAP.
I am not sure I understand. Currently in Cython on can do something like
from sage.libs.gap.libgap import libgap g=libgap.ProjectiveGeneralLinearGroup(3,3)
How will this change under what you propose?
comment:47 in reply to: ↑ 46 Changed 4 years ago by
Replying to dimpase:
from sage.libs.gap.libgap import libgap g=libgap.ProjectiveGeneralLinearGroup(3,3)How will this change under what you propose?
The Python interface won't change at all. I am talking about how sage.libs.gap
calls libGAP.
comment:48 Changed 4 years ago by
As far as I can tell, GAP 4.9 has not been released. Does anybody know how close we are to an actual release?
comment:49 Changed 4 years ago by
I think GAP 4.9.0 is considered a public beta for GAP 4.9.
The list of releases at
still displays GAP 4.8.10 as the latest release.
In the list of past releases there, the GAP 4.8, 4.7, 4.6, 4.5 series start at GAP 4.8.2, 4.7.2, 4.6.2, 4.5.4, so probably GAP 4.x.y with those x and lower y were considered beta.
Still, seeing how 4.9.0 works with Sage would be nice.
comment:50 Changed 4 years ago by
- Dependencies set to #25273
comment:51 follow-up: ↓ 54 Changed 4 years ago by
Where is the GAP source tarball which is supposed to be used here?
comment:52 Changed 4 years ago by
- Branch changed from u/nthiery/upgrade_to_gap_4_9 to u/jdemeyer/upgrade_to_gap_4_9
comment:53 Changed 4 years ago by
- Commit changed from 7c04025083d61cab671df3302c32f353c4e28313 to b7278a120c5db710d1e11297b3dd1411d69d302b
comment:54 in reply to: ↑ 51 Changed 4 years ago by
Replying to jdemeyer:
Where is the GAP source tarball which is supposed to be used here?
comment:55 Changed 4 years ago by
- Description modified (diff)
Branch pushed to git repo; I updated commit sha1. New commits:
#22626: remove GAP's symbol prefixing in libgap: libGAP_Foo -> Foo
#22626: doctest update w.r.t. minor changes of output in GAP
#22626: GMP detection patch for cygwin should not be needed anymore
#22626: Remove libgap spkg
#22626: replace patch for GAP's startup script template in favor of a custom script
#22626: remove GAP's symbol prefixing in libgap: libGAP_Foo -> Foo, and workaround GAP <-> Python symbol conflict
#22626: updated gap spkg w.r.t. GAP's devel version and its new build system; also include compilation and installation of libgap
Merge branch 'develop' into t/22626/upgrade_to_gap_4_9