Opened 7 years ago

Closed 15 months ago

#13426 closed enhancement (fixed)

Improve gap_packages

Reported by: vbraun Owned by: tbd
Priority: major Milestone: sage-8.3
Component: packages: optional Keywords: days85
Cc: dimpase, mmarco, vbraun Merged in:
Authors: Travis Scrimshaw Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 951e7e7 (Commits) Commit: 951e7e7345103ff6836f379131afa1a5aedcd838
Dependencies: Stopgaps:

Description (last modified by tscrim)

We should include more packages into GAP. This ticket adds the following:

  • cohomolo
  • CoReLG
  • GBNP
  • hecke
  • liealgdb
  • LiePRing
  • LieRing
  • loops
  • mapclass
  • qpa
  • quagroup
  • repsn
  • sla

new tarball

Change History (36)

comment:1 Changed 7 years ago by dimpase

  • Description modified (diff)

comment:2 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:3 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:4 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:5 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:6 Changed 2 years ago by tscrim

  • Milestone changed from sage-6.4 to sage-7.6

See also #22190 and #22244.

comment:7 Changed 2 years ago by tscrim

  • Authors set to Travis Scrimshaw
  • Branch set to public/packages/increase_gap_packages-22244
  • Commit set to 951e7e7345103ff6836f379131afa1a5aedcd838
  • Description modified (diff)
  • Status changed from new to needs_review

New commits:

951e7e7Added new packages to gap_packages.

comment:8 Changed 2 years ago by tscrim

  • Description modified (diff)
  • Keywords days85 added

comment:9 Changed 2 years ago by dimpase

to quote email by Alex K. on a GAP developers list:

In pull request https://github.com/gap-system/gap/pull/1135
I have attempted to reduce a list of packages loaded by default to

[ "autpgrp", "ctbllib", "factint", "fga", "irredsol", "tomlib" ],

With dependencies, this results in the following set of
packages being loaded at startup:

AtlasRep 1.5.1, AutPGrp 1.6, Browse 1.8.6, CRISP 1.4.4, CTblLib 1.2.2,
FactInt 1.5.3, FGA 1.3.1, GAPDoc 1.5.1, IO 4.4.6, IRREDSOL 1.3.1,
SpinSym 1.5, TomLib 1.2.6

so we should aim to have as many from the latter list as possible. (packages that need GAP kernel extensions, like IO, are not possible to support ATM, as this will need changes in libGAP).

comment:10 Changed 2 years ago by fbissey

AtlasRep is problematic in global installs. Debian has a package for it that does something but I may miss some bits that they put in gap itself. The problem is that AtlasRep fetch data from the net and tries to dump it where the package is installed. If you are a regular user and gap is installed in location belonging to root you are in trouble.

AtlasRep really need to dump data in ~/.gap.

comment:11 Changed 2 years ago by tscrim

To note, we already have (some form of) AtlasRep, AutPGrp, CTblLib, FactInt included in gap_packages.

The changes to libGAP that will allow IO should be handled in #22626 (or at least be doable afterwards). However, I don't want to hold up the addition of some other packages, a number of which are related to my research (I'm already using QuaGroup in #22623). So I'd appreciate it if we could push off making gap_packages "perfect" to a later ticket.

comment:12 Changed 2 years ago by tscrim

  • Milestone changed from sage-7.6 to sage-8.1

Let me reformulate that better as a question: What would you want to see included that currently is not?

comment:13 Changed 2 years ago by fbissey

I will take a more provocative stance.

You can install gap packages for personal use or testing in ~/.gap/pkg and they will be picked up automatically. This may need to be better documented but it means that we only need to care about packages that are difficult to install or require building against a library or executable found under SAGE_LOCAL. The rest is just a matter of unpacking a tarball in the right place. It is even easier than installing a R package.

comment:14 Changed 2 years ago by tscrim

If I understand you correctly, you want to restrict gap_packages to be just those packages and are "difficult," and let the "easy" packages be left to the individual user. I have no philosophical objections to this, but there are a few technical things that need to be addressed. (Documentation will, of course, depend on precisely what we do).

How do we want to treat the GAP packages wrt doctesting? Will they be treated as optional (i.e., automatically tested against if installed), experimental, or something else? #22623 might be a good test case (I need to add more doctests anyways).

Do we want a hook as part of sage -i or some other optional sage argument to help ease installation? Or will the "official" way to install a GAP package is to manually extract the tarball that a user downloads?

What will we do about the packages that are previously installed by gap_packages that would no longer be the case? I know this will not affect people with them currently installed, but imagine someone builds a new version from scratch (or after a make distclean). Then suddenly a particular optional GAP package does not work. So it becomes backwards incompatible in a way (unless we have Sage automatically download it when trying to call said package, but there might be a legal issue here too).

comment:15 Changed 2 years ago by fbissey

Note that I preceded my comment with the word "provocative" (instant success). More seriously:

  • there is a small list of package we may want to provide by default, we sort of already do. It is impossible to start gap without the gapdoc package being available for example.
  • if a gap package is essential for some sage functionality we should ship it as standard.
  • The spkg we currently have for gap packages and database are for convenience and are already optional. A ticket like this one wants to basically extend the convenience. Mainly in my opinion because people do not know how to install gap packages on their own. Gap upstream is not particularly verbose on that matter. They ship all the packages to you after all.

Documentation alone won't make the issue go away. Raising awareness of what you can do may achieve something. The mechanism to install R packages is well know. While debian seems to want to package CRAN we don't. People using R know what to do when they need a R packages. Gap users don't and I think that's the point that really need to be addressed.

comment:16 follow-up: Changed 2 years ago by tscrim

The bigger thing for me right now is QuaGroup. I got an e-mail where I could point towards #22623, but had some issues with telling them how to get the QuaGroup in a robust way.

I agree completely with points 1 and 2. For point 3, I feel that gap_packages is the closest thing that Sage has to an "official" way to install optional GAP pacakges because I think Sage tells users " you do not have to know anything about the underlying pacakges". So this would be a change to the official way, but I think in a good way.

Some other technical questions:

  • What if we want a GAP package to only be for one version of Sage, what should the official way to install it (do we even care)?
  • On my system, I do not have a ~/.gap folder (of course, I can create one), but I do not have a non-Sage copy of GAP. Do we want this in ~/.gap or some subfolder of ~/.sage (in particular, ~/.sage/gap)? At least, I don't think your proposal will work because:
    sage: gap.console()
    gap> Print( GAPInfo.RootPaths, "\n"); 
    [ "/home/travis/sage-build/local/gap/latest/" ]
    
  • How do we get users to run gap_reset_workspace() (IIRC it is necessary)?

comment:17 in reply to: ↑ 16 ; follow-up: Changed 2 years ago by fbissey

Replying to tscrim:

The bigger thing for me right now is QuaGroup. I got an e-mail where I could point towards #22623, but had some issues with telling them how to get the QuaGroup in a robust way.

I agree completely with points 1 and 2. For point 3, I feel that gap_packages is the closest thing that Sage has to an "official" way to install optional GAP pacakges because I think Sage tells users " you do not have to know anything about the underlying pacakges". So this would be a change to the official way, but I think in a good way.

Some other technical questions:

  • What if we want a GAP package to only be for one version of Sage, what should the official way to install it (do we even care)?

I am not sure we can enforce that. At the very least the version of gap associated with that version of sage would need to be unique.

  • On my system, I do not have a ~/.gap folder (of course, I can create one), but I do not have a non-Sage copy of GAP. Do we want this in ~/.gap or some subfolder of ~/.sage (in particular, ~/.sage/gap)? At least, I don't think your proposal will work because:
    sage: gap.console()
    gap> Print( GAPInfo.RootPaths, "\n"); 
    [ "/home/travis/sage-build/local/gap/latest/" ]
    

After inspection, it seems to be related to how gap is started from sage. That will need to be inspected.

fbissey@moonloop ~/sandbox/git-fork/sage $ ./sage -sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/fbissey/sandbox/git-fork/sage
(sage-sh) fbissey@moonloop:sage$ gap
 ┌───────┐   GAP 4.8.6, 12-Nov-2016, build of 2017-07-06 10:42:48 (NZST)
 │  GAP  │   http://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-gcc-default64
 Libs used:  gmp, readline
 Loading the library and packages ...
 Packages:   GAPDoc 1.5.1
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> Print( GAPInfo.RootPaths, "\n");
[ "/home/fbissey/.gap/", "/home/fbissey/sandbox/git-fork/sage/local/gap/latest/" ]

Related to your previous question, RootPaths? can be customised (and obviously it is done in sage already), so you could imagine adding a unique location for that version of sage.

  • How do we get users to run gap_reset_workspace() (IIRC it is necessary)?

I am aware of that problem and I am not sure. I wonder if upstream sees this as a problem or if it is particular to the way sage interface with gap.

comment:18 in reply to: ↑ 17 ; follow-ups: Changed 2 years ago by tscrim

Replying to fbissey:

Replying to tscrim:

  • What if we want a GAP package to only be for one version of Sage, what should the official way to install it (do we even care)?

I am not sure we can enforce that. At the very least the version of gap associated with that version of sage would need to be unique.

I guess we already have this sort of problem with multiple Sage versions and init.sage being in ~/.sage. So putting something in ~/.sage would not exacerbate the problem.

  • On my system, I do not have a ~/.gap folder (of course, I can create one), but I do not have a non-Sage copy of GAP. Do we want this in ~/.gap or some subfolder of ~/.sage (in particular, ~/.sage/gap)? At least, I don't think your proposal will work because:
    sage: gap.console()
    gap> Print( GAPInfo.RootPaths, "\n"); 
    [ "/home/travis/sage-build/local/gap/latest/" ]
    

After inspection, it seems to be related to how gap is started from sage. That will need to be inspected.

fbissey@moonloop ~/sandbox/git-fork/sage $ ./sage -sh
[snip]
gap> Print( GAPInfo.RootPaths, "\n");
[ "/home/fbissey/.gap/", "/home/fbissey/sandbox/git-fork/sage/local/gap/latest/" ]

Doing this, I get:

gap> Print( GAPInfo.RootPaths, "\n");
[ "/home/travis/.gap/", "/home/travis/sage-build/local/gap/latest/" ]

However, via gap.console() might be considered a special way of starting gap, but it is something I think we will need to address because

sage: %gap

  --> Switching to Gap <--

gap: Print( GAPInfo.RootPaths, "\n");
[ "/home/travis/sage-build/local/gap/latest/" ]

sage: gap('GAPInfo.RootPaths')
[ "/home/travis/sage-build/local/gap/latest/" ]

So within Sage calls to gap, it seems to use that version.

Related to your previous question, RootPaths? can be customised (and obviously it is done in sage already), so you could imagine adding a unique location for that version of sage.

I propose ~/.sage/gap.

I'm guessing anything installed as proposed would also be available via libgap. At least via gap_packages in #22623 I was able to make calls via libgap. (Granted, I think this will be made moot when we go to the next gap version).

  • How do we get users to run gap_reset_workspace() (IIRC it is necessary)?

I am aware of that problem and I am not sure. I wonder if upstream sees this as a problem or if it is particular to the way sage interface with gap.

I have no idea. At least if we had some way to install this using something like sage -i, we could run it as part of our installation process.

comment:19 in reply to: ↑ 18 ; follow-up: Changed 2 years ago by fbissey

Replying to tscrim:

Replying to fbissey:

Replying to tscrim:

  • What if we want a GAP package to only be for one version of Sage, what should the official way to install it (do we even care)?

I am not sure we can enforce that. At the very least the version of gap associated with that version of sage would need to be unique.

I guess we already have this sort of problem with multiple Sage versions and init.sage being in ~/.sage. So putting something in ~/.sage would not exacerbate the problem.

I may have misunderstood you. You meant a gap package only available to sage, while with your wording I understood sage-X.Y. Is it correct?

So far I haven't been able to dig where sage fiddle with the gap RootPaths?.

comment:20 in reply to: ↑ 19 Changed 2 years ago by tscrim

Replying to fbissey:

Replying to tscrim:

Replying to fbissey:

Replying to tscrim:

  • What if we want a GAP package to only be for one version of Sage, what should the official way to install it (do we even care)?

I am not sure we can enforce that. At the very least the version of gap associated with that version of sage would need to be unique.

I guess we already have this sort of problem with multiple Sage versions and init.sage being in ~/.sage. So putting something in ~/.sage would not exacerbate the problem.

I may have misunderstood you. You meant a gap package only available to sage, while with your wording I understood sage-X.Y. Is it correct?

Let me try to be more precise. Suppose I have two versions of sage, X and Y. I want to be able to install gap packages only available to X but not to Y.

Actually, thinking about this more, I don't think this would be a problem as if a gap package doesn't work with a particular version of gap, then it will fail normally. Plus there are probably better ways to compare two versions of a package. So I don't think this is really necessary.

comment:21 Changed 2 years ago by dimpase

A clean way to resolve this seems to be to pass the token upstream: ask them for a variant of gap -i facility to install GAPs packages.

I have opened up an issue #1499 on GAP's github repo.

Last edited 2 years ago by dimpase (previous) (diff)

comment:22 in reply to: ↑ 18 Changed 2 years ago by jdemeyer

Replying to tscrim:

At least if we had some way to install this using something like sage -i, we could run it as part of our installation process.

-1 to use sage -i to install packages in ~/.sage/gap or some variant of that. I believe that there should be a clear difference between installing packages inside Sage itself and inside some user directory.

comment:23 Changed 2 years ago by fbissey

OK after some rather intricate tracking I found why RootPaths is different in sage from outside sage: https://github.com/sagemath/sage/blob/master/src/sage/interfaces/gap.py#L200

The "-r" parameter passed to gap has the effect to make it ignore any .gaprc file and set IgnoreGapRC. But a side effect is that if IgnoreGapRC is set

src/system.c:        if (!IgnoreGapRC) {
src/system.c-          SySetGapRootPath(DotGapPath);
src/system.c-        }

.gap is not added to RootPaths.

comment:24 follow-ups: Changed 2 years ago by tscrim

What we would be doing is setting a custom place for the Sage version of gap packages, and and so we would need a custom root path passed in via gap -l <path>.

@jdemeyer I somewhat see this as installing a Sage package, just within the gap part of Sage. However, I am not really convinced of the feasibility of sage -i as we would need all of the metadata for the gap packages and mirrors and other stuff that (IMO) is not really worth dealing with. I can deal with installing it in a specific folder (or making that the official installation method). Yet I see ~/.sage as being something (mildly) hidden, and so we shouldn't need to tell users to go into there. IDK. What would be your preference?

Upstream does seem amenable to having gap -i, but not enough people with the time and/or skill to properly implement it. Also, we would want to have gap install it to a specified place, which would not be where gap is installed for Sage as they would get removed on upgrade. I would like to have something more workable in the near future.

comment:25 in reply to: ↑ 24 ; follow-ups: Changed 2 years ago by jdemeyer

Replying to tscrim:

What would be your preference?

What's wrong with keeping the current $SAGE_LOCAL/gap/latest? We install every Sage package in $SAGE_LOCAL, why would GAP packages be an exception?

comment:26 in reply to: ↑ 25 Changed 2 years ago by dimpase

Replying to jdemeyer:

Replying to tscrim:

What would be your preference?

What's wrong with keeping the current $SAGE_LOCAL/gap/latest? We install every Sage package in $SAGE_LOCAL, why would GAP packages be an exception?

Note that pip --user would install Python modules somewhere else.

comment:27 in reply to: ↑ 24 ; follow-up: Changed 2 years ago by dimpase

Replying to tscrim:

Upstream does seem amenable to having gap -i, but not enough people with the time and/or skill to properly implement it. Also, we would want to have gap install it to a specified place, which would not be where gap is installed for Sage as they would get removed on upgrade. I would like to have something more workable in the near future.

I'm going to work on gap -i; it appears to align well with the current ODK grant goals.

comment:28 in reply to: ↑ 25 Changed 2 years ago by tscrim

Replying to jdemeyer:

Replying to tscrim:

What would be your preference?

What's wrong with keeping the current $SAGE_LOCAL/gap/latest? We install every Sage package in $SAGE_LOCAL, why would GAP packages be an exception?

The problem as I understand it is that $SAGE_LOCAL/gap/latest is just a symlink and changes on each upgrade. So suppose you install a package P in gap-X, then you upgrade (in the process of upgrading Sage) to gap-Y. Then $SAGE_LOCAL/gap/latest points to gap-Y, which does not contain P, and your code that uses the package suddenly doesn't work. With the current gap_packages, this is not a problem because our current framework automatically updates that as well. Or am I misunderstanding something?

However, I was asking about your preference for how to tell users to install optional gap packages? I guess your proposal it to tell them untar in $SAGE_LOCAL/gap/latest and make that the official way (rather than on a mostly hidden sage wiki page).

comment:29 in reply to: ↑ 27 Changed 2 years ago by tscrim

Replying to dimpase:

Replying to tscrim:

Upstream does seem amenable to having gap -i, but not enough people with the time and/or skill to properly implement it. Also, we would want to have gap install it to a specified place, which would not be where gap is installed for Sage as they would get removed on upgrade. I would like to have something more workable in the near future.

I'm going to work on gap -i; it appears to align well with the current ODK grant goals.

Great. I would offer to help, but I have neither the time nor expertise to contribute much. However, let me know if there is anything I can do. I guess if that is in the works, then we can simply tell users to install in a specific place until that method is realized as a workaround.

comment:30 Changed 16 months ago by mantepse

What is the status of this ticket? It appears to work, and I'd like to have qpa easily accessible. Can I have some cake?

comment:31 Changed 16 months ago by dimpase

  • Milestone changed from sage-8.1 to sage-8.3
  • Reviewers set to Dima Pasechnik

So these added packages don't do any GAP kernel modules, right? Just checking---otherwise it looks good.

comment:32 Changed 16 months ago by tscrim

I believe so. The only thing that needs to be compiled is cohomolo (https://www.gap-system.org/Packages/cohomolo.html), but that is because of some extra C bindings IIRC.

comment:33 Changed 16 months ago by dimpase

  • Status changed from needs_review to positive_review

OK, send it to the bots then. I hope they can get the tar file.

comment:34 Changed 15 months ago by tscrim

  • Cc vbraun added

Volker, is there a reason why this ticket did not get merged in the past beta?

comment:35 Changed 15 months ago by tscrim

  • Dependencies #13211 deleted

comment:36 Changed 15 months ago by vbraun

  • Branch changed from public/packages/increase_gap_packages-22244 to 951e7e7345103ff6836f379131afa1a5aedcd838
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.