Add experimental gap3_jm package

[stumpc5]

We have been trying to have an optional GAP3 package for a while now. Instead of the official GAP3, we instead use a forked package ​gap3_jm which has regularly been updated.

Previously, this was made a Sage package (#8906) but it contained binaries, so it was removed again (#19164).

I removed all binaries, and prepared a new tarball at ​​http://downloads.findstat.org/gap3-jm5-2015-02-01.tar.gz .

[stumpc5]

How do I actually find all binaries? Such as

gap3_test$ grep -r -m 1 "^"  . | grep "^Binary file"
Binary file ./pkg/monoid/doc/manual.dvi matches
Binary file ./pkg/anusq/bin/Sq matches
Binary file ./pkg/anusq/doc/sq94.dvi matches
Binary file ./pkg/anusq/doc/sqn92.dvi matches
Binary file ./pkg/nq/bin/nq matches
Binary file ./pkg/cryst/doc/paper.dvi matches
Binary file ./pkg/meataxe/tests/m11.2 matches
Binary file ./pkg/meataxe/tests/Mat9 matches
Binary file ./pkg/meataxe/tests/a2 matches
Binary file ./pkg/meataxe/tests/a1 matches
Binary file ./pkg/meataxe/tests/Mat256 matches
Binary file ./pkg/meataxe/tests/Perm2 matches
Binary file ./pkg/meataxe/tests/Mat25 matches
Binary file ./pkg/meataxe/tests/Mat125 matches
Binary file ./pkg/meataxe/tests/ac.2 matches
Binary file ./pkg/meataxe/tests/Mat67 matches
Binary file ./pkg/meataxe/tests/Perm1 matches
Binary file ./pkg/meataxe/tests/Mat5 matches
Binary file ./pkg/meataxe/tests/C0.2 matches
Binary file ./pkg/meataxe/tests/C0.1 matches
Binary file ./pkg/meataxe/tests/Mat2 matches
Binary file ./pkg/meataxe/tests/ac.1 matches
Binary file ./pkg/meataxe/tests/m11.1 matches
Binary file ./pkg/sisyphos/bin/sis-i586-unknown-linux matches
Binary file ./pkg/sisyphos/doc/sisman.0.8.3.dvi matches
Binary file ./pkg/ve/examples/meout.pb matches
Binary file ./pkg/ve/bin/qme matches
Binary file ./pkg/ve/bin/me matches
Binary file ./pkg/ve/docs/nme.dvi matches
Binary file ./pkg/arep/bin/leonin matches
Binary file ./pkg/arep/bin/desauto matches
Binary file ./pkg/arep/bin/leonout matches
Binary file ./pkg/arep/src/leon/doc/manual.dvi matches
Binary file ./manual.pdf matches
Binary file ./bin/gapdjg.exe matches
Binary file ./bin/gap.mac matches
Binary file ./bin/gap.x86linux64 matches
Binary file ./bin/gapcyg.exe matches
Binary file ./bin/gap.macx86 matches
Binary file ./bin/gap.x86linux matches
Binary file ./bin/gapdjg.pif matches
Binary file ./bin/cygwin1.dll matches

Are any of these okay to keep, such as the dvi files?

[jdemeyer]

Replying to stumpc5:

• Download GAP3 from ​https://webusers.imj-prg.fr/~jean.michel/gap3/

I think this is the problem: it doesn't seem to be the "official" GAP3 package, just some random repackaging of GAP3.

[stumpc5]

Replying to jdemeyer:

I think this is the problem: it doesn't seem to be the "official" GAP3 package, just some random repackaging of GAP3.

The development of GAP3 has stopped a long time ago. But there are several GAP3 packages that have never been ported to GAP4, and which are still developed.

From that webpage:

To help people who are just interested in GAP3 because they need a package
which has not been ported to GAP4, I have prepared an easy-to install
minimal GAP3 distribution containing an up-to-date versions of the packages.

So this is the only place where we can find up-to-date versions of the GAP3 package *chevie* to work with unitary reflection groups and their representation theory.

I fail to see what might be wrong with adding support for those packages using that "random repackaging" since I and several others need them.

I install the packages myself and then use them from within Sage, but since several things inside Sage are about to depend on that package - see #11187, #11010, #10819 - I think it is worth providing the option for others to automatically install that version of GAP3 with the newest *chevie* package.

[jdemeyer]

Replying to stumpc5:

Replying to jdemeyer:

I think this is the problem: it doesn't seem to be the "official" GAP3 package, just some random repackaging of GAP3.

The development of GAP3 has stopped a long time ago. But there are several GAP3 packages that have never been ported to GAP4, and which are still developed.

From that webpage:

To help people who are just interested in GAP3 because they need a package
which has not been ported to GAP4, I have prepared an easy-to install
minimal GAP3 distribution containing an up-to-date versions of the packages.

All the above doesn't explain why you use the repackaged version and not the official GAP 3 release from ​http://www.gap-system.org/Gap3/Download3/download.html

So this is the only place where we can find up-to-date versions of the GAP3 package *chevie* to work with unitary reflection groups and their representation theory.

No, the official gap3r4p4 release also contains chevie.

I fail to see what might be wrong with adding support for those packages using that "random repackaging" since I and several others need them.

Because the repackaging seems to made in a messy way, see #19164.

[jdemeyer]

Frank Lübeck's page ​How to get CHEVIE also says to download the official GAP3 release and doesn't talk about the "repackaged" version.

[stumpc5]

Replying to jdemeyer:

Frank Lübeck's page ​How to get CHEVIE also says to download the official GAP3 release and doesn't talk about the "repackaged" version.

Neither this nor the above is up-to-date. Afaik, Jean Michel is the only person still actively developing *chevie* and his website is the only place where these new versions can be found. E.g., I found a small bug in chevie last month (so years after the release of gap3r4p4) through my Sage usage -- Jean fixed it within a few hours and posted the new version on his website.

Using any other version of gap3 and chevie will result in running into old mistakes that will slow down my understanding of research problems I compute!

[jdemeyer]

Jean fixed it within a few hours and posted the new version on his website.

With "new version", I guess you mean a new version of the repackaged GAP3?

It should be possible to just get the up-to-date chevie sources without all the extra clutter.

[stumpc5]

From my email discussion with Jean Michel:

>> These  days I sometimes change files in other places than Chevie, so indeed
>> each  time I change something one must take the whole distribution (however
>> I  almost  never  change  the  C  sources).

> Is the stuff that you change in other places needed for chevie to run?

Well,  yes. There were  patches to the  library loaded when loading Chevie.
Since  I maintain the whole distribution, I  decided it makes more sense to
*do*  these  patches  to  the  library  once  and  for  all (this could fix
undiscovered  bugs in other  packages). So the  correct behaviour of Chevie
depends on having my version of gap3/lib.

@Jeroen, what do you think how to proceed from here?

[jdemeyer]

First of all, thanks for the clarifications!

I think that somebody (either Jean Michel himself or somebody on this Sage ticket) needs to package a source-only repackaged GAP3.

Further questions to be asked to Jean Michel:

1. Could he state more clearly on his website that his version is not just a repackaged GAP3, it's really a fork of GAP3 with additional fixes which are not in GAP3 proper.

2. Can he provide a clean source-only repackaged GAP3?

3. If no to the previous question: is he willing to host such a source-only repackaged GAP3 on his website, if it is created by some Sage contributor?

[stumpc5]

Replying to jdemeyer:

I think that somebody (either Jean Michel himself or somebody on this Sage ticket) needs to package a source-only repackaged GAP3.

I don't understand how this differs from my initial suggestion on this ticket.

[jdemeyer]

Replying to stumpc5:

I don't understand how this differs from my initial suggestion on this ticket.

It does not differ. It's just that the situation has been clarified, making sure that your initial suggestion is the right thing to do.

[stumpc5]

Dear Christian, I cannot post directly on sage-trac so I reply to you.

Further questions to be asked to Jean Michel:

1. Could he state more clearly on his website that his version is not

just a repackaged GAP3, it's really a fork of GAP3 with additional fixes which are not in GAP3 proper.

OK, I will try to be clear on my webpages. The fact is I do not distribute a complete GAP3 distribution (all packages, all libraries) to save space. Perhaps I should distribute a complete distribution and this would obsolete the St Andrews distribution -- it would go from 15MB to 70MB.

2. Can he provide a clean source-only repackaged GAP3?

I could try if I understand exactly what this means. But this should certainly not be the default distribution since my distribution contains Mac and Windows executables (and executables for some packages which have executables separate from the gap3 executable). We thus have to agree to a name for this for-Sage distribution which I will keep separate from the default distribution.

3. If no to the previous question: is he willing to host such a source-

only repackaged GAP3 on his website, if it is created by some Sage contributor?

Yes.

[jdemeyer]

Replying to stumpc5:

I could try if I understand exactly what this means.

It means: delete all binaries (i.e. system-specific) from the package. Make sure that the package has a build system such that it can be built/installed in a standard Unix environment.

We thus have to agree to a name for this for-Sage distribution which I will keep separate from the default distribution.

Well, I wouldn't call it a "for-Sage distribution". Call it a "source distribution". The actual filename doesn't matter much, most likely we will rename it for Sage anyway.

[stumpc5]

Replying to jdemeyer:

It means: delete all binaries (i.e. system-specific) from the package. Make sure that the package has a build system such that it can be built/installed in a standard Unix environment.

I will try to do that.

[jdemeyer]

Also (this is mostly a comment for Jean Michel): ideally the binary distribution would be made from the source distribution, not the other way around.

[stumpc5]

I prepared a first version by

1. taking the tarball from 2016-02-01 from Jean Michel's website
2. deleting all binaries that were listed using grep -r -m 1 "^" . | grep "Binary file"
3. putting the result into a new tarball with the name gap3-jm5-2015-02-01.tar.gz, this can be downloaded at ​https://drive.google.com/file/d/0B1POEh8bhq1fLXdfTEZkZjVLOGs/view?usp=sharing
4. preparing an optional package using this upstream tarball. This was basically a blind trip through the documentation for creating packages + looking at the old optional package patch #8906.

@jdemeyer and @tscrim: I would very much appreciate if you could have a look!

If the linked tarball is okay, I would discuss with Jean Michel whether he could provide such a source distribution directly on his website.

---

New commits:

​5483a58

first working version, wandering in the darkness though

[jdemeyer]

This is far from an exhaustive list:

+if [[ $UNAME == 'Linux' ]]; then
+    if [[ $(uname -m) == 'x86_64' ]]; then
+        MAKE_COMMAND='x86linux-gcc64'
+    else
+        MAKE_COMMAND='x86linux-gcc'
+    fi;
+elif [[ $UNAME == 'Darwin' ]]; then
+        MAKE_COMMAND='macosx-gcc'
+fi

Is gap3 only supported on those architectures? In that case, it will need to be an experimental package.

[jdemeyer]

You should check for errors in spkg-install.

[stumpc5]

Replying to jdemeyer:

This is far from an exhaustive list:

I get

gap3_source/src$ make
usage: 'make <target>' where target is one of
'x86linux-gcc64'   for x86linux with gcc
'x86linux-gcc'     for x86linux with gcc in 32bit mode
'macosx-gcc'       for mac os x with gcc
'x86bsd-gcc'       for x86BSD with gcc
'sun-sparc-solaris-gcc' for SUN under Solaris with GNU cc
'sun-sparc-solaris-cc'  for SUN under Solaris with cc
'bsd'		       for others under Berkeley UNIX with cc
	
	  additional C compiler and linker flags can be passed with
	  'make <target> COPTS=<compiler-opts> LOPTS=<linker-opts>',
	  i.e., 'make x86linux-gcc COPTS=-g LOPTS=-g for debug'

and the GAP3 download page (​http://www.gap-system.org/Gap3/Download3/download.html) does not say anything about other hardware.

Can you tell me how to extend that if ... elif ... fi statement to also support these other options -- and if that is enough for the package to be optional rather than experimental?

Many thanks!

[jdemeyer]

Personally, I think it is better to have a look at the Makefile in gap3_jm and see why it so platform-dependent. Because there is very likely no fundamental reason why the platform should be given explicitly.

[stumpc5]

The critical part of the Makefile is

x86linux-gcc:
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -m32 -O2 -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
	@$(MAKE) gap CC=gcc CFLAGS="$(COPTS) -m32 -O " SYS_FILE=system.o LOPTS="$(LOPTS) -m32 -static"
	strip gap
	mv gap ../bin/gap.x86linux

x86linux-gccd:
	@$(MAKE) system.o   CC=gcc  CFLAGS="-g -m32 -O2 -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
	@$(MAKE) gap CC=gcc CFLAGS="-g -m32 -O " SYS_FILE=system.o LOPTS="-g -m32 -static"

x86linux-gcc64:
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -O -DSYS_IS_64_BIT -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
	@$(MAKE) gap CC=gcc CFLAGS="$(COPTS) -O -DSYS_IS_64_BIT " SYS_FILE=system.o LOPTS="$(LOPTS) -static"
	strip gap
	mv gap ../bin/gap.x86linux64

x86-dos-djgpp:
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -DSYS_IS_MSDOS_DJGPP -DSYS_HAS_MISC_PROTO"
	@$(MAKE) gapdjg.exe CC=gcc  CFLAGS="$(COPTS) -O1"  SYS_FILE=system.o  LOPTS="$(LOPTS)"

x86-dos-djgppcross:
	@$(MAKE) system.o   CC=dos-gcc	CFLAGS="$(COPTS) -DSYS_IS_MSDOS_DJGPP -DSYS_HAS_MISC_PROTO"
	@$(MAKE) gapdjg.exe CC=dos-gcc	CFLAGS="$(COPTS) -O1"  SYS_FILE=system.o  LOPTS="$(LOPTS)"

# sbrk doesn't work in macosx so we need to use vm_allocate
macosx-gcc:
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -m32 -DSYS_IS_MACOSX -DARCH_INCLUDE -DSYS_HAS_IOCTL_PROTO"
	@$(MAKE) gap	    CC=gcc  CFLAGS="$(COPTS) -m32 -O2"  SYS_FILE=system.o  LOPTS="$(LOPTS) -m32"

# sbrk doesn't work in macosx so we need to use vm_allocate
# This is using Frank Luebeck's optimized compile....but it doesn't
# change the GAPstones for the files in ../tst
macosx-gcc-686-optimized:
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -m32 -fomit-frame-pointer -pipe -fno-strength-reduce -march=i686 -falign-loops=2 -falign-jumps=2 -falign-functions=2 -DCPU=686 -g -O2 -DSYS_IS_MACOSX -DARCH_INCLUDE -DSYS_HAS_IOCTL_PROTO"
	@$(MAKE) gap	    CC=gcc  CFLAGS="$(COPTS) -m32 -fomit-frame-pointer -pipe -fno-strength-reduce -march=i686 -falign-loops=2 -falign-jumps=2 -falign-functions=2 -DCPU=686 -g -O3"  SYS_FILE=system.o  LOPTS="$(LOPTS) -m32"

sun-sparc-solaris-cc:
	@$(MAKE) system.o   CC=cc  CFLAGS="$(COPTS) -O -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSOLARIS2"
	@echo "Don't worry about 'out of range' and 'overflow' warnings"
	@echo "(29 in total)"
	@$(MAKE) gap	    CC=cc  CFLAGS="$(COPTS) -O2"  SYS_FILE=system.o  LOPTS="$(LOPTS)"

sun-sparc-solaris-gcc2:						   
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS)  -O6 -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSOLARIS2"						       
	@$(MAKE) gap	    CC=gcc  CFLAGS="$(COPTS)  -O6"  SYS_FILE=system.o LOPTS="$(LOPTS)"								     

# 'sys/times.h' claims 'times' returns 'clock_t' (how shall it return -1?)
sun-sparc-sunos-gcc:
	@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO"
	@$(MAKE) gap	    CC=gcc  CFLAGS="$(COPTS) -O2"  SYS_FILE=system.o  LOPTS="$(LOPTS)"

# 'sys/times.h' claims 'times' returns 'clock_t' (how shall it return -1?)
sun-sparc-sunos-cc:
	@$(MAKE) system.o   CFLAGS="$(COPTS) -DSYS_IS_USG -DSYS_HAS_TIME_PROTO"
	@$(MAKE) gap	    CFLAGS="$(COPTS) -O"  SYS_FILE=system.o  LOPTS="$(LOPTS)"

bsd:
	@$(MAKE) system.o   CC=$(CC)  CFLAGS="$(COPTS) -DSYS_IS_BSD"
	@$(MAKE) gap	    CC=$(CC)  CFLAGS="$(COPTS) -O"  SYS_FILE=system.o  LOPTS="$(LOPTS)"

I have no idea what of that is necessary and what can be simplified... Do you know?

[jdemeyer]

Does this Makefile come from gap upstream or from gap3_jm?

To start with, I would remove these: mv gap ../bin/gap.x86linux. Then at least the name of the executable won't be system-dependent.

Second, are the x86linux-gcc and x86linux-gcc64 rules really x86-specific? At first sight, this doesn't seem to be the case: I think it is just 32-bit Linux and 64-bit Linux.

[stumpc5]

This Makefile comes from gap3_jm, the Makefile in the last official version is this + even more options.

To start with, I would remove these: mv gap ../bin/gap.x86linux. Then at least the name of the executable won't be system-dependent.

Okay, no problem.

Second, are the x86linux-gcc and x86linux-gcc64 rules really x86-specific? At first sight, this doesn't seem to be the case: I think it is just 32-bit Linux and 64-bit Linux.

Is there a way to find that out? I compiled GAP3 for the first time ever to get this optional package.

[jdemeyer]

Replying to stumpc5:

Is there a way to find that out?

Assume that it works like I said and we'll see if it breaks.

[stumpc5]

I wanted to modify the Makefile accordingly, but it didn't work along the lines

ARCH = $(shell getconf LONG_BIT)
UNAME_S = $(shell uname -s)

ifeq ($(UNAME_S),Linux)
    ifeq ($(ARCH),32)
        # do foo
    endif
    ifeq ($(ARCH),64)
        # do bar
    endif
endif
ifeq ($(UNAME_S),Darwin)
    # do foobar
endif

unknown:
    ...

but I got many different type of errors that I wasn't able to resolve. Some were caused by tabs vs. spaces, others by the recursive calling of the makefile. Could you help me getting it correct?

[stumpc5]

Replying to stumpc5:

I wanted to modify the Makefile accordingly, but it didn't work along the lines

Okay, I think I got it now, I had to further mess with tabs vs spaces... I am going to upload a new version soon.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ce54879

modified the makefile to autodetect the system Linux/Darwin and 32/64 bit

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2ae234f

added explanation in the header of the makefile patch

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​696eaeb

added description of the patches to SPKG.txt

[stumpc5]

I took care of the two suggestions by Jeroen and now think it is ready for review -- many thanks!

[jdemeyer]

spkg-check doesn't actually do anything...

Does GAP3 have a testsuite?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8e0e38a

added the memory for gap3 depending on 32/64 bit systems

[dimpase]

I installed it once on OSX, then I decided to redo it with SAGE_CHECK=yes, and got

###########################################################
#  Building complete, you can run GAP3 from './../bin/gap3'
###########################################################
Installing (copying) files...
cp: symlink: me: File exists
cp: symlink: qme: File exists
Error installing (copying) gap3 files.

This comes from copying executables of ve package, me and qme somewhere; probably by the ve installation script.

you need to use the corresponding flag, -f, to avoid this trouble.

[dimpase]

It would also be great if you host the tarball on a "real" server. Some browsers would automatically untar this file, and this is no good. So I had to get the tarball on a Linux machine, and copy it to the OSX machine for testing (oh the joy of this...)

And the release manager will be very, very unhappy, too, without a wget-able link for the tarball.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​83d8906

added -r option to the cp command

[stumpc5]

Replying to dimpase:

Thanks for the testing!

I installed it once on OSX, then I decided to redo it with SAGE_CHECK=yes

I was not able to reproduce the error, but I added the -f flag to the cp command. Does that solve the issue?

And the release manager will be very, very unhappy, too, without a wget-able link for the tarball.

that's very true, I changed the download accordingly.

[stumpc5]

Replying to jdemeyer:

I only copied the skpg-check from the first version of the gap3 package. What should I change there -- or can I even remove it?

[jdemeyer]

Does GAP3 have a testsuite?

[stumpc5]

I am not sure, but I don't think so.

[jdemeyer]

If not, then remove spkg-check.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​138dcac

removed spkg-check

[stumpc5]

done.

---

New commits:

​138dcac

removed spkg-check

[dimpase]

there is some weirdness with installation of GAP packages. There are packages that need to be compiled, e.g. ve. Also, there some weird dangling symlinks in ve/bin:

...pkg/ve/bin$ ls -l
total 48
lrwxr-xr-x  1 dima  admin    2 Mar  9 08:38 me.exe -> me
-rw-r--r--  1 dima  admin  115 Aug  3  2014 me.sgl
-rw-r--r--  1 dima  admin  804 Aug  3  2014 me.sh
lrwxr-xr-x  1 dima  admin    3 Mar  9 08:38 qme.exe -> qme
-rw-r--r--  1 dima  admin  773 Aug  3  2014 qme.sh
-rw-r--r--  1 dima  admin  773 Aug  3  2014 zme.sh

when I see "building complete" message, only the main GAP system was built, but no packages. Then I see

mv gap ../bin/gap3
###########################################################
#  Building complete, you can run GAP3 from './../bin/gap3'
###########################################################
Installing (copying) files...
cp: pkg/ve/bin/me.exe: No such file or directory
cp: pkg/ve/bin/qme.exe: No such file or directory
Error installing (copying) gap3 files.

I also notice that gcc is called with option -m32 on the 64-bit OSX 10.11:

...
gcc -m32 -O2   -c -o coding.o coding.c
gcc -m32 -o gap gap.o system.o gasman.o scanner.o idents.o read.o eval.o integer.o rational.o cyclotom.o unknown.o finfield.o polynom.o permutat.o word.o costab.o tietze.o agcollec.o aggroup.o pcpresen.o list.o plist.o set.o vector.o vecffe.o range.o blister.o string.o record.o statemen.o function.o coding.o
ld: warning: ignoring file /Volumes/Sage/sage/local/lib/libgcc_ext.10.5.dylib, missing required architecture i386 in file /Volumes/Sage/sage/local/lib/libgcc_ext.10.5.dylib (1 slices)

and this looks wrong.

[jdemeyer]

I think it is best to never use -m32 or -m64 options and use the GCC default. That is how most other packages are compiled and that seems to work fine.

[dimpase]

Replying to jdemeyer:

I think it is best to never use -m32 or -m64 options and use the GCC default. That is how most other packages are compiled and that seems to work fine.

that makefile is 20+ years old, with minor tweaks only, you know...

And as I already said, it does not build packages. (Even on GAP4 it is not the case). Each package that needs something compiled has its own makefile, and one needs to run make in each of the corresponding subdirectories in pkg/.

[jdemeyer]

Replying to dimpase:

Replying to jdemeyer:

I think it is best to never use -m32 or -m64 options and use the GCC default. That is how most other packages are compiled and that seems to work fine.

that makefile is 20+ years old, with minor tweaks only, you know...

That doesn't really matter. It's trivial to just remove -m32 and -m64 everywhere.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6b1b7fb

removed the -m32 options from the make

[stumpc5]

to jdemeyer:

That doesn't really matter. It's trivial to just remove -m32 and -m64 everywhere.

I updated the Makefile and removed all -m32 options. The two makes for linux are now

@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -O -DSYS_IS_64_BIT -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
@$(MAKE) gap CC=gcc CFLAGS="$(COPTS) -O -DSYS_IS_64_BIT " SYS_FILE=system.o LOPTS="$(LOPTS) -static"

vs

@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -O2 -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
@$(MAKE) gap CC=gcc CFLAGS="$(COPTS) -O " SYS_FILE=system.o LOPTS="$(LOPTS) -static"

Do you know if I can also remove the -O -DSYS_IS_64_BIT vs -O2 option to completely get rid of the distinction between 32 and 64 bit versions ?

[stumpc5]

Replying to dimpase:

there is some weirdness with installation of GAP packages. There are packages that need to be compiled, e.g. ve. Also, there some weird dangling symlinks in ve/bin:

Jean Michel wrote about deleting the binaries of the packages:

You can get rid of all of them since they are little-used packages, or the
binaries provide additionnal functionality and are not essential.

So I decided to not touch that part -- if the functionality is wanted at some point, we can still try to figure how to handle them properly...

[dimpase]

Replying to stumpc5:

to jdemeyer:

That doesn't really matter. It's trivial to just remove -m32 and -m64 everywhere.

I updated the Makefile and removed all -m32 options. The two makes for linux are now

@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -O -DSYS_IS_64_BIT -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
@$(MAKE) gap CC=gcc CFLAGS="$(COPTS) -O -DSYS_IS_64_BIT " SYS_FILE=system.o LOPTS="$(LOPTS) -static"

vs

@$(MAKE) system.o   CC=gcc  CFLAGS="$(COPTS) -O2 -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO"
@$(MAKE) gap CC=gcc CFLAGS="$(COPTS) -O " SYS_FILE=system.o LOPTS="$(LOPTS) -static"

Do you know if I can also remove the -O -DSYS_IS_64_BIT vs -O2 option to completely get rid of the distinction between 32 and 64 bit versions ?

while -O2 is almost harmless (OK, the resulting executable might get slower), SYS_IS_64_BIT and other SYS_ variables are poor man's way to pass system parameters to the source code; if you just blindly remove them, you might easily get broken code, as the default might be, say, 32-bit, and this will break low-level stuff that is not platform-independent in this sense. (Have a look at the source, some of these variables are used very heavily).

A proper way to handle these flags is to figure out their values in spkg-install; you can certainly find tests for system being 32- or 64-bit in other spkg-install scripts.

Some other SYS_ things are clearly obsolete, on the other hand, i.e. they specify whether or not system has a feature certainly present on any system supported by Sage...

[stumpc5]

Replying to dimpase:

A proper way to handle these flags is to figure out their values in spkg-install; you can certainly find tests for system being 32- or 64-bit in other spkg-install scripts.

Some other SYS_ things are clearly obsolete, on the other hand, i.e. they specify whether or not system has a feature certainly present on any system supported by Sage...

Okay, I would then rather leave things as they are. Moving the testing to spkg-install might be cleaner, but the Makefile is designed to call itself and I would prefer not to mess with it even more.

[dimpase]

Replying to stumpc5:

Replying to dimpase:

there is some weirdness with installation of GAP packages. There are packages that need to be compiled, e.g. ve. Also, there some weird dangling symlinks in ve/bin:

Jean Michel wrote about deleting the binaries of the packages:

You can get rid of all of them since they are little-used packages, or the
binaries provide additionnal functionality and are not essential.

So I decided to not touch that part -- if the functionality is wanted at some point, we can still try to figure how to handle them properly...

well, "handle them properly" is not straightforward. Actually, I did port ve and anusq to modern world systems; I do use them in my research; the main issue was the reliance on an very old obsolete version of GMP. See ​http://github.com/gap-packages/ve and ​https://github.com/dimpase/anusq

And Simon King has a lot of experience with meataxe - maybe he can tell you how to deal with it.

(anusq, meataxe, and ve are all useless without their binary components).

[SimonKing]

Replying to dimpase:

And Simon King has a lot of experience with meataxe - maybe he can tell you how to deal with it.

My only experience is to provide an optional meataxe package for Sage. It does install libmtx.a and meataxe.h and the meataxe binaries. I don't know if that's enough for GAP to pick it up. Also note that I currently try to upgrade the optional meataxe package to fix some build issues --- see #20136.

[dimpase]

Replying to SimonKing:

Replying to dimpase:

And Simon King has a lot of experience with meataxe - maybe he can tell you how to deal with it.

My only experience is to provide an optional meataxe package for Sage. It does install libmtx.a and meataxe.h and the meataxe binaries. I don't know if that's enough for GAP to pick it up. Also note that I currently try to upgrade the optional meataxe package to fix some build issues --- see #20136.

they ship meataxe version 2.2.3. How far from this your meataxe is? In principle, GAP3 meataxe needs executables, that's all.

[SimonKing]

Replying to dimpase:

they ship meataxe version 2.2.3. How far from this your meataxe is?

Now that's ironic.

In my good old-style group cohomology spkg, I used meataxe-2.2.4, which is identical with version 2.2.3 except for its licence, which is about a decade old.

But old-style spkgs got broken on purpose, hence, I now try to create a new style cohomology spkg, and while I am at it I try to use a more recent version of meataxe. The current release (which is several years old) is version 2.4.24, and the current snapshot (which also includes one of my patches) is 2.5.0.

In principle, GAP3 meataxe needs executables, that's all.

What is the difference between 2.2.4 and 2.4.24? Well, the good news is that the executables kept their names, and the storage format has not changed. Hence, if GAP3 merely needs the executables then using the new meataxe should be fine.

The bad news --- for me! --- is that basically all function names and all type names have changed, the offset of counting matrix rows and columns has changed (in 2.2.4 it started counting with 1, now it starts with 0), and occasionally the semantics of the functions has changed as well. That's why creating a group cohomology package based on the new meataxe is a pain in the neck: I need to rebase someone else's C code, which thus is even more difficult than rebasing my own code.

Sigh.

[jdemeyer]

The tarball should extract to its own top-level directory, instead of having many top-level directories like doc and src.

And the strangest thing: the tarball even contains a broken copy of itself!

[jdemeyer]

The current tarball still contains binaries: src/*.o and bin/gap.x86linux64 and bin/gap3.

[jdemeyer]

This should go the description, it is not an Update/Build Instruction:

== Special Update/Build Instructions ==

This is a minimal GAP3 distribution containing packages that have
no equivalent in GAP4.

[jdemeyer]

You really need to document how you obtained the tarball. Ideally, it would just be a semi-official tarball from Jean Michel.

[jdemeyer]

You need to check for errors in spkg-install.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​af253f7

taking the requests of jdemeyer from 16/03/21 into account

[stumpc5]

Replying to jdemeyer:

The current tarball still contains binaries: src/*.o and bin/gap.x86linux64 and bin/gap3. And the strangest thing: the tarball even contains a broken copy of itself

I accidentally tar'ed the wrong folder when updating the tarball, so it was completely corrupted. I hope it is now the correct one without binaries or broken copies of itself...

You need to check for errors in spkg-install.

If you mean that the package didn't install, this was due to my wrong tarball. Otherwise, I don't know what you mean here.

The tarball should extract to its own top-level directory, instead of having many top-level directories like doc and src.

Done.

This should go the description, it is not an Update/Build Instruction:

Done.

[stumpc5]

Replying to jdemeyer:

You really need to document how you obtained the tarball. Ideally, it would just be a semi-official tarball from Jean Michel.

I have added it to the documentation. I used grep -r -m 1 "^" . | grep "^Binary file" to identify all binaries and then deleted them. This is the only difference to Jean Michel's tarball. At some point, it might be good to have this version semi-official from Jean.

[SimonKing]

Replying to stumpc5:

You need to check for errors in spkg-install.

If you mean that the package didn't install, this was due to my wrong tarball. Otherwise, I don't know what you mean here.

For example:

if [ "x$SAGE_CHECK" = xyes ]; then
    $MAKE check
else
    $MAKE install
fi

if [ $? -ne 0 ]; then
    echo >&2 "Error installing Foo."
    exit 1
fi

The check for errors is in the last "if"-clause. Without that text, the install script would report a successful installation of the package, even if building or testing it gave an error.

[SimonKing]

PS: That's my guess what is meant by "check for errors", I didn't look up what your spkg-install looks like.

[jdemeyer]

Yes, you need to ensure that, if $MAKE fails, then spkg-install exits with an error message. Usually this is done by something like

$MAKE
if [ $? -ne 0 ]; then
    echo >&2 "Error installing Foo."
    exit 1
fi

[jdemeyer]

Replying to stumpc5:

I accidentally tar'ed the wrong folder when updating the tarball, so it was completely corrupted.

This is why it's a good idea to script such tarball changes, then mistakes like this cannot happen. Usually this is done by a spkg-src script. For a good example, see build/pkgs/ecl/spkg-src after applying #13044.

I'm not saying that such a spkg-src is required for an optional package but it's at least recommended.

[jdemeyer]

Replying to stumpc5:

Replying to jdemeyer:

You really need to document how you obtained the tarball. Ideally, it would just be a semi-official tarball from Jean Michel.

I have added it to the documentation.

Good, but put this in the Special Update/Build Instructions block in SPKG.txt.

[jdemeyer]

The tarball looks good now.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cd1743c	checking for errors during make
​2f83382	added the decription of the tarball to the 'Special Update' section
​e4b56e6	preparing a spkg-src file

[stumpc5]

I think the spkg-src file is now working, but I still have two issues:

• I get a different checksum, even though I do not see what I did differently than before.
• The command grep -r -m 1 "^" . | grep "^Binary file" | xargs rm is not very clean because of the error

  rm: cannot remove ‘Binary’: No such file or directory
  rm: cannot remove ‘file’: No such file or directory
  rm: cannot remove ‘matches’: No such file or directory
  

  Do you know how to do that better?

Thanks!

[jdemeyer]

Replying to stumpc5:

I think the spkg-src file is now working, but I still have two issues:

• I get a different checksum, even though I do not see what I did differently than before.

Most likely different timestamps.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8257ea7

updated the checksum for the auto-generated tarball

[stumpc5]

Okay, the two tarballs contain the same files of the same sizes, so I expect spkg-src to actually do what it should. I uploaded the auto-generated tarball and updated the checksum.

[stumpc5]

I now have

patching file gap3/src/Makefile
Hunk #1 succeeded at 48 (offset 2 lines).
Hunk #2 succeeded at 108 (offset 2 lines).

that I need to fix.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aabc629

updated makefile patch fixing two hunks

[jdemeyer]

The tarball is gone (404: Not Found)

Also don't forget your author name.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​50aac0e

fixed one more spot where the folder structure of gap3 changed

[stumpc5]

I had to fix one more issue (I missed to update the gap3 folder structure in gap.sh). I also updated the tarball with several changes in Jean Michel's original tarball from the past few days.

I hope I have not forgot to update anything else, and mark it again as needs review -- thanks!

[stumpc5]

Asking the question about how to properly delete the binary files at ​http://stackoverflow.com/questions/36221028

[jdemeyer]

sage -t src/sage/interfaces/gap3.py
**********************************************************************
File "src/sage/interfaces/gap3.py", line 400, in sage.interfaces.gap3.Gap3._execute_line
Failed example:
    f([1,2,3])                                   #optional - gap3
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: Gap3 produced error output
    Error, List Element: <position> must be a positive integer at
    return L[0] ... in
    ... called from
    main loop
    brk> quit;
    ...
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.gap3.Gap3._execute_line[3]>", line 1, in <module>
        f([Integer(1),Integer(2),Integer(3)])                                   #optional - gap3
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 805, in __call__
        return getattr(P, self.name())(*args)
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 607, in __call__
        return self._parent.function_call(self._name, list(args), kwds)
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 921, in function_call
        res = self.eval(marker+cmd)
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 573, in eval
        result = Expect.eval(self, input_line, **kwds)
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1239, in eval
        for L in code.split('\n') if L != ''])
      File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 771, in _eval_line
        raise RuntimeError(message)
    RuntimeError: Gap3 produced error output
    Error, Function: <function> must be a function
    <BLANKLINE>
       executing __SAGE_LAST__:="__SAGE_LAST__";;x(sage0);;
**********************************************************************
File "src/sage/interfaces/gap3.py", line 447, in sage.interfaces.gap3.Gap3.help
Failed example:
    m                                        #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/interfaces/gap3.py", line 449, in sage.interfaces.gap3.Gap3.help
Failed example:
    m.Print()                                #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    "__SAGE_LAST__"
**********************************************************************
File "src/sage/interfaces/gap3.py", line 453, in sage.interfaces.gap3.Gap3.help
Failed example:
    m                                        #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/interfaces/gap3.py", line 455, in sage.interfaces.gap3.Gap3.help
Failed example:
    m.Print()                                #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    "__SAGE_LAST__"
**********************************************************************

[jdemeyer]

Given that this package contains chevie, I guess you should replace

# optional - chevie

by

# optional - gap3

[stumpc5]

There seems to be several problems with the GAP3 interface, partially described in #18971:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath Version 7.1.beta6, Release Date: 2016-03-03               │
│ Type "notebook()" for the browser-based notebook interface.        │
│ Type "help()" for help.                                            │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage:  
sage: gap3
Gap3
sage: gap3.execute("1+1")
''
sage: gap3.execute("1+1")
'2'
sage: 
sage: gap3.execute("1+1")
'2'
sage: gap3
Gap3
sage: gap3.eval('VERSION')[1:-1]
''
sage: gap3.eval('VERSION')[1:-1]
'lib: jm 12Mar2016, src: jm 28Feb2015, sys: usg gcc 64'
sage: gap3.execute("1+1")
'"lib: jm 12Mar2016, src: jm 28Feb2015, sys: usg gcc 64"'
sage: gap3.execute("1+1")
'2'

[stumpc5]

Replying to jdemeyer:

sage -t src/sage/interfaces/gap3.py

These doctests actually also did not work before either: I installed gap3 myself and made it start on the command-line using the command gap3. I then did

$ sage -t src/sage/interfaces/gap3.py 
too few successful tests, not using stored timings
Running doctests with ID 2016-03-25-15-51-57-0530368e.
Git branch: findstat/main
Using --optional=dot2tex,mpir,python2,sage,scons
Doctesting 1 file.
sage -t src/sage/interfaces/gap3.py
    [12 tests, 6.03 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 6.2 seconds
    cpu time: 0.2 seconds
    cumulative wall time: 6.0 seconds

and then

$ sage -t --optional=dot2tex,mpir,python2,sage,scons,gap3,chevie src/sage/interfaces/gap3.py 
too few successful tests, not using stored timings
Running doctests with ID 2016-03-25-15-52-25-c372d3dc.
Git branch: findstat/main
Using --optional=chevie,dot2tex,gap3,mpir,python2,sage,scons
Doctesting 1 file.
sage -t src/sage/interfaces/gap3.py
**********************************************************************
File "src/sage/interfaces/gap3.py", line 174, in sage.interfaces.gap3
Failed example:
    gap3.RequirePackage('"chevie"')             #optional - gap3 chevie
Expected:
    W...  to  the  CHEVIE  package, ...
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/interfaces/gap3.py", line 397, in sage.interfaces.gap3.Gap3._execute_line
Failed example:
    f([1,2,3])                                   #optional - gap3
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: Gap3 produced error output
    Error, List Element: <position> must be a positive integer at
    return L[0] ... in
    ... called from
    main loop
    brk> quit;
    ...
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.gap3.Gap3._execute_line[3]>", line 1, in <module>
        f([Integer(1),Integer(2),Integer(3)])                                   #optional - gap3
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 805, in __call__
        return getattr(P, self.name())(*args)
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 607, in __call__
        return self._parent.function_call(self._name, list(args), kwds)
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 894, in function_call
        ['%s=%s'%(key,value.name()) for key, value in kwds.items()])))
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 569, in eval
        result = Expect.eval(self, input_line, **kwds)
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1244, in eval
        for L in code.split('\n') if L != ''])
      File "/home/stumpc5/progs/sage/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 767, in _eval_line
        raise RuntimeError(message)
    RuntimeError: Gap3 produced error output
    Error, Function: <function> must be a function
    <BLANKLINE>
       executing x(sage0);
**********************************************************************
File "src/sage/interfaces/gap3.py", line 444, in sage.interfaces.gap3.Gap3.help
Failed example:
    m                                        #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/interfaces/gap3.py", line 446, in sage.interfaces.gap3.Gap3.help
Failed example:
    m.Print()                                #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    "__SAGE_LAST__"
**********************************************************************
File "src/sage/interfaces/gap3.py", line 450, in sage.interfaces.gap3.Gap3.help
Failed example:
    m                                        #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/interfaces/gap3.py", line 452, in sage.interfaces.gap3.Gap3.help
Failed example:
    m.Print()                                #optional - gap3
Expected:
    [ [ 1, 2, 3 ], [ 4, 5, 6 ] ]
Got:
    "__SAGE_LAST__"
**********************************************************************
3 items had failures:
   1 of  34 in sage.interfaces.gap3
   1 of   5 in sage.interfaces.gap3.Gap3._execute_line
   4 of  10 in sage.interfaces.gap3.Gap3.help
    [107 tests, 6 failures, 4.20 s]
----------------------------------------------------------------------
sage -t src/sage/interfaces/gap3.py  # 6 doctests failed
----------------------------------------------------------------------
Total time for all tests: 4.4 seconds
    cpu time: 0.8 seconds
    cumulative wall time: 4.2 seconds

[stumpc5]

It might or might not be related to #18971, but I currently do not know how to fix the issues (except that I could just copy the not correct output into the doctest).

[jdemeyer]

Alternatively, you could make gap3 an experimental package now and move it to optional in #18971.

One detail I just saw: spkg-src still mentions ECL :-)

And don't forget to fix the optional tags referring to chevie.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​82464a1

fixed ecl typo in spkg-src

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0cba2e8

fixed typo in package description, started work on the gap3 interface file

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cac3206

improved spkg-src

[stumpc5]

Replying to jdemeyer:

Alternatively, you could make gap3 an experimental package now and move it to optional in #18971.

I would _really_ like to get this as an optional package before we have a week meeting here in Berlin to work on (and hopefully finalize!) #11187 in two weeks from now.

[jdemeyer]

One issue, one ticket.

Let's keep the "adding a gap3 package" and "fixing the gap3 interface" separate tickets.

[jdemeyer]

That being said, if you find a reviewer who is willing to review your two-tickets-in-one branch (not me), it doesn't really matter.

[dimpase]

there are too many question marks now that need to be resolved before it can move on to become an optional package. Besides, the differences in functionality are very minor.

[dimpase]

IMvHO it is better to invest time into porting chevie to GAP4 than into this...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​181a352

undoing one thing in spkg-src

[stumpc5]

Replying to jdemeyer:

That being said, if you find a reviewer who is willing to review your two-tickets-in-one branch (not me), it doesn't really matter.

I didn't start doing anything concerning the other ticket in this branch, did I?

[stumpc5]

Replying to dimpase:

IMvHO it is better to invest time into porting chevie to GAP4 than into this...

Jean Michel clearly has a meaning about that, after not doing it in 20 years...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ec01e30

moved to experimental package

[stumpc5]

Replying to dimpase:

there are too many question marks now that need to be resolved before it can move on to become an optional package.

Just so I understand it: is it correct that for a package to be optional one needs all doctests with the optional package flag to pass? And in this specific case, the optional doctests in interfaces/gap3.py do not pass (for whatever reason) so this cannot be an optional package for now?

[dimpase]

Replying to stumpc5:

Replying to dimpase:

there are too many question marks now that need to be resolved before it can move on to become an optional package.

Just so I understand it: is it correct that for a package to be optional one needs all doctests with the optional package flag to pass? And in this specific case, the optional doctests in interfaces/gap3.py do not pass (for whatever reason) so this cannot be an optional package for now?

indeed it is a necessary condition. And of course removing tests won't fly, one really needs to dig up the real reasons for them to fail; all that "GAP3 does not do a thing on the 1st call".

[jmichel]

Replying to stumpc5:

Replying to dimpase:

IMvHO it is better to invest time into porting chevie to GAP4 than into this...

Jean Michel clearly has a meaning about that, after not doing it in 20 years...

I asked for a new password, so I can reply myself. I examined several times over the past 15 years a possible port of Chevie to GAP4. My conclusions: -it would be as much work as rewriting everything completely, thus it would take several months of my life. -I do not see any benefit for me doing that, and I do not see, if I rewrite everything, why GAP4 would be the best target. Maybe sage would be a better target. -On the other hand, I would benefit from an optional gap3 package in sage, would it be only to use the nice notebook interface to pepare my presentations about chevie. I would do it myself if I knew more about what is needed, and I thank Christian Stump for doing it. I will help him as I can.

[dimpase]

Replying to jmichel:

Replying to stumpc5:

Replying to dimpase:

IMvHO it is better to invest time into porting chevie to GAP4 than into this...

Jean Michel clearly has a meaning about that, after not doing it in 20 years...

I asked for a new password, so I can reply myself. I examined several times over the past 15 years a possible port of Chevie to GAP4. My conclusions: -it would be as much work as rewriting everything completely,

Really? What are the crucial features you miss in GAP4 that are used in Chevie?

I use GAP (and hack a bit on it, too) for some 23 years, and I saw transitions of packages like GRAPE from GAP3 to GAP4 without too much pain. You don't need to make use of new GAP4 features, good old records are still there.

I (half)ported ve and anusq to GAP4, it was certainly not very trivial, but quite doable and quick. (they are now useful in GAP4, although not as "proper" packages, but as collections of functions).

[jmichel]

Replying to dimpase:

Replying to jmichel:

Replying to stumpc5:

Replying to dimpase:

IMvHO it is better to invest time into porting chevie to GAP4 than into this...

Jean Michel clearly has a meaning about that, after not doing it in 20 years...

I asked for a new password, so I can reply myself. I examined several times over the past 15 years a possible port of Chevie to GAP4. My conclusions: -it would be as much work as rewriting everything completely,

Really? What are the crucial features you miss in GAP4 that are used in Chevie?

I use GAP (and hack a bit on it, too) for some 23 years, and I saw transitions of packages like GRAPE from GAP3 to GAP4 without too much pain. You don't need to make use of new GAP4 features, good old records are still there.

First, I should tell you that I looked at the problem of porting chevie with Frank Luebeck, and he agrees with my conclusions. This is not the place/time to give a detailed answer, but roughly: chevie contains about a hundred different types of mathematical objects, which in gap3 are just records with a certain .operations fields, and methods are just functions which dispatch based on various features, and then call the appropriate component field of the .operations record. The way a new type of objects or a method is constructed in GAP4 is completely different. The worst of it is that any effort (which has been done by various people) to port any substantial part of chevie (a few hundred lines) to gap4 almost always produce longer and uglier code.

I (half)ported ve and anusq to GAP4, it was certainly not very trivial, but quite doable and quick.

I wanted to thank you for your modifications to the makefile in ve (It seems you did not change anything else?). It also seems that you changed a few lines in one gap function in anusq to translate it to gap4. I am sorry to say that this is completely trivial compared to rewriting the several hundred thousand lines of chevie.

[dimpase]

Replying to jmichel:

Replying to dimpase:

Replying to jmichel:

Replying to stumpc5:

Replying to dimpase:

IMvHO it is better to invest time into porting chevie to GAP4 than into this...

Jean Michel clearly has a meaning about that, after not doing it in 20 years...

I asked for a new password, so I can reply myself. I examined several times over the past 15 years a possible port of Chevie to GAP4. My conclusions: -it would be as much work as rewriting everything completely,

Really? What are the crucial features you miss in GAP4 that are used in Chevie?

I use GAP (and hack a bit on it, too) for some 23 years, and I saw transitions of packages like GRAPE from GAP3 to GAP4 without too much pain. You don't need to make use of new GAP4 features, good old records are still there.

First, I should tell you that I looked at the problem of porting chevie with Frank Luebeck, and he agrees with my conclusions. This is not the place/time to give a detailed answer, but roughly: chevie contains about a hundred different types of mathematical objects, which in gap3 are just records with a certain .operations fields, and methods are just functions which dispatch based on various features, and then call the appropriate component field of the .operations record. The way a new type of objects or a method is constructed in GAP4 is completely different.

But you don't need this new type of objects!

The worst of it is that any effort (which has been done by various people) to port any substantial part of chevie (a few hundred lines) to gap4 almost always produce longer and uglier code.

I (half)ported ve and anusq to GAP4, it was certainly not very trivial, but quite doable and quick.

I wanted to thank you for your modifications to the makefile in ve (It seems you did not change anything else?). It also seems that you changed a few lines in one gap function in anusq to translate it to gap4. I am sorry to say that this is completely trivial compared to rewriting the several hundred thousand lines of chevie.

Wow, I really appreciate the way you talk about my work :-) If you cared to look at the diff of my changes to ve you could have seen the changes done in full glory, OK? ​https://github.com/gap-packages/ve/commits/master (the are mostly about switching to modern GMP, if you must know)

Whereas your GAP3 code would just work in GAP4, with very minimal and trivial changes.

[jmichel]

Wow, I really appreciate the way you talk about my work :-) If you cared to look at the diff of my changes to ve you could have seen the changes done in full glory, OK? ​https://github.com/gap-packages/ve/commits/master

Sorry. I went to ​https://github.com/gap-packages/ve and used the option "download zip" and did not see that many changes. Either you changes are quite old (several years ago) and I already got them without noticing, or I did something wrong. Anyway, please stick to technical points.

Whereas your GAP3 code would just work in GAP4, with very minimal and trivial changes.

No. But if you think otherwise, just do some non-trivial example and show me.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6e599cf

renamed optional package to experimental package in interfaces/gap3.py

[jdemeyer]

1. 92

2. In the documentation, use :trac:20107 instead of #20107`.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1583f99	replaced #20107 by :trac:20107
​38dbd60	replaced optional chevie by optional gap3 in combinat/root_system/coxeter_group.py

[stumpc5]

Replying to jdemeyer:

1. 92

I didn't realize you want me to do this also in combinat/root_system/coxeter_group.py. Done now...

2. In the documentation, use :trac:20107 instead of #20107`.

Fixed.

[stumpc5]

Just to quickly recheck: is there anything else I should prepare to get this ticket a positive review?

[dimpase]

About packages that need to be built: these ones which are pretty useless without this ought to be just removed.

[dimpase]

still does not install on OSX:

###########################################################
#  Building complete, you can run GAP3 from './../bin/gap3'
###########################################################
Installing (copying) files...
cp: gap3/pkg/ve/bin/me.exe: No such file or directory
cp: gap3/pkg/ve/bin/qme.exe: No such file or directory
Error installing (copying) gap3 files.

real	0m18.661s
user	0m26.515s
sys	0m2.467s
************************************************************************
Error installing package gap3-jm5-2015-02-01

[dimpase]

Replying to jmichel:

Wow, I really appreciate the way you talk about my work :-) If you cared to look at the diff of my changes to ve you could have seen the changes done in full glory, OK? ​https://github.com/gap-packages/ve/commits/master

Sorry. I went to ​https://github.com/gap-packages/ve and used the option "download zip" and did not see that many changes. Either you changes are quite old (several years ago) and I already got them without noticing, or I did something wrong. Anyway, please stick to technical points.

Whereas your GAP3 code would just work in GAP4, with very minimal and trivial changes.

No. But if you think otherwise, just do some non-trivial example and show me.

Earlier releases of GAP4 had GAP3 compatibility modes (a choice of various options). Did you try any of these?

[stumpc5]

Replying to dimpase:

About packages that need to be built: these ones which are pretty useless without this ought to be just removed.

Jean, it is okay to comment out the lines

#RequirePackage("anusq");
#RequirePackage("arep");
#RequirePackage("meataxe");
#RequirePackage("nq");
#RequirePackage("sisyphos");
#RequirePackage("ve");

in init.g and delete those folders in gap3/pkg, without getting into any trouble for chevie?

Otherwise, I could also try to build those packages that needs compilation, but already meataxe failed on a first try.

[jmichel]

Jean, it is okay to comment out the lines

#RequirePackage("anusq");
#RequirePackage("arep");
#RequirePackage("meataxe");
#RequirePackage("nq");
#RequirePackage("sisyphos");
#RequirePackage("ve");

in init.g and delete those folders in gap3/pkg}}, without getting into any trouble for {{{chevie?

Otherwise, I could also try to build those packages that needs compilation, but already meataxe failed on a first try.

You can comment out all of them but arep.g The function MovedPointsPerm? defined in arep/lib/tools.g is used in various places including chevie.

Generally speaking, arep has some functionality which does not need its binaries. You could delete the subdirectories bin and src of arep if you keep the rest.

In the version I am preparing (complete gap3 distribution), I will make sure all included packages can be compiled and work on x86 64bits at least.

[jmichel]

Earlier releases of GAP4 had GAP3 compatibility modes (a choice of various options). Did you try any of these?

Of course. They do not deal at all with the operations records which is the crux of the matter.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d4f3a1d	updated the spkg-src to delete gap3 packages that need compilation
​7696a2c	added patch for init.g and fixed typo in patch for gap.sh

[stumpc5]

Dima, thanks for checking!

I deleted all packages that need compilation. This solved your request on this behalf.

I would appreciate if you could retry to build it on OSX (don't forget to download the new tarball first); at least the two broken symlinks

cp: gap3/pkg/ve/bin/me.exe: No such file or directory
cp: gap3/pkg/ve/bin/qme.exe: No such file or directory

are now gone together with the complete package.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3d123df

merged 20107 to the newest develop

[dimpase]

it does not fly on OSX:

sage: gap3_console()
             ########            Lehrstuhl D fuer Mathematik, RWTH Aachen
           ###    ####             #######            #########          
          ##         ##           #      ##          ## #     ##         
         ##          #           #       ##             #      ##        
        ##                       ##       #             #      ##        
        ##           #           ##      ##             ##    ##         
        ####        ##            #########             #######          
         #####     ###                                  #                
           ######### #  Version 3 Release 4.4 18 Apr 97 #                
                     #                                  #                
                    ##  Alice Niemeyer, Werner Nickel,  Martin Schoenert 
                   ###  Johannes Meier, Alex Wegner,    Thomas Bischops  
                  ## #  Frank Celler,   Juergen Mnich,  Udo Polis        
                 ##  #  Thomas Breuer,  Goetz Pfeiffer, Hans U. Besche   
                ##   #  Volkmar Felsch, Heiko Theissen, Alexander Hulpke 
               ##    #  Ansgar Kaup,    Akos Seress,    Erzsebet Horvath 
              ##     #  Bettina Eick                                     
              ###   ##  
               ######   For help enter: ?<return>                        
 lib: jm 12Mar2016, src: jm 28Feb2015, sys: macosx gcc
 gap3-jm5 final 21Mar2016 -- see webusers.imj-prg.fr/~jmichel/gap3
 Minimal distribution -- loading the following packages:
--- Loading package arep --------- version 1.0, 16Mar1998--------------------
(C) Sebastian Egner, Markus Pueschel -- Abstract REPresentations
--- Loading package autag -------- version Jan1997---------------------------
Gasman: last bag of type integer (> 2^28) and size       16 has overwritten the free bag.
sage: 

[dimpase]

The reason for this must be that the flag -DSYS_IS_64_BIT is not passed during compilation on OSX. Indeed, gap3_makefile.patch has no ifeq ($(ARCH),64/32) block for Darwin. Fixed in this branch.

---

New commits:

​f5cdde2

correct options for OSX make targets

[dimpase]

also, is this meant to fail:

$ sage -tp --optional=ccache,gap_packages,mpir,python2,gap3,sage src/sage/combinat/root_system/coxeter_group.py 
too few successful tests, not using stored timings
Running doctests with ID 2016-04-03-00-22-03-2f808575.
Git branch: g3
Using --optional=ccache,gap3,gap_packages,mpir,python2,sage
Doctesting 1 file using 4 threads.
sage -t src/sage/combinat/root_system/coxeter_group.py
**********************************************************************
File "src/sage/combinat/root_system/coxeter_group.py", line 226, in sage.combinat.root_system.coxeter_group.CoxeterGroupAsPermutationGroup._element_class
Failed example:
    W._element_class() is W.element_class                      # optional - gap3
Exception raised:
    Traceback (most recent call last):
      File "/home/dima/software/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/dima/software/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.root_system.coxeter_group.CoxeterGroupAsPermutationGroup._element_class[1]>", line 1, in <module>
        W._element_class() is W.element_class                      # optional - gap3
      File "sage/structure/parent.pyx", line 854, in sage.structure.parent.Parent.__getattr__ (build/cythonized/sage/structure/parent.c:7994)
        attr = getattr_from_other_class(self, self._category.parent_class, name)
      File "sage/structure/misc.pyx", line 253, in sage.structure.misc.getattr_from_other_class (build/cythonized/sage/structure/misc.c:1666)
        raise dummy_attribute_error
    AttributeError: 'CoxeterMatrixGroup_with_category' object has no attribute '_element_class'
**********************************************************************
1 item had failures:
   1 of   3 in sage.combinat.root_system.coxeter_group.CoxeterGroupAsPermutationGroup._element_class
    [53 tests, 1 failure, 3.95 s]
----------------------------------------------------------------------
sage -t src/sage/combinat/root_system/coxeter_group.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 13.2 seconds
    cpu time: 0.8 seconds
    cumulative wall time: 3.9 seconds

[stumpc5]

Replying to dimpase:

Thanks for rechecking and for fixing the make target for osx!

----------------------------------------------------------------------
sage -t src/sage/combinat/root_system/coxeter_group.py  # 1 doctest failed
----------------------------------------------------------------------

I doubt this worked before, I only replaced # optional - chevie by # optional - gap3. It has been there since 2013:

$ git blame -L220,+10 src/sage/combinat/root_system/coxeter_group.py
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 220)         A temporary workaround for compatibility with Sage's
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 221)         permutation groups
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 222) 
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 223)         TESTS::
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 224) 
1452e665 (Jeroen Demeyer 2013-03-02 13:53:59 +0100 225)             sage: W = CoxeterGroup(["H",3])                                  # optional - chevi
1452e665 (Jeroen Demeyer 2013-03-02 13:53:59 +0100 226)             sage: W._element_class() is W.element_class                      # optional - chevi
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 227)             True
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 228)         """
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 229)         return self.element_class

Getting this properly working will be part of our *finally-get-reflection-groups-smoothly-in-sage-days next week*.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a0d0e18

3 trivial doctest fixes

[stumpc5]

Many thanks to both of you for your reviewing work on this ticket!

[jdemeyer]

Replying to stumpc5:

I doubt this worked before, I only replaced # optional - chevie by # optional - gap3.

Exactly. This ticket is not about fixing all doctests, just about adding the package.

[jdemeyer]

gap3 is currently broken: #21722