Opened 5 years ago

Closed 18 months ago

#18514 closed defect (fixed)

Upgrade of p_group_cohomology spkg

Reported by: SimonKing Owned by:
Priority: major Milestone: sage-8.3
Component: packages: optional Keywords: group cohomology
Cc: jhpalmiery, vbraun, jdemeyer, schilly, frederichan, david.green@… Merged in:
Authors: Simon King Reviewers: Travis Scrimshaw, Jeroen Demeyer
Report Upstream: None of the above - read trac for reasoning. Work issues:
Branch: 6576915 (Commits) Commit: 65769150a96fcd8a5fcfad88a4c2173dedb11b42
Dependencies: Stopgaps:

Description (last modified by SimonKing)

In the past, p_group_cohomology-2.1.4 was an optional spkg. Sage dropped support for old style spkgs, and thus only an experimental spkg remained.

The purpose of this ticket is to provide a new group cohomology package for Sage. I suggest to make it optional (not experimental).

In the experimental version 2.1.5, I have improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbert-driven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.

Package structure

Version 3.0 is refactored as follows:

  • It depends on the optional meataxe package (see #24359), which is based on a fork of MeatAxe.
  • The spkg-install script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.
  • The C code written by David Green has become an autotoolized shared library.
  • Gap and Singular code is put into SAGE_SHARE/, where GAP and Singular libraries belong.
  • The rest of the package is formed by python and cython code, and uses pip for installation.
  • The documentation is built in an improved way.

Further changes

  • The custom logging in the old spkg is replaced by Python's logging module.
  • The tests avoid side-effects by using a reset function. Hence, no custom doc tester is needed: sage -t is enough.
  • In the previous spkg, I tried to be as backward compatible as possible. But now I'm dropping support for very old Singular versions.
  • Some experimental options for the cohomology computation are removed.

Many tests have changed, since the ring presentations of the computed cohomology rings, specifically for non-primepower groups, have changed. I do not fully understand why they changed. but it seems that it is a consequence of changes in Singular. The changes did (of course...) not affect the isomorphism types of the rings.

Upstream tar ball

https://users.fmi.uni-jena.de/cohomology/p_group_cohomology-3.0.tar.xz

Change History (568)

comment:1 Changed 5 years ago by SimonKing

  • Status changed from new to needs_review

For me, all but two tests fail. Both tests fail with "connection timed out" during access to a web data base, and both work fine when done in an interactive session.

Has something in the test framework changed, so that web access via urllib2 fails during tests?

comment:2 follow-up: Changed 5 years ago by jdemeyer

I see you are compiling some extensions with the csage library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile against csage only if it actually exists.

comment:3 in reply to: ↑ 2 Changed 5 years ago by SimonKing

Replying to jdemeyer:

I see you are compiling some extensions with the csage library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile against csage only if it actually exists.

OK. So I should see what I actually use of csage.

comment:4 follow-up: Changed 5 years ago by jdemeyer

The sig_on() framework was in csage until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link against csage if the Sage version is <= 6.7

comment:5 in reply to: ↑ 4 ; follow-up: Changed 5 years ago by SimonKing

Replying to jdemeyer:

The sig_on() framework was in csage until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link against csage if the Sage version is <= 6.7

OK. What does that mean for the code? Do I need to import anything if I am using sig_on/off, or would this just work automatically with the new Cython version?

comment:6 in reply to: ↑ 5 Changed 5 years ago by jdemeyer

Replying to SimonKing:

OK. What does that mean for the code? Do I need to import anything if I am using sig_on/off, or would this just work automatically with the new Cython version?

You just include sage/ext/interrupt.pxi, that doesn't change.

The main difference is that you no longer need to add csage as library. Currently, it doesn't hurt if you add the library anyway, but in the future, csage might not exist anymore.

And there is one additional caveat which is relevant for your package: since #18027, you must include interrupt.pxi only in .pyx files and not .pxd files (unless you really know what you're doing).

Last edited 5 years ago by jdemeyer (previous) (diff)

comment:7 follow-up: Changed 5 years ago by jdemeyer

On the topic of .pxd files, this is what they should contain:

  1. The API of your module, that is all cdef classes and c(p)def functions that you want to export for use in other modules (in the same package or by other packages).
  1. If your module provides bindings to some library (the mtx module in your package for example), then add the declarations of the library functions in the .pxd file.
  1. Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from RingElement. In your .pxd file, you want to write cdef class MySpecialElement(RingElement). This makes RingElement a dependency, so you need to write from ... cimport RingElement in the .pxd file.

And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the .pyx file. This is important in order to minimize dependencies.

NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different .pxd file. In Sage, we usually put the library bindings in sage/libs, look at sage/libs/gmp for a good example.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 5 years ago by SimonKing

Replying to jdemeyer:

On the topic of .pxd files, this is what they should contain:

  1. The API of your module, that is all cdef classes and c(p)def functions that you want to export for use in other modules (in the same package or by other packages).
  1. If your module provides bindings to some library (the mtx module in your package for example), then add the declarations of the library functions in the .pxd file.
  1. Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from RingElement. In your .pxd file, you want to write cdef class MySpecialElement(RingElement). This makes RingElement a dependency, so you need to write from ... cimport RingElement in the .pxd file.

And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the .pyx file. This is important in order to minimize dependencies.

OK. That's something that I should change. Actually, until not so long ago, I thought that cimports were only possible in pxd files.

NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different .pxd file. In Sage, we usually put the library bindings in sage/libs, look at sage/libs/gmp for a good example.

Do I understand correctly: You suggest that I should create a new pxd file for the MeatAxe bindings, say, mtx_lib.pxd, which is then cimported in mtx.pxd?

comment:9 in reply to: ↑ 8 Changed 5 years ago by jdemeyer

Replying to SimonKing:

Do I understand correctly: You suggest that I should create a new pxd file for the MeatAxe bindings, say, mtx_lib.pxd, which is then cimported in mtx.pxd?

I cannot really speak about this specific case, but it certainly would make sense to split it up.

comment:10 Changed 5 years ago by kcrisman

  • Component changed from PLEASE CHANGE to packages: optional

comment:11 Changed 5 years ago by SimonKing

  • Status changed from needs_review to needs_work
  • Work issues set to Cope with yet another backward incompatible change of Sage

I just found that the package won't build with the latest beta of Sage. Reason:

pGroupCohomology/cohomology.c:248:36: fatal error: sage/ext/interrupt/pxi.h: No such file or directory
 #include "sage/ext/interrupt/pxi.h"
                                    ^
compilation terminated.
error: command 'gcc' failed with exit status 1

So, can you please tell me how to use interrupts, after all these backward incompatible changes? It really sucks.

comment:12 Changed 5 years ago by SimonKing

Good news! On my machine, I did some refactoring of code according to the hints you gave me---and that did the trick.

My plan is to continue refactoring, taking into account what you said about where to import stuff, and then push a new package version to the official page.

comment:13 follow-up: Changed 5 years ago by SimonKing

Bad news. It builds, but I can't import it. Which brings up the question again why there are all these incompatible changes in Sage. They didn't use to occur so frequently in earlier versions of Sage, IIRC.

comment:14 Changed 5 years ago by SimonKing

  • Status changed from needs_work to needs_review
  • Work issues Cope with yet another backward incompatible change of Sage deleted

I got it fixed. So, you'll find the updated version of p_group_cohomology-2.1.5 on my web page.

Some of your concerns about importing/including stuff are addressed in the new version: I moved some of it from .pxd files into .pyx files. What I did not do is to split off library bindings and put them into separate .pxd files. That would still make sense, but for now I put the ticket back to "needs review".

comment:15 Changed 5 years ago by SimonKing

Oh, and I also created a new module pGroupCohomology.auxiliaries, in which I've put some auxiliary functions that have previously been in pGroupCohomology.resolution.

comment:16 Changed 5 years ago by frederichan

  • Cc frederichan added

I have a built failure for sage -i upstream/p_group_cohomology-2.1.5.spkg with sage 6.8beta3+trac18666 (but success with master)

gcc -pthread -shared -L/home/fred/dev/sage/local/lib build/temp.linux-x86_64-2.7/pGroupCohomology/resolution.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/aufloesung.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/aufnahme.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/djgerr.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/fileplus.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/nBuchberger.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/pgroup.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/pincl.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/slice.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/urbild.o build/temp.linux-x86_64-2.7/pGroupCohomology/c_sources/uvr.o -L/home/fred/dev/sage/local/lib -lmtx -lpython2.7 -o build/lib.linux-x86_64-2.7/pGroupCohomology/resolution.so
cythoning pGroupCohomology/cohomology.pyx to pGroupCohomology/cohomology.c
building 'pGroupCohomology.cohomology' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/fred/dev/sage/src/c_lib/include -I/home/fred/dev/sage/local/share/sage/ext -I/home/fred/dev/sage/src/sage/ext -Imtx2.2.4/src -IpGroupCohomology/c_sources -IpGroupCohomology -I/home/fred/dev/sage/local/include/python2.7 -c pGroupCohomology/cohomology.c -o build/temp.linux-x86_64-2.7/pGroupCohomology/cohomology.o -O3
In file included from pGroupCohomology/cohomology.c:247:0:
mtx2.2.4/src/meataxe.h:15:0: warning: "_GNU_SOURCE" redefined
 #define _GNU_SOURCE
 ^
In file included from pGroupCohomology/cohomology.c:8:0:
/home/fred/dev/sage/local/include/python2.7/pyconfig.h:1160:0: note: this is the location of the previous definition
 #define _GNU_SOURCE 1
 ^
pGroupCohomology/cohomology.c:262:37: fatal error: sage/libs/ntl/ntlwrap.cpp: No such file or directory
 #include "sage/libs/ntl/ntlwrap.cpp"
                                     ^
compilation terminated.
error: command 'gcc' failed with exit status 1
Error installing Cython code.

real	0m43.412s
user	0m40.636s
sys	0m2.736s
************************************************************************
Error installing package p_group_cohomology-2.1.5
************************************************************************

comment:17 Changed 5 years ago by jdemeyer

You might want to wait for #18494, which should make it easier to install external packages using the Sage library.

comment:18 follow-up: Changed 5 years ago by SimonKing

The real questions are: Why is it looking for ntl? I don't think it is explicitly requested by my package, thus, it probably is an indirect dependency, i.e., one that is in fact a dependency of something in Sage that I cimport into my package. But if it is a dependency of something in Sage, then why does it not work?

I suppose that I have to provide an include path to fix it. If the dependency really is via something in the Sage library, then this "have to" is not very nice---Sage could figure it out without human help.

comment:19 in reply to: ↑ 18 Changed 5 years ago by jdemeyer

Replying to SimonKing:

The real questions are: Why is it looking for ntl?

Because the Sage <-> NTL interface is a real mess. In particular, any module which needs just a fragment of NTL automatically cimports essentially all of the NTL interface. In particular, sage.rings.integer does that.

comment:20 Changed 5 years ago by jdemeyer

In setup.py, replace

              include_dirs = CSAGE_PATH + [
                              os.path.join(SAGE_SRC,"c_lib","include"),
                              SAGE_EXTCODE, SRC_EXT,
                              os.path.join("mtx2.2.4","src"),
                              os.path.join("pGroupCohomology","c_sources"),
                              "pGroupCohomology"]

by

try:
    # Sage >= 6.8
    from sage.env import sage_include_directories
except ImportError:
    # Sage < 6.8
    def sage_include_directories():
        return [
            os.path.join(SAGE_LOCAL, "include", "csage"),
            os.path.join(SAGE_SRC, "c_lib", "include"),
            SAGE_EXTCODE, SRC_EXT]  # Not sure that you really need SAGE_EXTCODE

...

              include_dirs = sage_include_directories() + [
                             os.path.join("mtx2.2.4","src"),
                             os.path.join("pGroupCohomology","c_sources"),
                             "pGroupCohomology"]

and use #18494.

comment:21 Changed 5 years ago by SimonKing

  • Dependencies set to #18494

I hope it is safe to make #18494 a dependency. If I understand correctly, there is only a very trivial doctest failure, and so hopefully it will be in Sage soonish.

Meanwhile I almost gave up to make my package backwards compatible with sage-3.x...

comment:22 Changed 5 years ago by SimonKing

With sage_include_directories() I will not need to construct CSAGE_PATH, right?

comment:23 Changed 5 years ago by SimonKing

I have updated the spkg, and successfully installed it on the Sage version that is provided by #18494.

comment:24 in reply to: ↑ 13 Changed 5 years ago by jdemeyer

Replying to SimonKing:

Which brings up the question again why there are all these incompatible changes in Sage.

Essentially, because of #17854 and its sub-tickets. Removing the low-level c_lib (a.k.a. libcsage) did indeed have consequences for all packages compiling against Sage.

They didn't use to occur so frequently in earlier versions of Sage, IIRC.

And they probably will not occur anymore in the future, now that #17854 is dealt with. And if there are changes, now we can just change the function sage_include_directories() from #18494.

comment:25 follow-up: Changed 4 years ago by tmonteil

Is there a reason not to make a new-style packages for p_group_cohomology ?

comment:26 in reply to: ↑ 25 Changed 4 years ago by SimonKing

Replying to tmonteil:

Is there a reason not to make a new-style packages for p_group_cohomology ?

Yes. I need someone who explains to me in ALL detail how to create a new-style package.

It begins with: How do I create an "upstream" repository for my sources? Note that I will not change the meataxe matrix wrapper that is included in the package into a wrapper for the latest meataxe upstream, as this would require to rewrite most of the c- and much of the cython-code. So, the files in the current package are an old version of meataxe with massive patches (i.e., basically a fork), plus c- and cython- and singular- and gap-code, for which this package currently is the only source.

comment:27 Changed 4 years ago by tscrim

You create a folder in the $SAGE_ROOT/build/pkgs with files similar to the others. You will also need to put the tarball with the source code in $SAGE_ROOT/upstream (and post it here or online so that it can be put on the online repo once merged) and then run sage -sh sage-fix-pkg-checksums. All of this folder should be under version control.

See also: http://doc.sagemath.org/html/en/developer/packaging.html

comment:28 Changed 4 years ago by SimonKing

It seems that there is no way around creating a new-style spkg, although I still find it odd to artificially create an upstream source in a situation where the spkg IS upstream. Sigh.

I have to

  • get a github account (or reactivate an existing account?),
  • learn how to convert the existing hg repository in the spkg to git,
  • look up how to push my private repository to github,
  • do what is said in comment:27.

Did I forget something?

Now comes the complicated part.

The spkg has three sources, namely

  1. C-MeatAxe
  2. C programs written by David Green
  3. Cython, Python, Gap and Singular code written by me.

Ad 1.

Actually the starting point is a very old version of C-MeatAxe. If I see that correctly, upstream currently isn't very active. However, compared with the version used in the spkg, upstream has changed most function names.

I apply big patches to C-MeatAxe, as I implemented Strassen multiplication. It improves performance a lot (in the old sources). However, Strassen still is not implemented in the current upstream sources. Two years ago I attended a presentation of one C-MeatAxe developer, and he said that they prefer to optimise the existing implementation rather than adding asymptotically fast implementations. It didn't convince him when I told him that in the old MeatAxe, Strassen beats school book multiplication for matrices as small as 200x200.

Ad 2.

David Green's programs have never been published independently. Hence, the spkg *is* upstream. The code uses the old C-MeatAxe extensively.

Ad 3.

My code has never been published independently. Hence, the spkg *is* upstream. It contains a Cython wrapper for the old C-MeatAxe.

Conclusion

I guess in an ideal world, I would create a new spkg for C-MeatAxe, containing my patches and my Cython wrapper and the MeatAxe executable, but starting with current upstream. I would then create a repository for Green's and my code. p_group_cohomology would then depend on the new MeatAxe spkg.

If I recall correctly, I actually have a wrapper for a recent version of MeatAxe, perhaps even with my Strassen implementation. Hence, creating the new MeatAxe spkg might be a feasible first step.

However, rewriting Green's programs in terms of the new MeatAxe could be a pain in the neck.

So, as a short term solution, I'd like to keep the proposed version 2.1.5 of my spkg, that still uses the old C-MeatAxe, with the prospect that version 3.0 should be based on the current MeatAxe upstream.

comment:29 follow-up: Changed 4 years ago by SimonKing

Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.

comment:30 follow-up: Changed 4 years ago by tscrim

I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).

comment:31 Changed 4 years ago by SimonKing

  • Description modified (diff)

comment:32 in reply to: ↑ 29 Changed 4 years ago by dimpase

Replying to SimonKing:

Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.

how about you make a copy of your spkg on geom.math.washington.edu or combinat.math.washington.edu somewhere readable, e.g. to /tmp ?

At least then we can pick it up and put up somewhere...

comment:33 in reply to: ↑ 30 Changed 4 years ago by SimonKing

Replying to tscrim:

I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).

Thank you! For now, I have put the old-style spkg at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg

comment:34 Changed 4 years ago by SimonKing

PS: The spkg uses a data base of pre-computed cohomology rings that used to be at sage.math.washington. Hence, now it is not accessible.

If the spkg can't find a cohomology ring in the data base, then it will compute it from scratch. So, most self-tests should work. However, some tests explicitly try to find data in the sage.math.washington data base. These will fail.

It seems that the data base (20-30GB) can be relocated to Jena as well. But it will take at least till next week, and when it's done I need to change the spkg to using a new default url of the data base.

comment:35 Changed 4 years ago by SimonKing

From the README file of the latest upstream sources (which is version 2.4.24 from October 2011(!)).

================================================================================
INSTALLATION (for Unix systems)
================================================================================

1) Copy Makefile.conf.dist to Makefile.conf. Edit Makefile.conf
   and fill in the necessary parameters.

Is that how it is supposed to work with a modern build system?

comment:36 Changed 4 years ago by SimonKing

There is #12103 for using MeatAxe implementation of matrix multiplication over small finite non-prime fields. But that was based on version 2.2.4, not 2.4.xx.

comment:37 follow-up: Changed 4 years ago by jhpalmieri

Do you have to convert the hg repository to a git repository? What we really need is (1) the directory build/pkgs/p_group_cohomology with spkg-install etc., and (2) a tarball containing the upstream source. Since you are the maintainer, it is up to you how to distribute that tarball, so I think if you want to distribute it with an hg repository, that's okay. If you just want to release a particular version with no repo at all included, that should be fine, too. (We don't always distribute repository information in the tarballs for other packages, for what that's worth.)

comment:38 Changed 4 years ago by jhpalmieri

For now, by the way, I think the easiest path is the right one: use your current versions of everything, don't bother trying to use newer versions of MeatAxe, etc.

comment:39 in reply to: ↑ 37 Changed 4 years ago by SimonKing

Replying to jhpalmieri:

Do you have to convert the hg repository to a git repository? What we really need is (1) the directory build/pkgs/p_group_cohomology with spkg-install etc., and (2) a tarball containing the upstream source.

Is it that easy? That would be nice.

I was told somewhere that moving from hg (even with queues?) to git is not particularly difficult. I was reluctant to use git for Sage development and I still miss hg sometimes. But using git for Sage and at the same time hg for a Sage package seems awkward to me

comment:40 Changed 4 years ago by tscrim

I agree with John: try to keep things as intact as possible. It should be possible to simply detach the SPKG.txt and sage-install scripts from spkg and put those (and sage-check and any patches) in build/pkgs/p_group_cohomology (with a package-version.txt, checksums.ini, and type files; see any other spkg). Then run sage -sh sage-fix-pkg-checksums. You should put a clean tarball of the necessary source in the upstream folder (this will be the upstream tarball that will go on the Sage mirrors), which should be everything else. Then, and only then, check that it installs (otherwise I found it deletes the tarball).

Just to be clear, I think it would be better to simply transition the spkg style here and now and consider doing upgrades later (with a version bump of your spkg).

Also have you considered bitbucket? They do hg repositories and you can keep your upstream there (and under hg control instead of git). Also they seem to be more generous than github for free (private) academic repositories.

comment:41 follow-up: Changed 4 years ago by jdemeyer

So, what is the plan here? Keep the old-style package on this ticket or change to a new-style package? In the latter case, the ticket should be set back to needs_work.

comment:42 in reply to: ↑ 41 Changed 4 years ago by SimonKing

  • Status changed from needs_review to needs_work
  • Work issues set to Create a new-style package with least effort

Replying to jdemeyer:

So, what is the plan here? Keep the old-style package on this ticket or change to a new-style package?

If I understood correctly, the suggestion of John and Travis was to create a new-style package with the least effort. I.e., take the current sources contained in the spkg and post them somewhere (aka "upstream"), and install new-stylish from there.

Or perhaps a little better and still feasible: Use the *proper* existing upstream for MeatAxe (even though the package uses a very old version), and only put the rest of the sources to a new upstream.

In the latter case, the ticket should be set back to needs_work.

OK.

comment:43 Changed 4 years ago by SimonKing

I have moved the documentation of the package. In case you are interested in testing an old-style version 2.1.5, see http://users.minet.uni-jena.de/cohomology/documentation/#installation

Next, I plan to create a least-effort new-style version 2.1.6. It will be basically identical with 2.1.5, except for the package style. And changing everything to the latest MeatAxe upstream will be considerable work taking some time, justifying a new ticket and a major new version 3.0.

comment:44 follow-up: Changed 4 years ago by SimonKing

Argh. There is more work to do. 2.1.5 installs, but doesn't work for non-primepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.

comment:45 Changed 4 years ago by SimonKing

Sigh. Apparently there is also a change in the pickling of Singular objects that I am responsible for and that I have now to cope with...

comment:46 in reply to: ↑ 44 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

Argh. There is more work to do. 2.1.5 installs, but doesn't work for non-primepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.

I guess it's because of the Cython upgrade. Earlier versions of Cython did not give an error for code like

f(foo="hello", foo="world")

but now that's an error just like in Python.

comment:47 in reply to: ↑ 46 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

I guess it's because of the Cython upgrade.

Probably. That error (somewhere doing mutable = True, mutable = True) is fixed. But the Singular pickling error is likely to be tricky.

Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.

comment:48 in reply to: ↑ 47 Changed 4 years ago by jdemeyer

Replying to SimonKing:

Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.

I'm afraid I can't help you there...

comment:49 Changed 4 years ago by SimonKing

In the previous versions, I already had a class GapPickler to help with pickles of Gap interface objects (by default, their content is lost when pickling, although it can be reconstructed from the string representation in most cases). I guess I could do basically the same for other interfaces.

comment:50 Changed 4 years ago by SimonKing

I replaced the spkg by a version that *almost* passes the test suite. The only remaining problem is that in the stored data the old pickling of Singular elements is used, which results in a deprecation warning.

So, my plan is: Re-compute the cohomology for all groups of order 64, store them in a database, put the database into the package, and put it online.

Then, finally create a basic version of a new-style package.

comment:51 follow-up: Changed 4 years ago by SimonKing

I created the current version 2.1.5 on my laptop and tested on the computer in my office. Most tests passed. Problems:

  • It seems that since a while, web access during doctests is not possible. It used to work, and the failing tests do work in interactive sessions. No idea how that could be fixed; I guess it was caused by a change in the doctest framework.
  • There have still been deprecation warnings when unpickling old pickled cohomology rings. Whether or not we will see the deprecation warning depends on the user: If (s)he has old pickles, then the tests will fail. If (s)he removes the old pickles before re-installing the package, then the tests will work.
  • There has been one error where a different homogeneous system of parameters was found. Mathematically there is no substantial difference, but it may be needed to change the expected output.

comment:52 in reply to: ↑ 51 Changed 4 years ago by SimonKing

Replying to SimonKing:

  • It seems that since a while, web access during doctests is not possible. It used to work, and the failing tests do work in interactive sessions. No idea how that could be fixed; I guess it was caused by a change in the doctest framework.

Strangely, these tests worked on my laptop. Is it possible that "screen" (which I use on the office computer) prevents internet access during doctests?

  • There have still been deprecation warnings when unpickling old pickled cohomology rings. Whether or not we will see the deprecation warning depends on the user: If (s)he has old pickles, then the tests will fail. If (s)he removes the old pickles before re-installing the package, then the tests will work.

Question to the reviewer: Should one add a remark in the documentation, telling that a certain deprecation warning is expected if old data is present?

  • There has been one error where a different homogeneous system of parameters was found. Mathematically there is no substantial difference, but it may be needed to change the expected output.

I get the same on my laptop. Thus I changed it, and updated the package. Now I redo the tests on my office computer.

comment:53 Changed 4 years ago by tscrim

Is there anything I can do to help right now?

comment:54 Changed 4 years ago by SimonKing

I guess I have to do some more fixes. It turns out that one can not install the spkg if before one has installed the meataxe package that I propose at #12103. Probable reason: Conflicting types in the meataxe version used here and used there.

Actually the cleanest solution would be to remove the old meataxe sources from the spkg, and rewrite the code of David Green by using the new meataxe sources. Hence, the meataxe package would become a dependency of the cohomology spkg.

But my plan was to do this (a MAJOR rewrite!) on the long run, not now. I will do some tests, to see how much work will be involved. If it merely is a cut-and-paste modification (changing ALL function and type names used in meataxe) then it might be doable.

comment:55 follow-up: Changed 4 years ago by tscrim

A more hack solution would be to make the meataxe that you include in this spkg have a different name, and so that it could be used along #12103. However, as you said, the cleanest solution would be to base this off of the meataxe spkg on #12103 as a dependency.

I would imagine that the majority of the work would just be some name/type replacements as long as you weren't doing anything too exotic...at least I doubt they made major changes to their API...

comment:56 in reply to: ↑ 55 Changed 4 years ago by SimonKing

Replying to tscrim:

I would imagine that the majority of the work would just be some name/type replacements as long as you weren't doing anything too exotic...at least I doubt they made major changes to their API...

When translating my implementation of Winograd-Strassen multiplication from meataxe-2.2.4 to meataxe-2.4.24, most of the work has indeed been cut-and-paste.

However, we talk here about changing third party code (namely David Green's), and since I didn't write it I don't know if the transition will be smooth.

comment:57 Changed 4 years ago by SimonKing

At least the data structure has remained basically the same (representing matrices as a contiguous block of memory organised in rows of the matrix, each byte contains up to 8 matrix coefficients, depending on the field size, and all arithmetic operations are done by table lookup).

comment:58 follow-ups: Changed 4 years ago by SimonKing

  • Cc david.green@… added

I am now confident that it is feasible to make the group cohomology software work with the new optional meataxe package from #12103. I managed to rewrite the stand-alone programs of David Green needed in the package, and the rest "should work" (TM) by search-and-replace operations (really, ALL names in meataxe have changed).

But I have questions concerning the new code structure.

The old spkg (p_group_cohomology-2.1.5) consists of

  • Original sources of meataxe-2.2.4, together with my "fork" of meataxe,
  • C-programs written by David Green, in src/present/,
  • GAP-functions written by David Green, in src/pGroupCohomology,
  • Singular-functions written by me, in src/pGroupCohomology,
  • Cython modules written by me, in src/pGroupCohomology,
  • documentation, including a doc builder taken (and modified) from old Sage versions,
  • a data base of cohomology rings for the groups of order 64.

For the new package, I suggest:

  • meataxe-2.4.24 is used, as provided by the package from #12103.
  • The C-programs of David Green, converted to work with the new MeatAxe, will be a new optional package called modular_resolutions-1.0 (or do you prefer modres-1.0?) depending on meataxe, adding me as a second author, as the conversion to the new MeatAxe was very nontrivial, and I also added error handling.
  • The database will also be part of modular_resolutions.
  • The Cython modules will become optional extensions in the SageMath library, say, sage.groups.modular_cohomology, depending on modular_resolutions and on database_gap. In that way, the documentation would be taken care of.
  • Where shall the Singular and GAP functions go? Is there a natural place? There is SAGE_LOCAL/share/singular/ for the Singular functions. But for GAP?

Remark: database_gap is needed because of the SmallGroups library. Technically, the upcoming modular_resolutions (or modres) package would not depend on it; but it is needed in the cohomology computations, thus, perhaps it makes still sense to add database_gap as a dependency of modular_resolutions?

comment:59 in reply to: ↑ 58 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

or do you prefer modres-1.0?

Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions.

comment:60 in reply to: ↑ 58 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

  • Where shall the Singular and GAP functions go? Is there a natural place? There is SAGE_LOCAL/share/singular/ for the Singular functions. But for GAP?

About the GAP code: is it C code for GAP or GAP code? In the latter case, can you use src/ext/gap?

comment:61 in reply to: ↑ 59 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

or do you prefer modres-1.0?

Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions.

Concerning "upstream": David Green is the original upstream. But as much as I know his code has never been published outside of the old spkg. And since the rebase on top of meataxe-2.4.24 was major, I believe that I am to some extent upstream, too.

The original sources that David Green gave me are in a folder called "present". But I doubt that a user having a look at the list of optional packages would guess that "present" is about modular resolutions of p-groups. From my perspective, "modular_resolutions" is a bit lengthy. I will ask David, after all he is native speaker.

comment:62 in reply to: ↑ 60 Changed 4 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

  • Where shall the Singular and GAP functions go? Is there a natural place? There is SAGE_LOCAL/share/singular/ for the Singular functions. But for GAP?

About the GAP code: is it C code for GAP or GAP code? In the latter case, can you use src/ext/gap?

Pure GAP code, in the files src/pGroupCohomology/GapMB, .../GapMaxels and .../GapSgs.

Is src/ext/gap somehow known to Gap? Or does one need to give the full path when loading code from there into Gap?

comment:63 follow-up: Changed 4 years ago by dimpase

you might consider packaging your GAP code as a GAP package...

comment:64 in reply to: ↑ 63 ; follow-up: Changed 4 years ago by SimonKing

Replying to dimpase:

you might consider packaging your GAP code as a GAP package...

Hopefully that's a joke. I don't want to learn for yet another CAS how to contribute a package, in particular for code that I didn't author.

comment:65 in reply to: ↑ 61 Changed 4 years ago by SimonKing

Replying to SimonKing:

Replying to jdemeyer:

Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions.

I got answer of David Green.

  • He agrees that, by porting the code on top of the new meataxe, I became a co-author.
  • He coined his package "present", because the original purpose was to deal with presentations of modular group algebras of prime power groups.
  • If I understand correctly, he wouldn't mind renaming it.

I believe that the main purpose of the code is not to compute a presentation of a modular group algebra, but to compute a minimal projective resolution of the modular group algebra. At least that's what I am using it for.

Since the old package is called p_group_cohomology and the cohomology part will be moved to the Sage library, the package's responsibility will be to compute resolutions. Perhaps (to remind the old name) p_resolution? I find modular_resolution too long, Jeroen finds modres too abbreviated (I guess).

comment:66 in reply to: ↑ 64 Changed 4 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

you might consider packaging your GAP code as a GAP package...

Hopefully that's a joke. I don't want to learn for yet another CAS how to contribute a package, in particular for code that I didn't author.

well, by package I just meant a way to package GAP files in a way understood by GAP.

Of course, you can also use GAP's Read() command to load any GAP code from anywhere in the filesystem.

sage: gap.eval('Read("'+SAGE_LOCAL+'/foo/blah.g")') # GAP
sage: libgap.Read('"'+SAGE_LOCAL+'/foo/blah.g"') # libGAP

HTH, Dima

comment:67 follow-up: Changed 4 years ago by dimpase

by the way, would you mind renaming GAP source code files, by giving them suffix, '.g' or '.gap' ?

comment:68 in reply to: ↑ 67 Changed 4 years ago by SimonKing

Replying to dimpase:

by the way, would you mind renaming GAP source code files, by giving them suffix, '.g' or '.gap' ?

No problem. And then put it into src/ext/gap, as I was told.

comment:69 follow-up: Changed 4 years ago by SimonKing

I think I need help, as I am virtually a beginner with Makefiles, compiling c-sources, and/or creating a library.

For creating .o-files, I have the following rule in my Makefile, where

  • VPATH is the path to the sources
  • CFLAGS=-I$(MTXINC) -I$(VPATH) -Wall $(OO) -g -fPIC
  • MTXINC is the SAGE_LOCAL/include
    %.o: $(VPATH)/pgroup.h $(VPATH)/pincl.h $(VPATH)/pincl_decls.h $(VPATH)/$*.c
    	@echo "Compiling $*.c"
    	@$(CC) $(CFLAGS) -c $(VPATH)/$*.c -o $@
    

The above is a modification of MeatAxe's Makefile, which says

tmp/%.o: tmp/mk.dir src/%.c src/meataxe.h tmp/config.h
	@echo "Compiling $*.c"
	@$(CC) $(CFLAGS) -c src/$*.c -o $@

While the rule in MeatAxe's Makefile works, in my Makefile it is not executed. For example, when making pinc.o, the following is printed to stdout:

gcc -I/home/king/Sage/git/sage/local/include -I/home/king/Projekte/coho/p_group_cohomology-3.0/src/present/src -Wall  -g -fPIC    -c -o pincl.o /home/king/Projekte/coho/p_group_cohomology-3.0/src/present/src/pincl.c

In particular, the @echo "compiling ..." statement is not executed.

So, what is happening here? Is there an implicit make rule that takes precedence over the explicit rule??

And further, when I try to build an executable (say, makeInclusionMatrix), the rule says

makeInclusionMatrix: libmodres.a mim.o
	$(CC) $(LFLAGS) -o makeInclusionMatrix mim.o libmodres.a

with

  • LFLAGS=-l$(MTXLIBNAME) -L$(LIBDIR),
  • MTXLIBNAME=mtx
  • LIBDIR=$(SAGE_LOCAL)/lib

Building the executable, the output indeed contains

gcc -lmtx -L/home/king/Sage/git/sage/local/lib  -o makeInclusionMatrix mim.o libmodres.a

but aborts with an error, saying

/home/king/Projekte/coho/p_group_cohomology-3.0/src/present/src/mim.c:51: undefined reference to `MatFree'

But MatFree and other "undefined references" are defined in /home/king/Sage/git/sage/local/lib/libmtx.a!

What is happening here?

comment:70 Changed 4 years ago by SimonKing

When I instead do

gcc -L/home/king/Sage/git/sage/local/lib  -o makeInclusionMatrix mim.o pgroup.o pincl.o -lmtx

then it works. But why? And why did the other command not work?

The difference is that it says pgroup.o pincl.o instead of libmodres.a. And libmodres.a was created with ar r libmodres.a pincl.o pgroup.o aufloesung.o. Shouldn't then pgroup.o and pincl.o be available in libmodres.a anyway?

comment:71 in reply to: ↑ 69 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

I think I need help, as I am virtually a beginner with Makefiles, compiling c-sources, and/or creating a library.

If you're manually writing a Makefile, you're already doing it wrong. The only way to reliably build a library is to use libtool.

comment:72 in reply to: ↑ 71 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

If you're manually writing a Makefile, you're already doing it wrong. The only way to reliably build a library is to use libtool.

OK, then MeatAxe is doing it wrong as well, but after all it is third party code and I won't touch it.

Can you point me to a good introduction to libtool? What I need to create is (a) a couple of executables and (b) a library against which I can link my cython code. So, the library is not all!

comment:73 in reply to: ↑ 72 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

OK, then MeatAxe is doing it wrong as well

Absolutely. Don't use the meataxe build system as an example.

Can you point me to a good introduction to libtool?

I don't know. For an example in Sage, you can look at the planarity package.

comment:74 in reply to: ↑ 73 Changed 4 years ago by dimpase

Replying to jdemeyer:

Replying to SimonKing:

OK, then MeatAxe is doing it wrong as well

Absolutely. Don't use the meataxe build system as an example.

Can you point me to a good introduction to libtool?

I don't know. For an example in Sage, you can look at the planarity package.

as far as I can see, meataxe does not depend on any external libraries, right? Then it's really easy to autotoolize/libtoolize. An example of a Sage package I can suggest is csdp (well, I did this libtoolization for it :-)) https://github.com/dimpase/csdp it's more complicated as it needs Lapack and Blas.

I'll email you a pdf of a readable book about autotools, so that you can get an idea what's going on. (there is no readable free online up to day text for these purposes, AFAIK).

comment:75 Changed 4 years ago by SimonKing

Still I'd like to understand what went wrong in the scenario of comment:69 and comment:70. After all, before I can autotoolize it, I should probably first be able to build it.

Last edited 4 years ago by SimonKing (previous) (diff)

comment:76 Changed 4 years ago by vbraun

As to what went wrong, I think gcc will not use static libraries if you just list them on the command line. You need -lmodres.

While building binaries is somewhat the same on all platforms, the process of building a shared library is very platform dependent. Hence Jeroen's very good advice about libtool.

comment:77 Changed 4 years ago by SimonKing

Meanwhile I really see the point of using autotools. Perhaps the experts (i.e. you) have comments/suggestions on the following, but to me it seems to work.

src/Makefile.am

AM_CFLAGS = -O3

# -----> Shared library
lib_LIBRARIES               = libmodres.a
libmodres_a_SOURCES         = pincl.c pgroup.c aufloesung.c

# -----> Headers
include_HEADERS             = modular_resolution.h

# -----> Executable (built from the shared library)

bin_PROGRAMS = makeActionMatrices makeNontips groupInfo perm2Gap makeInclusionMatrix
makeActionMatrices_SOURCES  = mam.c
makeActionMatrices_LDADD    = $(lib_LIBRARIES)
makeNontips_SOURCES         = mnt.c
makeNontips_LDADD           = $(lib_LIBRARIES)
groupInfo_SOURCES           = gi.c
groupInfo_LDADD             = $(lib_LIBRARIES)
perm2Gap_SOURCES            = perm2Gap.c
perm2Gap_LDADD              = $(lib_LIBRARIES)
makeInclusionMatrix_SOURCES = mim.c
makeInclusionMatrix_LDADD   = $(lib_LIBRARIES)

configure.ac

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ([2.69])
AC_INIT([modular_resolution], [1.0], [simon.king@uni-jena.de])
AM_INIT_AUTOMAKE
LT_INIT
AC_CONFIG_SRCDIR([src/fp_decls.h])
AC_CONFIG_HEADERS([config.h])

# Checks for programs.
AC_PROG_CC
AC_PROG_INSTALL

# Checks for libraries.
AC_SEARCH_LIBS([MtxError], [mtx])

# Checks for header files.
AC_CHECK_HEADERS([stdlib.h string.h unistd.h meataxe.h])

# Checks for typedefs, structures, and compiler characteristics.
AC_CHECK_HEADER_STDBOOL
AC_C_INLINE
AC_TYPE_SIZE_T

# Checks for library functions.
AC_FUNC_MALLOC

AC_CONFIG_FILES([Makefile src/Makefile])
AC_OUTPUT

So far I only see a single change needed: My email address will very soon change.

comment:78 Changed 4 years ago by SimonKing

Can you help me to solve the following problem, please?

I would like to add a little test script that relies on a file test.reg that can be found in the same directory as modular_resolution.h.

In configure.ac, I have AC_CONFIG_SRCDIR([src/modular_resolution.h])

When I go to a new directory and do

/path/to/sources/of/modular_resolution/configure --prefix=`pwd`

then make check works. The file test.reg is sought in /path/to/source/of/modular_resolution/src, where it can indeed be found.

However, when I do make distcheck (which is a target provided by autotools), then first the distdir is created, and in distdir/_build the package is built and tested. Building it works fine. However, the test fails, because this time test.reg is sought in ../../src. Apparently the test is executed in distdir/_build, not in distdir/_build/src. Thus, it should be sought in ../src.

In other words, when doing "make check", an absolute path is used, whereas in "make distcheck" a wrong relative path is used.

How can that be fixed?

comment:79 follow-up: Changed 4 years ago by jdemeyer

Use $(srcdir) to refer to the source directory in Makefile.am

Another thing: avoid AM_CFLAGS = -O3. Just pass whatever flags you want for Sage to configure.

comment:80 in reply to: ↑ 79 Changed 4 years ago by SimonKing

Replying to jdemeyer:

Use $(srcdir) to refer to the source directory in Makefile.am

I did use it. Meanwhile I could fix the problem: the test script was in fact passing a wrong command line argument to some executable.

comment:81 follow-up: Changed 4 years ago by SimonKing

How to indicate that the package-to-be depends both on meataxe and I guess as well on some toolchain? I see the following in some dependencies files:

$(INST)/$(SAGE_MP_LIBRARY) $(INST)/$(ZLIB)

----------
All lines of this file are ignored except the first.
It is copied by SAGE_ROOT/build/make/install into SAGE_ROOT/build/make/Makefile.

or

$(INST)/$(MARKUPSAFE) $(INST)/$(SETUPTOOLS) $(INST)/$(DOCUTILS)

----------
All lines of this file are ignored except the first.
It is copied by SAGE_ROOT/build/make/install into SAGE_ROOT/build/make/Makefile.

The developer manual doesn't seem to mention it.

comment:82 in reply to: ↑ 81 Changed 4 years ago by SimonKing

Meanwhile I have a version of the C-part of the old group cohomology package that installs fine and even passes a self-test (namely tests that several of the installed executables work together to build the basic data for the cohomology computation of the dihedral group of order 8).

However, so far I have no idea how to make it so that the package is only built if meataxe (from #12103) is installed.

And then of course I have to put all Singular and Gap files with the appropriate file name extensions into the appropriate folders, and put the Cython and Python code into the Sage library, as an optional extension.

If I understand correctly, it is needed to add "optional: modular_resolution" basically to each single line of test (several hundreds, I guess). Is there really no way to state that a whole file should only be tested optionally? I believe it is a fairly natural thing to test an optional extension optionally.

comment:83 Changed 4 years ago by SimonKing

  • Branch set to u/SimonKing/upgrade_of_group_cohomology_spkg

comment:84 follow-up: Changed 4 years ago by SimonKing

  • Commit set to ad6cea059d383bbb0a97c8430caca125d2ca78b7

I have extracted the second stand-alone ingredient of the group cohomology package.

The next step will be to relocate all the Cython and Python source files from the old package into the SageMath library, say, sage.groups.modular_cohomology.

And the final step will be to fix the doctests somehow. I think there should be a way to declare that the tests of all files in sage.group.modular_cohomology are optional.

Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the to-be-created wrapper in the Sage library. Since database_gap is non-GPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages? Or better move the dependency to the wrapper?


Last 10 new commits:

c7d75fbImplement and use Strassen-Winograd matrix multiplication in MeatAxe
9e2a8c6A very basic MeatAxe Cython wrapper
4bdd285A full wrapper for MeatAxe matrices
d889e0bImprove echelon computation in MeatAxe, and fix some compiler warnings
2e64257Doctests and error handling for MeatAxe
7c38969Fix computation of row-reduced echelon form
55a278dFix doctests when meataxe is installed
f733377Use and propagate specific return values on error in matrix-related MeatAxe functions.
efa1d75Merge branch 't/12103/meataxe' into t/18514/upgrade_of_group_cohomology_spkg
ad6cea0Add package modular_resolution

comment:85 Changed 4 years ago by jdemeyer

  • Description modified (diff)

Please update the ticket description (the last comment you made is a good starting point).

comment:86 in reply to: ↑ 84 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the to-be-created wrapper in the Sage library. Since database_gap is non-GPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages?

No, that's not possible within the dependency framework. Also, I personally don't think it's a good idea to add this kind of interactive install procedures.

Or better move the dependency to the wrapper?

The wrapper will become part of the Sage library, right? So what do you mean with "move the dependency to the wrapper"?

comment:87 in reply to: ↑ 86 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the to-be-created wrapper in the Sage library. Since database_gap is non-GPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages?

No, that's not possible within the dependency framework. Also, I personally don't think it's a good idea to add this kind of interactive install procedures.

Then we might be in trouble. If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a non-GPL package without asking the user.

Perhaps I should elaborate on the dependency on database_gap. The code does not link against database gap. However, it needs to be able to deal with elementary abelian groups. Sounds trivial, and it can certainly be solved abstractly, but for now the elementary abelian group is simply looked up in the small groups library.

Also, since the p_group_cohomology package is some kind of data base for cohomology rings, it is needed to assign a unique identifier to a group-with-a-fixed-minimal-generating-set. That is easier with the small groups library, but the package also allows to assign a name to a group (e.g., Co_3 for the third Conway group), and use the name as unique identifier for some item in the cohomology ring database (the fixed minimal generating set is stored along with the ring).

Or better move the dependency to the wrapper?

The wrapper will become part of the Sage library, right? So what do you mean with "move the dependency to the wrapper"?

Similarly to the MeatAxe wrapper, I plan to use OptionalExtension(). Hence, the Cython code would only result in an extension module under the condition that both modular_resolution and database_gap are installed.

So, if the current framework does not allow that installation of modular_resolutions stops for asking the users permission to install database_gap first, then we can make it so that modular_resolution only makes sure that meataxe is installed (no problem, both are GPL compatible, and modular_resolution will build and test fine), and then the OptionalExtension() will depend on both modular_resolution *and* database_gap.

comment:88 in reply to: ↑ 87 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a non-GPL package without asking the user.

Where did you get this idea from?

comment:89 Changed 4 years ago by SimonKing

  • Description modified (diff)

comment:90 in reply to: ↑ 88 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a non-GPL package without asking the user.

Where did you get this idea from?

From discussions on sage-devel.

comment:91 Changed 4 years ago by SimonKing

  • Description modified (diff)

comment:92 in reply to: ↑ 90 Changed 4 years ago by jdemeyer

Replying to SimonKing:

From discussions on sage-devel.

I recall that discussion and I asked exactly the same question. I don't think that was ever answered. I think it was even mentioned that we should only ask this question as a courtesy to the user, not that it was a requirement.

comment:93 follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

Similarly to the MeatAxe wrapper, I plan to use OptionalExtension(). Hence, the Cython code would only result in an extension module under the condition that both modular_resolution and database_gap are installed.

Surely, the module will compile without database_gap. So I think you should just add meataxe as dependency for the Cython module.

The need of database_gap will make it fail at runtime. That's no problem. It's even more user-friendly, since you can show a nice error message at runtime if database_gap is not installed.

comment:94 in reply to: ↑ 93 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

The need of database_gap will make it fail at runtime. That's no problem. It's even more user-friendly, since you can show a nice error message at runtime if database_gap is not installed.

OK, sounds good, except for doctests: It means that all tests in the OptionalExtension need to be marked as optional: modular_resolution database_gap, which is rather awkward to do hundreds of times. Unless of course one can somehow declare that ALL tests of a file are optional.

comment:95 in reply to: ↑ 94 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

It means that all tests in the OptionalExtension need to be marked as optional: modular_resolution database_gap, which is rather awkward to do hundreds of times.

Well, it's not more awkward than just optional: modular_resolution hundreds of times :-)

Unless of course one can somehow declare that ALL tests of a file are optional.

I think this is intentionally not supported. A doctest should make sense by itself, not depend on external context like the file it appears in.

comment:96 in reply to: ↑ 95 Changed 4 years ago by SimonKing

Replying to jdemeyer:

I think this is intentionally not supported. A doctest should make sense by itself, not depend on external context like the file it appears in.

Does a cohomology computation make sense if your CAS doesn't know about cohomology??

comment:97 Changed 4 years ago by jdemeyer

  • Dependencies changed from #18494 to #12103

comment:98 follow-up: Changed 4 years ago by SimonKing

It turns out that I need to expose more stuff in modular_resolution.h. My wrapper in the old spkg imports functions from various headers but I suppose it is nicer if modular_resolution-1.0 only installs a single header, modular_resolution.h, and not in addition nDiag.h etc.

comment:99 in reply to: ↑ 98 ; follow-up: Changed 4 years ago by SimonKing

Replying to SimonKing:

It turns out that I need to expose more stuff in modular_resolution.h. My wrapper in the old spkg imports functions from various headers but I suppose it is nicer if modular_resolution-1.0 only installs a single header, modular_resolution.h, and not in addition nDiag.h etc.

Or should I better create a folder SAGE_LOCAL/include/modular_resolution/, where *all* headers of the package will be put?

comment:100 in reply to: ↑ 99 Changed 4 years ago by dimpase

Replying to SimonKing:

Replying to SimonKing:

Or should I better create a folder SAGE_LOCAL/include/modular_resolution/, where *all* headers of the package will be put?

sure, that's the way to.

comment:101 Changed 4 years ago by SimonKing

Now I have a technical problem: I can not scp the new tarball to my account in Jena. rcp doesn't work (I suppose Jena is blocking it), scp doesn't work (my landlord is blocking ssh).

Do you have a better solution than to instrument sagemathcloud as a tool to scp the file?

comment:102 Changed 4 years ago by SimonKing

It doesn't work via sagemathcloud either: ssh: connect to host login.minet.uni-jena.de port 22: Network is unreachable

comment:103 Changed 4 years ago by dimpase

as a last resort, share it on sagemathcloud, and we'll put it somewhere.

comment:104 Changed 4 years ago by SimonKing

I need to work on the package a bit more anyway: As it turns out, I didn't put into the library what I want to import later. So, I can post the new package tomorrow, when I am at university.

comment:105 follow-up: Changed 4 years ago by SimonKing

Odd. When I try to use the resulting library in Sage, I get the complaint

/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: /home/king/Sage/git/sage/local/lib/libmodres.a(pgroup.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/home/king/Sage/git/sage/local/lib/libmodres.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

Is it not the case that autotools automatically takes care of -fPIC?

comment:106 in reply to: ↑ 105 Changed 4 years ago by SimonKing

Replying to SimonKing:

Is it not the case that autotools automatically takes care of -fPIC?

Apparently not. OK, I simply have to add it to Makefile.am.

comment:107 follow-up: Changed 4 years ago by vbraun

Autotools by itself doesn't care about pic (since it doesn't know anything about shared libraries, really). Libtool does the right thing, though.

Also, I'm confused: libmodres.a is a static library, but the ld error message refers to shared library creation.

comment:108 Changed 4 years ago by git

  • Commit changed from ad6cea059d383bbb0a97c8430caca125d2ca78b7 to 499dfbd2a4a25d3ab2f089d36e0d592ba6a0d99e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

499dfbdAdd package modular_resolution

comment:109 Changed 4 years ago by SimonKing

I have updated the tarball, consequently updated the checksum, and added a .pxd header for the package. The package contains a test ("Out of given basic data for the dihedral group of order 8, compute some additional data, and test that these data are readable and contain a certain result").

A little sanity test that I did:

sage: cython("""#!clib modres mtx
....: from sage.libs.modular_resolution cimport *
....: def test():
....:     cdef group_t *G = newGroupRecord()
....:     freeGroupRecord(G)
....: """)
sage: test()

comment:110 in reply to: ↑ 107 Changed 4 years ago by SimonKing

Replying to vbraun:

Also, I'm confused: libmodres.a is a static library, but the ld error message refers to shared library creation.

Good point. Out of lack of experience I have no preference for either static or dynamic libraries (I heard that the former can be a little faster, but not by much). And I am really not sure if my Makefile.am results in a dynamic or a static library:

lib_LIBRARIES               = libmodres.a
libmodres_a_SOURCES         = aufloesung.c aufnahme.c fileplus.c nBuchberger.c pgroup.c pincl.c slice.c urbild.c
libmodres_a_CFLAGS          = -fPIC

I guess it is dynamic. But how to create a static library (if that's desired)?

comment:111 Changed 4 years ago by SimonKing

libmodres_a_LDFLAGS = -static?

comment:112 Changed 4 years ago by SimonKing

If I should change to creating a static library then please tell me how, and if you say that I am creating the dynamic library in the wrong way then please tell me how to do better.

comment:113 follow-up: Changed 4 years ago by vbraun

Always prefer shared libraries over static libraries

With libtool the Makefile.am should contain

lib_LTLIBRARIES = libmodres.la
libmodres_la_SOURCES = aufloesung.c aufnahme.c fileplus.c nBuchberger.c pgroup.c pincl.c slice.c urbild.c

Are the upstream meataxe filenames already in German?

comment:114 in reply to: ↑ 113 Changed 4 years ago by SimonKing

Replying to vbraun:

Always prefer shared libraries over static libraries

OK.

With libtool the Makefile.am should contain

lib_LTLIBRARIES = libmodres.la
libmodres_la_SOURCES = aufloesung.c aufnahme.c fileplus.c nBuchberger.c pgroup.c pincl.c slice.c urbild.c

So, the difference is that it is lib_LTLIBRARIES, not lib_LIBRARIES, and that the file name extension is .la, not .a.

What about -fPIC? Well, I will see if it is still needed.

Are the upstream meataxe filenames already in German?

Don't mix the two things ;-)

  • Meataxe is at #12103, the filenames are English, I only added one file window.c (thus, English as well), and I will *not* do upstream's job of autotoolisation.
  • Here is modular_resolution, which was originally written by a native English speaker in Wuppertal, and he chose the filenames. By rebasing it on top of the new meataxe and by using autotools, I became part of upstream, but I didn't change the filenames.

After all, SageMath is an international project...

By the way, I am really glad that you guys convinced me of using autotools. It really is so much easier!

Last edited 4 years ago by SimonKing (previous) (diff)

comment:115 follow-ups: Changed 4 years ago by SimonKing

There is one more file whose new location has to be determined: GroupNames.sobj. Thus, it is a Sage pickle, and it contains a dictionary assigning human-readable names (such as DihedralGroup(8)) to some addresses from the small groups library (such as SmallGroup(8,3)).

So, questions:

  • What is a natural location for a data file that is part of the SageMath distribution?
  • Do you happen to know a Gap function that can provide human-readable names (and other useful information) for a wider range of groups?

comment:116 Changed 4 years ago by git

  • Commit changed from 499dfbd2a4a25d3ab2f089d36e0d592ba6a0d99e to cd644a301bd2a9c57b5c719bb47a4733c75e347b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

cd644a3Add package modular_resolution

comment:117 Changed 4 years ago by SimonKing

I have posted an updated version of the upstream tarball. After changing from lib_LIBRARIES to lib_LTLIBRARIES, using -fPIC was not needed any longer.

Now, I try to move the Cython wrappper.

comment:118 in reply to: ↑ 115 Changed 4 years ago by SimonKing

Replying to SimonKing:

  • What is a natural location for a data file that is part of the SageMath distribution?

Perhaps I should add that the old spkg used several default locations for data.

  • SAGE_SHARE/pGroupCohomology for data that are obtained during installation of the package, which are shared among different users of one Sage installation, and which are not supposed to be modified by a user, unless (s)he knows what (s)he is doing and has write permission.
  • DOT_SAGE/pGroupCohomology for data that are created by a user, and which are only available to this user.

Is SAGE_SHARE/pGroupCohomology still a good location for shared data? If that is the case, then I think GroupNames.sobj should go there. But how? If I understand correctly, SAGE_SHARE is not under version control. Is it a possibility to add these data files in the modular_resolution tarball, and modifying its Makefile so that the data are installed in SAGE_SHARE/pGroupCohomology?

comment:119 in reply to: ↑ 115 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

There is one more file whose new location has to be determined: GroupNames.sobj. Thus, it is a Sage pickle

Does it really have to be a pickle? Why not just a Python file?

comment:120 in reply to: ↑ 119 ; follow-up: Changed 4 years ago by SimonKing

Replying to jdemeyer:

Does it really have to be a pickle? Why not just a Python file?

Well, I could also put it directly into the code (the file is loaded in exactly one spot in the cohomology code). So, right, I could get rid of the pickle.

But can the shared database remain in SAGE_SHAR/pGroupCohomology? I think it is reasonable that the new modular_resolution package shall provide the cohomology ring data base, even though the actual code to access the data will only be in the Sage library.

comment:121 in reply to: ↑ 120 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

But can the shared database remain in SAGE_SHARE/pGroupCohomology?

Of course. Just as an example, the conway_polynomials database is also in $SAGE_SHARE.

comment:122 in reply to: ↑ 121 Changed 4 years ago by SimonKing

Replying to jdemeyer:

Of course. Just as an example, the conway_polynomials database is also in $SAGE_SHARE.

Hrmph. I need more help.

I created a data/ directory and put the untar'd cohomology data into that directory. Thus, data/ now has 268 sub-directories, namely 8gp3/ and 64gp1/ up to 64gp267/. The aim is to copy all these sub-directories into $(prefix)/share/pGroupCohomology (and to create $(prefix)/share/pGroupCohomology in the first place).

What I tried is to put the following into data/Makefile.am:

dbdir    = $(prefix)/share/pGroupCohomology
nobase_db_DATA  = $(wildcard *gp*)

expecting that "make install" would create the directory and then recursively (because of nobase_) copy all folders *gp* into it.

However, autoreconf complains that wildcard *gp*: non-POSIX variable name, and when I do make install, then there is no share/pGroupCohomology, and from the log I don't see any attempt to install any data.

What do I need to do instead?

comment:123 follow-up: Changed 4 years ago by SimonKing

Perhaps I should have a single tar file containing all the data, and then add an install-data-hook that untars the data file?

comment:124 in reply to: ↑ 123 Changed 4 years ago by SimonKing

Replying to SimonKing:

Perhaps I should have a single tar file containing all the data, and then add an install-data-hook that untars the data file?

It works *almost*: I can do make install and make clean and make uninstall and so on, but nonetheless make distcheck fails. And I don't understand why.

I have this in the top-level Makefile.am:

ACLOCAL_AMFLAGS = -I m4
SUBDIRS = src

dbdir      = $(datadir)/pGroupCohomology
dist_db_DATA    = group_cohomology_database.tar

install-data-hook:
        cd $(dbdir) && tar xf $(dist_db_DATA)

uninstall-hook:
        rm -r $(dbdir)

clean-local:
        rm $(dbdir)/$(dist_db_DATA)

When I run make distcheck, then the command cd $(dbdir) in install-data-hook fails. The odd reason: datadir is /home/king/Projekte/coho/bin/modular_resolution-1.0/_inst/share (I checked that), but during "make distcheck" the data are installed NOT in datadir but in /tmp/am-dc-11743//home/king/Projekte/coho/bin/modular_resolution-1.0/_inst/share/pGroupCohomology. Consequently, when I want to cd into dbdir, the folder can't be found.

Do you have an idea why distcheck installs data not into datadir but into a temporary folder, whereas make install correctly installs into datadir?

comment:125 Changed 4 years ago by SimonKing

Got it. make distcheck uses a staged install, using the DESTDIR variable. And I somewhere found the comment: "Support for DESTDIR is implemented by coding it directly into the install rules. If your Makefile.am uses a local install rule (e.g., install-exec-local) or an install hook, then you must write that code to respect DESTDIR."

comment:126 Changed 4 years ago by git

  • Commit changed from cd644a301bd2a9c57b5c719bb47a4733c75e347b to 9999966fb6d48fa4516e7d11220efab69ed457e2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

9999966Add package modular_resolution

comment:127 Changed 4 years ago by SimonKing

Interesting commit number: 9999966...

Anyway. I have again updated the upstream tarball. Now, it contains the database of the modular cohomology rings of all groups of order 64 that is part of the old spkg. Recall that a much larger database is not part of the tarball but is available in internet.

And of course it will be needed to have code in the Sage library for actually reading the data. But I hope that's ok.

comment:128 follow-up: Changed 4 years ago by SimonKing

I can easily put the Gap code into $SAGE_ROOT/src/ext/gap/modular_cohomology/: It is under version control.

However, $SAGE_ROOT/local/share/singular is not under version control. Does that mean I should better use a different location for the Singular code?

comment:129 in reply to: ↑ 128 ; follow-up: Changed 4 years ago by jdemeyer

Replying to SimonKing:

Singular code?

If it's Singular code, it should go into src/ext/singular by analogy with src/ext/gap and src/ext/pari.

comment:130 in reply to: ↑ 129 Changed 4 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Singular code?

If it's Singular code, it should go into src/ext/singular ...

... which I need to create first, but of course I can do.

comment:131 follow-up: Changed 4 years ago by SimonKing

A question about copyright statements: When I put all the Cython and Python files into the Sage library, should I preserve the old copyright statements such as Copyright (C) 2009 Simon A. King <simon.king@uni-jena.de>, or should I add new ones, i.e. Copyright (C) 2015 Simon A. King <simon.king@uni-koeln.de>?

comment:132 in reply to: ↑ 131 Changed 4 years ago by dimpase

Replying to SimonKing:

A question about copyright statements: When I put all the Cython and Python files into the Sage library, should I preserve the old copyright statements such as Copyright (C) 2009 Simon A. King <simon.king@uni-jena.de>, or should I add new ones, i.e. Copyright (C) 2015 Simon A. King <simon.king@uni-koeln.de>?

well, it's good to have an up to date contact info there, while keeping the copyright dates, such as Copyright (C) 2009, 2015 Simon A. King <simon.king@uni-koeln.de>

comment:133 Changed 4 years ago by SimonKing

  • Work issues changed from Create a new-style package with least effort to Debug the modification of the modular resolution code

comment:134 Changed 4 years ago by SimonKing

Information on "Debug the modification of the modular resolution code": The old meataxe had row and column numbers one-based, the new has it zero-based. It seems to me that in some locations of the modular resolution code, this needs to be coped with.

comment:135 in reply to: ↑ description Changed 3 years ago by SimonKing

The work is almost done, but according to discussions on sage-devel I should not proceed according to the original ticket description. Hence, in this comment, I explain why, and then I will change the ticket description.

Replying to SimonKing:

The current "official" version of p_group_cohomology is 2.1.4. However, due to recent backward incompatible changes in Sage, the package would not install, respectively it would install if some header were present but wouldn't work. Note that such backward incompatible changes happened repeatedly.

And it has happened again: Meanwhile I offer an old-style spkg version 2.1.6, and both new versions do nothing but coping with backward incompatible changes.

The new package shall be modularised as follows.

  • First, install meataxe (see #12103) and database_gap.

This is still the case --- based on meataxe, there is an alternative implementation of matrices in the Sage src tree, and that will be used

Note that soon I will create a ticket to add a few methods to the meataxe wrapper that I intend to use in my cohomology programs.

  • Third, get the sources for modular_resolution-1.0. It is based on code of David Green, which I refactored rather extensively:

... The next step will be to relocate all the Cython and Python source files from the old package into the SageMath library, say, sage.groups.modular_cohomology.

I was told on sage-devel to not move the code to the Sage src tree, but make it an external package that depends on the meataxe spkg, and is formed by two parts: The first is a C library together with some executables provided by what above was called modular_resolution-1.0, and a pip-installable package formed by cython and python files. Reasons:

  • My package adds functionality, but doesn't modify existing functionality.
  • In order to use it, it isn't needed to cimport from my package.
  • People wouldn't like to have more than 10,000 lines of code in the src tree that is useless without some optional package.
  • There currently is no easy way to make all doctests in one file optional.

That said, I still believe that for me working in the src tree would be better; for the record:

  • Currently I do the development in the src tree anyway. So, moving stuff from the src tree into an external package is a nontrivial additional step for me.
  • Building the docs of python/cython code outside of the src tree is a nontrivial additional step for me.
  • Being in the src tree would make it less likely that backward incompatible Sage changes breaking my package will happen unnoticed again.
  • Having the documentation of my optional package in the Sage docs would make more people aware, so that they would be more likely to use my programs in research.

comment:136 Changed 3 years ago by SimonKing

  • Branch u/SimonKing/upgrade_of_group_cohomology_spkg deleted
  • Commit 9999966fb6d48fa4516e7d11220efab69ed457e2 deleted
  • Description modified (diff)
  • Work issues Debug the modification of the modular resolution code deleted

comment:137 Changed 3 years ago by SimonKing

My old-style package does contain a setup.py for installing the cython and python code, and it is able to build a documentation (basically by a modified copy of an old version of Sage's docbuilder). However, very likely both is not how it should be done.

I have three requests to the reviewers:

  1. Please point me to a good example of a setup.py in a new-style spkg for installing Cython code that cimports Sage types (such as Element and Parent).
  2. Please point me to a good example of a new-style spkg that teaches me how to build a nice documentation based on docstrings.
  3. Please tell me what I need to do in order to have the cohomology spkg automatically rebuilt as soon as the cimported Sage types change.

comment:138 Changed 3 years ago by SimonKing

  • Dependencies changed from #12103 to #12103 #21437

I need some modifications to the MeatAxe wrapper, and I'd appreciate it to be reviewed tat #21437.

comment:139 Changed 3 years ago by SimonKing

  • Branch set to u/SimonKing/upgrade_of_group_cohomology_spkg

comment:140 Changed 3 years ago by SimonKing

  • Commit set to f1c0dbd51d0cab48b1c21f3faca9c7e2245764e4

I guess it is better to provide code at some point. There are many doctest failures (many of them just because of different logging), none of them is marked optional, but as much as I see it would be possible to do non-trivial computations with the new code version.

INSTALLATION

HOWEVER...

The current branch does everything in Sage's src tree, which (as I was told) is evil.

But perhaps some of you can answer the questions that I have posted earlier: What do I need to do with the current code base in order to create a pip-installable module and build its documentation?


Last 10 new commits:

93d59dcUse Python's logging module for logging
a6144a3Add reset-function, fix some tests
09e76a6New return format of verifyGroupIsAbelian; use some boilerplate
16c7912Replace MatEchelonize by full echelon computation; use set_unsafe_int resp. set_unsafe more carefully
36bcb8efix various doctests
2cc1070Fix MTX matrix slicing; fix doctest continuation
d900786Add docs to the reference manual. Remove Ordinals(), fix one occurence of get_slice.
d384968Use boilerplate instead of set_unsafe_int, since this is affected from meataxe being static library
8fbaad9_reset() now involves the logger. Fix some tests
f1c0dbdFix usage of some matrix related functions

comment:141 Changed 3 years ago by git

  • Commit changed from f1c0dbd51d0cab48b1c21f3faca9c7e2245764e4 to 710f6f5848226bffd0895ed30f93c46b421c8460

Branch pushed to git repo; I updated commit sha1. New commits:

ee392f5Make logging at 'info' level less verbose
710f6f5Let unit_test64 accept keyword arguments

comment:142 follow-ups: Changed 3 years ago by SimonKing

There are good news:

I have just finished unit_test_64, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.

The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.

Out of curiosity, I will repeat the test on the same machine with the old package version, and see how long it takes.

comment:143 in reply to: ↑ 142 Changed 3 years ago by SimonKing

Replying to SimonKing:

Out of curiosity, I will repeat the test on the same machine with the old package version, and see how long it takes.

It takes roughly the same time. Actually the old package was two minutes faster.

comment:144 in reply to: ↑ 142 ; follow-up: Changed 3 years ago by SimonKing

Replying to SimonKing:

I have just finished unit_test_64, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.

The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.

Since the old version was a little faster, I did some profiling. The profiler told me that much time is spent (both with the old and the new version) on do_system. With the help of sage-devel, I could track it down to several calls to system(), which all turned out to be useless.

After removing all the system calls, the performance has drastically improved! On the same machine, unit_test_64 now only needed 9:26 minutes wall time resp. 8:37 min CPU time, which is only about one third resp. one half of the old timing, and I guess a new world record for the computation of the modular cohomology rings of all groups of order 64.

I currently do not have the bandwidth to post the new version of modular_resolution-1.0.tar.gz, and will do so on Monday.

comment:145 Changed 3 years ago by git

  • Commit changed from 710f6f5848226bffd0895ed30f93c46b421c8460 to 0eed64d245f15296a3beda5535cabd3c417ed25c

Branch pushed to git repo; I updated commit sha1. New commits:

966ec0eFix documentation of unit_test_64
0aa2980Update modular_resolution-1.0, removing all system calls
3e130aeAlways try to save the basic ring setup on disc
d11e678Fix attribute reconstruction, to make unit_test_64 work with 'nosave'
e5f88a4Improve handling of global options
d9ffcf4Various small speedups
e17e4dfDocument and test _rowlist_()
291f4dcSome minor tweaks for performance
0eed64dMerge branch 't/21437/fix-mtx-matrices' into t/18514/upgrade_of_group_cohomology_spkg

comment:146 in reply to: ↑ 144 Changed 3 years ago by SimonKing

Replying to SimonKing:

Replying to SimonKing:

I have just finished unit_test_64, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.

The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.

Since the old version was a little faster, I did some profiling. The profiler told me that much time is spent (both with the old and the new version) on do_system. With the help of sage-devel, I could track it down to several calls to system(), which all turned out to be useless.

After removing all the system calls, the performance has drastically improved! On the same machine, unit_test_64 now only needed 9:26 minutes wall time resp. 8:37 min CPU time, which is only about one third resp. one half of the old timing, and I guess a new world record for the computation of the modular cohomology rings of all groups of order 64.

I currently do not have the bandwidth to post the new version of modular_resolution-1.0.tar.gz, and will do so on Monday.

Done.

And in addition I obtained some further minor improvements, basically by using a C profiler. The computation of the modular cohomology rings of all groups of order 64 on my laptop now takes about 6:50 min of wall time resp. 6:10 min of CPU time (plus the time spent by Singular and GAP, which counts for the wall time but not for the CPU time).

I started to do time series to optimise the default choice of some options, but the standard deviation is considerable. So, I guess I'll stick with the C profiler...

Let me come back to my question: How can I move this away from the src tree and make it an independent package while being able to build a nice documentation and to run doc tests? Note that all this did work in the old spkg, but certainly can be done a lot better.

comment:147 Changed 3 years ago by git

  • Commit changed from 0eed64d245f15296a3beda5535cabd3c417ed25c to 1c43593103884c96021568553cf37d2b3884bf3a

Branch pushed to git repo; I updated commit sha1. New commits:

1c43593Sanitise handling of global options

comment:148 Changed 3 years ago by git

  • Commit changed from 1c43593103884c96021568553cf37d2b3884bf3a to 54e1a0071d54fe3bc6189e2faa3af5d956b67f2a

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

aa1985aFix tests in resolution.pyx
5554cefUse SageMath's framework for _richcmp_ of elements
fbbdff9Use boilerplate to avoid transformation of matrix rows into lists
77bdc99Cache whether the group is abelian
3ce31d4Remove more instances of _rowlist_
d04b020update of modular_resolution
8d2c563Change modular_resolution to a more efficient monomial ordering
e8759a7Documentation of new functions. Fix doctest in cochain.pyx
3413fadFixing almost all tests in cohomology.pyx. Change Simon King's address
54e1a00Remove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx

comment:149 Changed 3 years ago by SimonKing

  • Description modified (diff)
  • Status changed from needs_work to needs_info

If MeatAxe was a standard spkg (which might actually be something to consider) and the SmallGroups library was GPL compatible (something which the authors refuse), I could promote the modular group cohomology computation as a new standard package: With the changes that I just pushed, all tests pass (on my machine at least), and moreover the computations have still become faster.

I have changed the status to "needs info", stating my information request in the ticket description.

comment:150 in reply to: ↑ description Changed 2 years ago by jdemeyer

Replying to SimonKing:

  • Please point me to an example of a new style spkg that builds a C library and Cython modules (not in Sage's src tree) that link against the library.

This is most easily treated as two separate things (the C library + the Cython wrapper). There are plenty examples for each of those two. Simple examples are https://github.com/graph-algorithms/edge-addition-planarity-suite for the C library and https://github.com/defeo/cypari2 for the Cython wrapper.

  • Please point me to an example of a new style spkg that builds documentation based on doc strings in the Cython code.

https://github.com/sagemath/cysignals (result: http://cysignals.readthedocs.io/en/latest/)

See also http://opendreamkit.org/2017/06/09/CythonSphinx/

These are all standard Sage packages (planarity, cypari and cysignals)

comment:151 Changed 2 years ago by SimonKing

  • Dependencies changed from #12103 #21437 to #12103 #21437 #23352

comment:152 Changed 2 years ago by git

  • Commit changed from 54e1a0071d54fe3bc6189e2faa3af5d956b67f2a to 63355a85360616cff0d4befb3d5e0c78b4d129e5

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

08b1724Use SageMath's framework for _richcmp_ of elements
b3156e4Use boilerplate to avoid transformation of matrix rows into lists
883dae6Cache whether the group is abelian
c80d8a5Remove more instances of _rowlist_
6100c34update of modular_resolution
c7a1ae9Change modular_resolution to a more efficient monomial ordering
33bcb59Documentation of new functions. Fix doctest in cochain.pyx
eb3b6e2Fixing almost all tests in cohomology.pyx. Change Simon King's address
8f7001aRemove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx
63355a8Get the cohomology code to build

comment:153 Changed 2 years ago by git

  • Commit changed from 63355a85360616cff0d4befb3d5e0c78b4d129e5 to ae029297d600471d77f3de795c9ad8d726988a3b

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d0102a4update of modular_resolution
919d7f7Change modular_resolution to a more efficient monomial ordering
15464b4Documentation of new functions. Fix doctest in cochain.pyx
72837b7Fixing almost all tests in cohomology.pyx. Change Simon King's address
67b3ccfRemove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx
5360988Get the cohomology code to build
aa07e43Modified handling of MtxLibDir
32a411dUse a fixed copy of MtxLibDir
55ddf00More clearly document from_scratch option
ae02929Use meataxe_init() instead of manually setting the parameters

comment:154 follow-up: Changed 2 years ago by SimonKing

  • Dependencies changed from #12103 #21437 #23352 to #21437

Now I guess the meataxe problems are basically dealt with in #23352, #23400, #23399 and #21437. Therefore I have rebased the current cohomology code.

The current layout is:

  1. MeatAxe is a separate optional package
  2. The C code of David Green to compute minimal resolutions is a separate package, which also installs the data base of cohomology rings of all groups of order 64 and which also installs certain GAP and Singular code interpreted modules.
  3. The Python and Cython code is in the Sage source tree. The Cython code is optional extension module, which needs the package from 2. as a build time dependency. Moreover, at run time, it depends on the small groups library.

What layout should there be, instead? I guess MeatAxe has to stay separate. But shall everything else be included in one optional (new-style) spkg? So, that spkg would provide 2. and 3.?

comment:155 in reply to: ↑ 154 ; follow-up: Changed 2 years ago by SimonKing

Replying to SimonKing:

Now I guess the meataxe problems are basically dealt with in #23352, #23400, #23399 and #21437. Therefore I have rebased the current cohomology code.

The current layout is:

  1. MeatAxe is a separate optional package
  2. The C code of David Green to compute minimal resolutions is a separate package, which also installs the data base of cohomology rings of all groups of order 64 and which also installs certain GAP and Singular code interpreted modules.
  3. The Python and Cython code is in the Sage source tree. The Cython code is optional extension module, which needs the package from 2. as a build time dependency. Moreover, at run time, it depends on the small groups library.

Apparently, the Cython code can not stay in the source tree, as it doesn't qualify as optional extension module.

Possible layouts:

  1. Two packages
    • meataxe (as it is now)
    • everything else put into one upstream tarball, giving rise to one new-style spkg
  1. Three packages
    • meataxe (as it is now)
    • C code in one upstream tarball, giving rise to a new-style spkg "modular_resolution" that provides a C library and some executables but would probably be of no interest on its own?
    • Cython and Python code in another upstream tarball, giving rise to another new-style spkg "p_group_cohomology" that depends on "modular_resolution".

Would it be OK to have an spkg "modular_resolution" if it basically has no other purpose than being a dependency for "p_group_cohomology"?

And I still wonder what to do with the GAP code and the Singular code. On the one hand, such code clearly belongs to src/ext/gap respectively to src/ext/singular. On the other hand, an spkg is not supposed to inject code into src/ext.

Is it ok to put the relevant files into src/ext even when the optional packages are not installed? I would think so; after all, the files I am talking about are not particularly large.

comment:156 in reply to: ↑ 155 ; follow-up: Changed 2 years ago by tscrim

Replying to SimonKing:

Apparently, the Cython code can not stay in the source tree, as it doesn't qualify as optional extension module.

Can you elaborate a bit more on this? I don't quite understand why not from a quick glance.

Possible layouts:

  1. Two packages
    • meataxe (as it is now)
    • everything else put into one upstream tarball, giving rise to one new-style spkg
  1. Three packages
    • meataxe (as it is now)
    • C code in one upstream tarball, giving rise to a new-style spkg "modular_resolution" that provides a C library and some executables but would probably be of no interest on its own?
    • Cython and Python code in another upstream tarball, giving rise to another new-style spkg "p_group_cohomology" that depends on "modular_resolution".

Would it be OK to have an spkg "modular_resolution" if it basically has no other purpose than being a dependency for "p_group_cohomology"?

Yes, this is perfectly fine to have this kind of granularity. Although if it could not have any independent interest/development, then it probably would be better to fold it into p_group_cohomology from a more macro-scale maintenance viewpoint. I guess it makes the p_group_cohomology less homogeneous as a project? I also had the impression that it would be able to be a standalone library.

And I still wonder what to do with the GAP code and the Singular code. On the one hand, such code clearly belongs to src/ext/gap respectively to src/ext/singular. On the other hand, an spkg is not supposed to inject code into src/ext.

I feel like those are no different than having Cython files whose compilation depends on an optional spkg or Python files that are only useful if an optional spkg is installed. So, by analogy, they should not be injected by the spkg during its installation, but instead part of the main Sage git control.

Is it ok to put the relevant files into src/ext even when the optional packages are not installed? I would think so; after all, the files I am talking about are not particularly large.

IMO, I think it is fine to put it in the source tree if it is meant to be something for Sage as suggested above. I'm not sure if I have a complete understanding of everything at this point to say anything definitive about what you should do in this situation. Also, someone might come along and say you should do it all in a pip package, but then I feel it is more likely to break on future upgrades of Sage.

comment:157 in reply to: ↑ 156 ; follow-up: Changed 2 years ago by SimonKing

Replying to tscrim:

Replying to SimonKing:

Apparently, the Cython code can not stay in the source tree, as it doesn't qualify as optional extension module.

Can you elaborate a bit more on this? I don't quite understand why not from a quick glance.

I have asked about it on sage-devel, and was told that a cython OptionalExtension should only be used if an existing functionality (i.e., a functionality that is provided by vanilla Sage) can be implemented in a different (better) way if the user has installed some optional package. Example: You have matrices over GF(p^n) with p>2 and n>1 in Vanilla Sage, but if you install meataxe then a much faster implementation is available. Or you could also think of different graph backends.

However, the situation here is different: This is about a functionality that is not available in vanilla Sage and can only be provided if the user installs some optional package. I was told on sage-devel that I shouldn't use OptionalExtension in that situation.

Would it be OK to have an spkg "modular_resolution" if it basically has no other purpose than being a dependency for "p_group_cohomology"?

Yes, this is perfectly fine to have this kind of granularity. Although if it could not have any independent interest/development, then it probably would be better to fold it into p_group_cohomology from a more macro-scale maintenance viewpoint. I guess it makes the p_group_cohomology less homogeneous as a project? I also had the impression that it would be able to be a standalone library.

David Green's code to compute resolutions of modular group algebras of finite p-groups has originally been a bunch of executables operating on data stored on disk. I made a standalone library out of it.

So, it could be used on its own, but I guess nobody has ever done.

Currently, I am developer for both the C / Gap code that I inherited from David Green, and for the Cython / Python / Singular code that I wrote myself. I do not foresee active development for the C code in the near future. So, from a development point of view, it would be easier to make a separate package out of the C code --- it is not supposed to change in future versions. On top of that, the active development will take place in the Cython / Python / Singular part of the code, which thus should be another separate package.

And I still wonder what to do with the GAP code and the Singular code. On the one hand, such code clearly belongs to src/ext/gap respectively to src/ext/singular. On the other hand, an spkg is not supposed to inject code into src/ext.

I feel like those are no different than having Cython files whose compilation depends on an optional spkg or Python files that are only useful if an optional spkg is installed. So, by analogy, they should not be injected by the spkg during its installation, but instead part of the main Sage git control.

It is more like "some code snippets which can be used by some optional package that the user may install". So, I could understand that we wouldn't like to have these code snippets in the src tree. However, they are supposed to be used in Sage's GAP and Singular, and thus it makes sense to put them into src/sage/libs. That's the dilemma...

Is it ok to put the relevant files into src/ext even when the optional packages are not installed? I would think so; after all, the files I am talking about are not particularly large.

IMO, I think it is fine to put it in the source tree if it is meant to be something for Sage as suggested above. I'm not sure if I have a complete understanding of everything at this point to say anything definitive about what you should do in this situation. Also, someone might come along and say you should do it all in a pip package, but then I feel it is more likely to break on future upgrades of Sage.

I just thought: What does database_gap do? It is installing databases into Sage's version of GAP. What is the purpose of the GAP / Singular files that are used in the cohomology package? Well, they are to be installed into Sage's version of GAP respectively Singular. So, I should try to understand how database_gap manages to install stuff into Sage's GAP without inserting stuff into Sage's src tree.

comment:158 in reply to: ↑ 157 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

It is more like "some code snippets which can be used by some optional package that the user may install". So, I could understand that we wouldn't like to have these code snippets in the src tree. However, they are supposed to be used in Sage's GAP and Singular, and thus it makes sense to put them into src/sage/libs. That's the dilemma...

An important question is how exactly are these GAP/Singular files going to be used? For example, database_gap and gap_packages contain GAP packages which are installed in exactly the same way as GAP packages would be installed outside of Sage. They are not used by Sage directly, only by the GAP-in-Sage. Would that work for you?

comment:159 in reply to: ↑ 158 Changed 2 years ago by SimonKing

Replying to jdemeyer:

An important question is how exactly are these GAP/Singular files going to be used? For example, database_gap and gap_packages contain GAP packages which are installed in exactly the same way as GAP packages would be installed outside of Sage. They are not used by Sage directly, only by the GAP-in-Sage. Would that work for you?

The GAP functions currently are in files and are loaded into GAP like this:

        if not gap('IsBoundGlobal("exportMTXLIB")'):
            gap.eval('Read("%s/src/ext/gap/modular_cohomology/GapMaxels.g");'%(SAGE_ROOT))
            gap.eval('Read("%s/src/ext/gap/modular_cohomology/GapMB.g");'%(SAGE_ROOT))
            gap.eval('Read("%s/src/ext/gap/modular_cohomology/GapSgs.g");'%(SAGE_ROOT))
            gap.eval('InstallValue(exportMTXLIB,"MTXLIB=%s; export MTXLIB; ")'%(os.path.join(DOT_SAGE,"meataxe")))

The Singular functions are in interpreted libraries that are loaded like this:

        singular.load('{}/dickson.lib'.format(os.path.join(SAGE_ROOT,'src','ext','singular')))
        singular.load('{}/filterregular.lib'.format(os.path.join(SAGE_ROOT,'src','ext','singular')))

It would be OK from my point of view to copy filterregular.lib and dickson.lib into the shared folder containing all Singular libraries. Then, one could simply do

        singular.LIB('dickson.lib')
        singular.LIB('filterregular.lib')

I certainly could make the modular cohomology package insert stuff into SAGE_LOCAL/share/singular/LIB/. Actually I'd prefer this.

And it would also be OK to have some kind of GAP package, so that one could say

        gap.LoadPackage('WhateverThisPackageIsCalled')

However, I simply don't know how to create a Gap package, thus, I'd appreciate a pointer.

comment:160 follow-up: Changed 2 years ago by dimpase

have a look at https://github.com/gap-packages/example

(if your GAP code does not call external programs, this is an overkill

comment:161 in reply to: ↑ 160 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

have a look at https://github.com/gap-packages/example

Thank you for the pointer!

(if your GAP code does not call external programs, this is an overkill

Actually the GAP code is the only part of the group cohomology computation that calls external programs --- currently, for example, like that:

  Exec(Concatenation("mkdir -p ", dir));
...
  buffer := Concatenation(exportMTXLIB, "makeInclusionMatrix ", Gstem, " ", Hstem,
    " ", Istem);
  Print(buffer, "\n");
  Exec(buffer);

I somehow have the feeling that external programs (here: mkdir and !makeInclusionMatrix) shouldn't be called in that way. So, if gap_packages/example tells me how to do better, I'd be glad!

comment:162 in reply to: ↑ 161 ; follow-up: Changed 2 years ago by SimonKing

Replying to SimonKing:

(if your GAP code does not call external programs, this is an overkill

Actually the GAP code is the only part of the group cohomology computation that calls external programs

Maybe I misunderstood what you meant be "call external programs". In contrast to gap-packages/example, my gap package would simply consist of a bunch of gap-readable files, but it would not provide the c sources for the external programs it is calling.

Anyway, is Exec(...) still the way to call programs from within GAP?

comment:163 in reply to: ↑ 162 ; follow-up: Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to SimonKing:

(if your GAP code does not call external programs, this is an overkill

Actually the GAP code is the only part of the group cohomology computation that calls external programs

Maybe I misunderstood what you meant be "call external programs". In contrast to gap-packages/example, my gap package would simply consist of a bunch of gap-readable files, but it would not provide the c sources for the external programs it is calling.

is there a reason for not packaging them in? Are they called from anywhere else?

Anyway, is Exec(...) still the way to call programs from within GAP?

Yes. But I suppose calling mkdir the way you do is a bit rough. GAP has its own facilities for that, see DirectoryTemporary.

comment:164 in reply to: ↑ 163 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

Replying to SimonKing:

Maybe I misunderstood what you meant be "call external programs". In contrast to gap-packages/example, my gap package would simply consist of a bunch of gap-readable files, but it would not provide the c sources for the external programs it is calling.

is there a reason for not packaging them in? Are they called from anywhere else?

The executables are part of a C code base which also provides a library. My Cython code is linked against that library. So, no, the executables aren't called anywhere else, but, yes, the package which the executables belong to is used somewhere else.

Anyway, is Exec(...) still the way to call programs from within GAP?

Yes. But I suppose calling mkdir the way you do is a bit rough. GAP has its own facilities for that, see DirectoryTemporary.

The directory to be created is supposed to be permanent.

comment:165 in reply to: ↑ 164 ; follow-up: Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

Replying to SimonKing:

Maybe I misunderstood what you meant be "call external programs". In contrast to gap-packages/example, my gap package would simply consist of a bunch of gap-readable files, but it would not provide the c sources for the external programs it is calling.

is there a reason for not packaging them in? Are they called from anywhere else?

The executables are part of a C code base which also provides a library. My Cython code is linked against that library. So, no, the executables aren't called anywhere else, but, yes, the package which the executables belong to is used somewhere else.

OK, makes sense then. This means it's simpler, in the sense that one won't really need much configure/makefile stuff, if at all.

Anyway, is Exec(...) still the way to call programs from within GAP?

Yes. But I suppose calling mkdir the way you do is a bit rough. GAP has its own facilities for that, see DirectoryTemporary.

The directory to be created is supposed to be permanent.

This sounds like trouble. E.g. think about a multi-user situation... Does it mean to be some soft of caching? There is a package like this in GAP, AtlasRep. There data is fetched from the net, rather than computed, IIRC.

comment:166 in reply to: ↑ 165 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

The directory to be created is supposed to be permanent.

This sounds like trouble. E.g. think about a multi-user situation...

I thought of a multi-user situation.

Does it mean to be some sort of caching? There is a package like this in GAP, AtlasRep. There data is fetched from the net, rather than computed, IIRC.

It's kind of similar.

  • There is a repository in the web.
  • There is a local repository that all users of a Sage installation can read; I call this the public database. Of course, only a user with write privileges can add data to the public database.
  • Each user has his or her own local repository (I call it "private database"). The data in the private database is soft links to the public database (when the data is available there), otherwise is fetched from the web (when the data is available there), otherwise (or if explicitly requested) is computed from scratch.

In particular, in a multi-user setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.

comment:167 in reply to: ↑ 166 ; follow-up: Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

The directory to be created is supposed to be permanent.

This sounds like trouble. E.g. think about a multi-user situation...

I thought of a multi-user situation.

Does it mean to be some sort of caching? There is a package like this in GAP, AtlasRep. There data is fetched from the net, rather than computed, IIRC.

It's kind of similar.

  • There is a repository in the web.
  • There is a local repository that all users of a Sage installation can read; I call this the public database. Of course, only a user with write privileges can add data to the public database.
  • Each user has his or her own local repository (I call it "private database"). The data in the private database is soft links to the public database (when the data is available there), otherwise is fetched from the web (when the data is available there), otherwise (or if explicitly requested) is computed from scratch.

In particular, in a multi-user setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.

Then you perhaps can borrow their design. Surely Thomas Breuer, the main developer of AtlasRep, would be glad to help.

Last edited 2 years ago by dimpase (previous) (diff)

comment:168 in reply to: ↑ 167 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

In particular, in a multi-user setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.

Then you perhaps can borrow their design. Surely Thomas Breuer, the main developer of AtlasRep, would be glad to help.

On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?

comment:169 in reply to: ↑ 168 ; follow-up: Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

In particular, in a multi-user setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.

Then you perhaps can borrow their design. Surely Thomas Breuer, the main developer of AtlasRep, would be glad to help.

On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?

I haven't looked at its source for many years; it's reasonably well-documented, have a look.

comment:170 in reply to: ↑ 169 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?

I haven't looked at its source for many years; it's reasonably well-documented, have a look.

I just had a look, and apparently it is just a bunch of gap readable files. It seems that it contains no compiled code, in particular no linking against third party libraries.

comment:171 Changed 2 years ago by git

  • Commit changed from ae029297d600471d77f3de795c9ad8d726988a3b to 12e1bc50e4ef6ceedaaf30e66e1fd03d5d1cadf6

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

d4f8496Documentation of new functions. Fix doctest in cochain.pyx
282ae43Fixing almost all tests in cohomology.pyx. Change Simon King's address
99f2f3bRemove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx
10ea380Remove 'shebang lines' from modular_resolution's spkg-* scripts
4bf8addConversion of matrix data into Matrix_t* moved from matrix_gfpn_dense to libs/meataxe
c032ac7Add libraries that the cohomology code is to be linked with
13482a1Remove everything but CohomologyRing from __init__.py
a73ef8fFix import locations and types in cochain.*
83ec3ef(modular_)cohomology.pyx: Fix import locations, cope with changes in matrix_gfpn_dense
12e1bc5resolution.pyx: Fix import locations, cope with changes in matrix_gfpn_dense, fix dict keys in LIFTContainer

comment:172 in reply to: ↑ 170 ; follow-up: Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?

I haven't looked at its source for many years; it's reasonably well-documented, have a look.

I just had a look, and apparently it is just a bunch of gap readable files. It seems that it contains no compiled code, in particular no linking against third party libraries.

Why do you need the latter? Don't you just call executables? They call wget.

comment:173 Changed 2 years ago by SimonKing

  • Description modified (diff)

I got the code running! Probably some tests need fixing (some log strings may have changed, I am confident that the mathematical content didn't). Also, I should apply what I recently learnt about sig_on/sig_off vs. sig_check.

To be on the safe side, I tried to post the latest modular_resolution-1.0.tar.gz, but apparently there is a problem on the webserver in Jena, and it is too large to post here. Anyway, IIRC, the file at http://users.minet.uni-jena.de/cohomology/modular_resolution-1.0.tar.gz is still the correct version (not 100% sure though).

It certainly isn't ready for a formal review. The main part of the code still consists of cython OptionalExtension, which should eventually be moved into a pip installable module. Also, the location of the GAP and Singular code files should be fixed.

In any case, it should now provide the possibility to do some testing. Installation:

  • checkout the branch
  • sage -i database_gap
  • sage -i meataxe
  • copy modular_resolution-1.0.tar.gz into SAGE_ROOT/upstream
  • sage -i modular_resolution
  • make

IIRC, Sage won't build the documentation of OptionalExtension. For the documentation of the old version of the package, see http://users.minet.uni-jena.de/cohomology/documentation/, but note that in the current shape of the code, from pGroupCohomology import ... needs to be replaced by from sage.groups.modular_cohomology import ....

comment:174 in reply to: ↑ 172 ; follow-ups: Changed 2 years ago by SimonKing

Replying to dimpase:

Why do you need the latter? Don't you just call executables? They call wget.

The meataxe package provides a library "mtx" and some executables.

The C code in modular_resolution-1.0 (which is based on David Green's code but for which I became upstream) needs to link against mtx. It provides a library "modres" and some executables.

The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!) need to be linked both against mtx and against modres.

I am not calling the meataxe executables directly. However, IIRC, the executables provided by modular_resolution-1.0 do call some meataxe executables.

At several stages of the cohomology computation, I need to call the modular_resolution executables; it results in many data files stored on disk in special folders, all data together determine an initial segment of the minimal projective resolution of the group algebra, and the data is then used by Cython code to compute the ring structure of the cohomology ring.

The calls to modular_resolution executables partially occur in GAP functions, and GAP functions are also responsible to create the special folders. However, the path of the special folders and also the names of the data files are determined by the Cython code. So, it should (when needed) be possible to move all the calls from GAP code to Cython code.

Concerning wget: It is the responsibility of my Python code to check whether the data of a cohomology ring are found locally. If data is to be fetched from web, which is done with Python's urllib2 module. I guess urllib2 uses wget, but I am not using wget directly.

I hope that clarified the structure of the code base.

comment:175 Changed 2 years ago by SimonKing

PS: Perhaps I should tell why the data of a single cohomology ring are distributed in dozens of data files.

When developing the very first version of the group cohomology spkg, I tried to turn the modular_resolution executables into library functions that simply create the resolution data in memory, rather then storing them on disk. But I soon ran out of memory. So, I kept the executables and use the data files.

The cohomology computation is in increasing degree, and I originally tried to keep the cohomology data of lower degrees in memory when computing higher degrees. But I soon ran out of memory again. So, I started to cache data on disk.

Originally, I tried to pickle the cohomology data into one single file (plus all the files constituting the resolution data). However, in some examples, the process of pickling a cohomology ring took hours --- simply the data was too large. Hence, I started to dump cohomology data in several smaller files, basically one in each degree, and then some more defining the ring maps induced by inclusion of certain special subgroups.

I hope that explains why I care about how data are stored. I am sure that the computation of the mod-2 cohomology of the third Conway group or of all groups of order 128 would have been hardly possible when using a simple "one group, one file" approach to store data. Instead, it is "one group, one folder".

comment:176 in reply to: ↑ 174 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

The meataxe package provides a library "mtx" and some executables.

The C code in modular_resolution-1.0 (which is based on David Green's code but for which I became upstream) needs to link against mtx. It provides a library "modres" and some executables.

The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!) need to be linked both against mtx and against modres.

Is meataxe still a static library? I remember that we had some discussion about that. If you link from different libraries or Python modules to meataxe, every instance will get an independent copy of meataxe. This is generally considered to be bad practice. For example, I remember that meataxe uses some kind of data directory. That needs to be set for every individual copy of the meataxe static library. With a dynamic library on the other hand, a program uses only one copy of meataxe. Setting the data directory once should then be sufficient.

comment:177 in reply to: ↑ 174 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!)

If you plan to move it to a separate package, why take the detour via Sage? Maybe you could do it just for testing purposes, that might make sense. However, it seems like you are making things difficult for yourself if you first want your code to be merged in Sage and then taken out again as a separate package.

comment:178 in reply to: ↑ 176 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Is meataxe still a static library? I remember that we had some discussion about that. If you link from different libraries or Python modules to meataxe, every instance will get an independent copy of meataxe. This is generally considered to be bad practice. For example, I remember that meataxe uses some kind of data directory. That needs to be set for every individual copy of the meataxe static library. With a dynamic library on the other hand, a program uses only one copy of meataxe. Setting the data directory once should then be sufficient.

I totally agree with that, but recall that I am not upstream for MeatAxe (I only contributed patches to the current MeatAxe spkg). Hence, unless someone tells me how to easily change MeatAxe's Makefile, it will remain static. However, if there is an easy way to change things, I'll be happy to include yet another patch to the MeatAxe spkg.

comment:179 in reply to: ↑ 177 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!)

If you plan to move it to a separate package, why take the detour via Sage? Maybe you could do it just for testing purposes, that might make sense.

Exactly, that was my reason.

However, it seems like you are making things difficult for yourself if you first want your code to be merged in Sage and then taken out again as a separate package.

That's why I did not change this ticket to "needs review" in the last couple of years.

Last edited 2 years ago by SimonKing (previous) (diff)

comment:180 follow-up: Changed 2 years ago by dimpase

If database_gap is a dependency of meataxe (it seems so from the commands above) then it should be in build/pkgs/meataxe/dependencies.

Same commentary for build/pkgs/modular_resolution/dependencies - it should mention meataxe.

Last edited 2 years ago by dimpase (previous) (diff)

comment:181 in reply to: ↑ 180 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

If database_gap is a dependency of meataxe (it seems so from the commands above) then it should be in build/pkgs/meataxe/dependencies.

It isn't.

database_gap is a run-time dependency for the cohomology package (in the sense that it provides the SmallGroups library), but it has nothing to do with MeatAxe or with modular_resolution.

comment:182 in reply to: ↑ 181 ; follow-ups: Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

If database_gap is a dependency of meataxe (it seems so from the commands above) then it should be in build/pkgs/meataxe/dependencies.

It isn't.

database_gap is a run-time dependency for the cohomology package (in the sense that it provides the SmallGroups library), but it has nothing to do with MeatAxe or with modular_resolution.

run-time dependency is a dependency too.


OK, I see, you are saying that installing modular_decomposition should trigger installation of meataxe and database_gap (the format of dependencies there is weird, I am not sure these $(INST)/ are the right thing - most if not all packages don't have it`)

Last edited 2 years ago by dimpase (previous) (diff)

comment:183 in reply to: ↑ 182 Changed 2 years ago by SimonKing

Replying to dimpase:

database_gap is a run-time dependency for the cohomology package (in the sense that it provides the SmallGroups library), but it has nothing to do with MeatAxe or with modular_resolution.

run-time dependency is a dependency too.

I can only repeat myself. database_gap has NOTHING TO DO WHATSOEVER with MeatAxe.

So, if anything, we are talking here about a runtime dependency for what currently is (but soon will not be) OptionalExtension in sage/groups/modular_cohomology/. IIRC, I was told that the dependencies listed in OptionalExtension only concern build time dependencies (which is mtx and modres), but not runtime dependencies.

comment:184 follow-up: Changed 2 years ago by dimpase

files in src/sage/groups/modular_cohomology/ are not python3-ready (all these print blah instead of print(blah)), this causes havoc while attempting to run tests, it seems. E.g.

       print CohomologyRing(Integer(64),Integer(12))  # indirect doctest
                           ^
    SyntaxError: invalid syntax

comment:185 in reply to: ↑ 182 Changed 2 years ago by SimonKing

Replying to dimpase:

OK, I see, you are saying that installing modular_decomposition [ I guess you mean modular_resolution ] should trigger installation of meataxe and database_gap (the format of dependencies there is weird, I am not sure these $(INST)/ are the right thing - most if not all packages don't have it`)

No, an automatic installation (without explicit consent of the user) of database_gap is not allowed (licence issue) and not proposed.

Also, there are various ways to install the SmallGroups library. So, I think the correct way to deal with the problem is:

  • meataxe clearly has no dependency whatsoever.
  • modular_resolution will still have the dependency meataxe, but no other dependency. I just checked: There is no use of SmallGroup in the whole code.
  • take the current experimental OptionalExtension from sage/groups/modular_cohomology and put them into a pip installable upstream tarball modular_group_cohomology.
  • build/pkgs/modular_group_cohomology/dependencies is, as far as I understand, for build time dependencies, thus, it will only list meataxe and modular_resolution, but not database_gap.
  • build/pkgs/spkg-install will check that the SmallGroups library is in fact installed (it doesn't matter if it comes from database_gap or another source). If it isn't it will fail and advise the user to install it one way or another.
Last edited 2 years ago by SimonKing (previous) (diff)

comment:186 follow-up: Changed 2 years ago by dimpase

there are also some import errors, like

Running doctests with ID 2017-12-10-09-59-37-95694ca7.
Git branch: cohomolo
Using --optional=4ti2,cbc,ccache,cmake,database_gap,dot2tex,gambit,gap_packages,latte_int,libhomfly,libogg,lidia,lrslib,meataxe,modular_resolution,mpir,normaliz,pynormaliz,python2,qhull,sage,scons
Doctesting 1 file using 4 threads.
sage -t --warn-long 68.7 barcode.py
**********************************************************************
File "barcode.py", line 99, in sage.groups.modular_cohomology.barcode
Failed example:
    B158 = H158.bar_code('UpperCentralSeries')
Exception raised:
    Traceback (most recent call last):
      File "/home/dima/Sage/sage-dev/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 515, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/dima/Sage/sage-dev/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 885, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.groups.modular_cohomology.barcode[18]>", line 1, in <module>
        B158 = H158.bar_code('UpperCentralSeries')
      File "sage/groups/modular_cohomology/cohomology.pyx", line 1673, in sage.groups.modular_cohomology.cohomology.permanent_result.__call__ (build/cythonized/sage/groups/modular_cohomology/cohomology.c:28178)
        val = self._f(inst,*args,**kwds)
      File "sage/groups/modular_cohomology/cohomology.pyx", line 11034, in sage.groups.modular_cohomology.cohomology.COHO.bar_code (build/cythonized/sage/groups/modular_cohomology/cohomology.c:156393)
        OUT[i,j] = C[j].hom(GH[i,j],C[i]).poincare_of_image()
      File "sage/groups/modular_cohomology/cochain.pyx", line 7054, in sage.groups.modular_cohomology.cochain.ChMap.poincare_of_image (build/cythonized/sage/groups/modular_cohomology/cochain.c:76428)
        from sage.groups.modular_cohomology import singular
    ImportError: cannot import name singular

comment:187 in reply to: ↑ 184 Changed 2 years ago by SimonKing

Replying to dimpase:

files in src/sage/groups/modular_cohomology/ are not python3-ready (all these print blah instead of print(blah)), this causes havoc while attempting to run tests, it seems. E.g.

       print CohomologyRing(Integer(64),Integer(12))  # indirect doctest
                           ^
    SyntaxError: invalid syntax

OMG, I thought the old syntax is still fine in tests. Sure, I need to change all print statements in the tests, then.

comment:188 in reply to: ↑ 186 Changed 2 years ago by SimonKing

Replying to dimpase:

there are also some import errors, like

    ImportError: cannot import name singular

Thanks for finding this. I did do some computations in interactive sessions, but didn't run the test suite yet.

The point is that all the modules involved in the computation are supposed to use the same singular instance. And apparently I have changed something.

While we are at it: Is it OK to just import singular from sage.interfaces.singular? Or would you think it makes sense to provide a hook so that a user can plug in his/her own singular interface to the computation? I guess the former is easier and the latter doesn't add an advantage.

comment:189 Changed 2 years ago by SimonKing

I created #24359 to change MeatAxe into a dynamic library.

comment:190 Changed 2 years ago by SimonKing

  • Dependencies changed from #21437 to #24359

comment:191 Changed 2 years ago by SimonKing

After dealing with #24359, I'm back at the cohomology package.

There now remains to deal with two things:

  • C code providing a library libmodres and some binaries; it is autotoolized and links against libmtx.
  • Python and Cython code linked against libmtx and libmodres and is NOT standalone as it makes use of Sage types and Sage infrastructure.

How would you recommend to proceed?

  1. Have two separate packages modular_resolution (installing libmodres) and p_group_cohomology (pip-installing the Python/Cython? modules) that the user has to install both?
  2. Have a single package that both installs libmodres and the Python/Cython? code?
  3. Something else?

I somehow tend towards 2., since libmodres and the executables hardly are of any use without the Cython and Python code. But I can also see the advantage of a strict modularisation, as in 1.

comment:192 follow-up: Changed 2 years ago by SimonKing

Also, I need some pointers on how to create a pip installable module.

  • Somewhere I found the information that both setup.py and Manifest.in are required for pip installation. However, in some pip installable packages in SAGE_ROOT/upstream I do not find Manifest.in. So, what's the matter with it? If setup.py is enough, is there any advantage of pip install over python setup.py install?
  • In the old spkg, I was using distutils in setup.py. But somewhere I found the information that pip uninstall won't work with distutils. But I reckon that in future Sage will support uninstalling spkgs. So, should I use setuptools instead? Would that be able to build Cython code as well?

comment:193 follow-ups: Changed 2 years ago by SimonKing

The third topic for which I need pointers: Documentation.

When I was young, Sage's documentation used a script builder.py to build its documentation. In the first versions of the cohomology spkg, I copied and modified it. Now, I see that builder.py has gone from Sage.

The question is how I could most easily build a reference manual for my spkg from all the docstrings in Cython and Python.

  • Somewhere I found a blog post of Jeroen about "sphinx and Cython", advising to compile the Cython code with binding=True. Is that currently done in Sage? Wouldn't that involve a mild slow-down?
  • My impression is that using Sphinx with Cython is currently done in conf.py by replacing inspect.getargspec() by some custom version of getargspec() that works on Cython code. Would that already be enough? I could easily provide such a function, by creating a modified copy sage.misc.sageinspect or perhaps even by monkey-patching it.
  • Would it be worth-while to exploit Sage's doc build infrastructure? Say, by overriding SAGE_DOC and SAGE_DOC_SRC in sage_setup.docbuild.__init__? Does that have a chance to work?

comment:194 Changed 2 years ago by SimonKing

  • Dependencies changed from #24359 to #24359 #24468

comment:195 in reply to: ↑ 193 Changed 2 years ago by SimonKing

For the record:

Replying to SimonKing:

  • My impression is that using Sphinx with Cython is currently done in conf.py by replacing inspect.getargspec() by some custom version of getargspec() that works on Cython code. Would that already be enough? I could easily provide such a function, by creating a modified copy sage.misc.sageinspect or perhaps even by monkey-patching it.

This works. At least when using compiler_directives={'embedsignature': True} and then adding some helper function to doc/sources/conf.py that removes the signature from the docstring before passing it to sphinx.

comment:196 Changed 2 years ago by git

  • Commit changed from 12e1bc50e4ef6ceedaaf30e66e1fd03d5d1cadf6 to 5e011619bc4814aeb02f785e327fb7ecff8eda1f

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8d26842Upgrade meataxe spkg to SharedMeatAxe 1.0 (dynamic library)
13828d1Various fixes to spkg-install
1bc5815Use compatible implementation for MatrixSpace in mtx_unpickle
09ef3e0Merge branch 'u/jdemeyer/fix_comparison_of_matrices_that_live_in_equivalent_matrix_spaces' of git://trac.sagemath.org/sage into coho_rebase_base
2ac7e04More stuff in the meataxe interface, and a meataxe helper function
fbc779eExpose matrix_gfpn_dense.FieldConverter to the interface
17787a1Fix reading MeatAxe matrix from a file
5e01161Add p_group_cohomology-3.0 spkg

comment:197 Changed 2 years ago by SimonKing

  • Description modified (diff)
  • Status changed from needs_info to needs_review

comment:198 Changed 2 years ago by SimonKing

  • Description modified (diff)

comment:199 Changed 2 years ago by SimonKing

As usual, documentation is built, if SAGE_SPKG_INSTALL_DOCS=yes.

sage -i -c p_group_cohomology installs and tests the package. The tests are formed by a very short stand-alone test for David Greens programs and by a thorough test suite for the python/cython part of the package.

comment:200 Changed 2 years ago by git

  • Commit changed from 5e011619bc4814aeb02f785e327fb7ecff8eda1f to 5f6033d8b17a7915e55733a9161b58a0d00c0048

Branch pushed to git repo; I updated commit sha1. New commits:

5f6033dRemove old cohomology doc before installing the new doc

comment:201 in reply to: ↑ 142 ; follow-up: Changed 2 years ago by SimonKing

Replying to SimonKing:

There are good news:

I have just finished unit_test_64, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.

The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.

There are even better news.

On the same machine(!) it now only takes 7:18.12 min walltime and 6:32.00 CPU-time. So, it is a very clear improvement compared with previous package version. For reference: The very first computation of these cohomology rings by the Magma programs of Jon Carlson took several months.

I don't know where the improvement is coming from. Can the change from a static meataxe library to a dynamic meataxe library explain the improvement?

comment:202 in reply to: ↑ 192 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

  • Somewhere I found the information that both setup.py and Manifest.in are required for pip installation. However, in some pip installable packages in SAGE_ROOT/upstream I do not find Manifest.in. So, what's the matter with it?

MANIFEST.in has nothing to do with pip. It is needed to create a source package from the source tree. With "source tree" I mean what git checkout gives you. That is not the same as a source package (a tarball created by make sdist in Python or make dist with autotools). The difference may not be so important for Python, so simple packages can get away with ignoring this difference. Typically, a source tarball would contain certain auto-generated files that the source tree does not.

  • In the old spkg, I was using distutils in setup.py. But somewhere I found the information that pip uninstall won't work with distutils.

That is partially correct. To be more precise: a project installed using distutils (i.e. with ./setup.py install) cannot be uninstalled with pip. However, a project installed with pip (using pip install ...) can be uninstalled with pip, even if the setup.py script uses distutils.

I know, it's confusing :-) You should remember that pip is just a tool on top of either distutils or setuptools, it doesn't replace them. And many features of pip come from pip itself, not from the underlying distutils or setuptools.

So, should I use setuptools instead? Would that be able to build Cython code as well?

Personally, I would say: keep using distutils if that works for you. I'm sure that other people will disagree and say that you really should use setuptools.

comment:203 in reply to: ↑ 193 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

The question is how I could most easily build a reference manual for my spkg from all the docstrings in Cython and Python.

  • Somewhere I found a blog post of Jeroen about "sphinx and Cython", advising to compile the Cython code with binding=True.

Yes, that should work "out of the box".

Is that currently done in Sage?

No. See #22747 for that.

Wouldn't that involve a mild slow-down?

Yes, it does involve a significant (in the statistics sense) slowdown, but only at the Python -> Cython boundary. If your package is all Cython, then there should be no slowdown within your package. So that leaves the boundary between user code and your package. Do you expect many calls to your package in tight Python loops? If not, then I would go for the binding=True solution.

comment:204 in reply to: ↑ 202 Changed 2 years ago by SimonKing

Thank you for your explanations!

Replying to jdemeyer:

Personally, I would say: keep using distutils if that works for you. I'm sure that other people will disagree and say that you really should use setuptools.

The new package's setup.py does all imports from setuptools, whereas the old package's setup.py did the imports from distutils. So, that was a trivial change.

comment:205 in reply to: ↑ 203 ; follow-up: Changed 2 years ago by SimonKing

Replying to jdemeyer:

Wouldn't that involve a mild slow-down?

Yes, it does involve a significant (in the statistics sense) slowdown, but only at the Python -> Cython boundary. If your package is all Cython, then there should be no slowdown within your package. So that leaves the boundary between user code and your package. Do you expect many calls to your package in tight Python loops? If not, then I would go for the binding=True solution.

I do care for speed, but I think all tight loops occur in Cython, not Python.

In any case, the only real problem for building the docs is to determine the argspec of methods and functions. The new package solves it with compiler_directives={'embedsignature': True} and some helper functions that you can find in pyxsources/doc/sources/conf.py

comment:206 in reply to: ↑ 201 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

Can the change from a static meataxe library to a dynamic meataxe library explain the improvement?

All other things being equal, a dynamic library has a little bit of overhead to deal with the dynamic linking, so it should rather be slower instead of faster.

So there must be another explanation. Are you using different compiler flags maybe? Are there certain computations in meataxe which are cached and which might be re-used?

comment:207 in reply to: ↑ 205 Changed 2 years ago by jdemeyer

Replying to SimonKing:

I do care for speed, but I think all tight loops occur in Cython, not Python.

Then I think that binding=True is the way to go because it uses standard tools without hacks like a custom getargspec.

In any case, you could use binding=True and do the timings again to check.

Last edited 2 years ago by jdemeyer (previous) (diff)

comment:208 in reply to: ↑ 206 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Can the change from a static meataxe library to a dynamic meataxe library explain the improvement?

All other things being equal, a dynamic library has a little bit of overhead to deal with the dynamic linking, so it should rather be slower instead of faster.

Yes, that's why I am surprised.

So there must be another explanation. Are you using different compiler flags maybe? Are there certain computations in meataxe which are cached and which might be re-used?

comment:142 was with the then-current meataxe spkg, i.e., static, with patches, using a custom Makefile. If I remember correctly, it did use some -O flag, but I don't remember which one (-O2?). The difference with respect to the current meataxe is (1) it is now dynamic library, and (2) autotools is used. The current compiler flags for meataxe are chosen by autotools.

comment:209 Changed 2 years ago by jdemeyer

autotools defaults to -g -O2 for CFLAGS.

comment:210 follow-up: Changed 2 years ago by jdemeyer

The old Sage meataxe spkg used -O (which means -O1) as optimization flag. So the change of flag might explain the increase in performance that you saw.

comment:211 in reply to: ↑ 210 ; follow-up: Changed 2 years ago by SimonKing

Replying to jdemeyer:

The old Sage meataxe spkg used -O (which means -O1) as optimization flag. So the change of flag might explain the increase in performance that you saw.

OK, if -O was hardcoded in the Makefile shipped with upstream MeatAxe 2.4.24, then the flags chosen by autotools in SharedMeatAxe 1.0 could have improved things.

By the way, I didn't try the binding=True compiler flag yet. Do you think that not having it and instead using a helper function for building the docs is a problem?

comment:212 in reply to: ↑ 211 ; follow-ups: Changed 2 years ago by jdemeyer

Replying to SimonKing:

Do you think that not having it and instead using a helper function for building the docs is a problem?

If it works, then it's not a problem. I do think that it's more fragile.

comment:213 in reply to: ↑ 212 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Do you think that not having it and instead using a helper function for building the docs is a problem?

If it works, then it's not a problem. I do think that it's more fragile.

As you can see here, building the docs does work, i.e., the signatures of the methods of cdef classes are correctly given, and also the signature of methods that are wrapped with @tempory_result or @permanent_result.

In any case, I will soon(ish) test what happens with binding:True. If the performance loss is acceptable, then p_group_cohomology-3.1 will use it.

Last edited 2 years ago by SimonKing (previous) (diff)

comment:214 in reply to: ↑ 212 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Do you think that not having it and instead using a helper function for building the docs is a problem?

If it works, then it's not a problem. I do think that it's more fragile.

Remark: The process_* functions in src/doc/source/conf.py are only mild modifications of the corresponding functions in Sage's documentation's conf.py.

Anyway, I now try to do without it.

comment:215 Changed 2 years ago by SimonKing

Is it advised to replace "old" string formatting such as "%s bar"%foo with "new" string formatting such as "{} bar".format(foo)? If it is: Is there a tool that converts the first syntax into the second? I wouldn't like to change it manually, as there are hundreds of occurrences in the code.

comment:216 follow-up: Changed 2 years ago by SimonKing

Jeroen, I just tested binding=True.

First problem: The signature of methods of extension classes are not correctly determined. For example, I get

sage: from inspect import getargspec
sage: from pGroupCohomology.cochain import COCH    
sage: getargspec(COCH.setname)
ArgSpec(args=['self', 's', 'is_polyrep'], varargs=None, keywords=None, defaults=None)

but the method is

    def setname(self, s, is_polyrep=False):

and therefore the html documentation would not be built correctly.

Second problem: Performance.

sage: from pGroupCohomology.factory import unit_test_64
sage: L,t = unit_test_64()
...
#267: Walltime   9:24.40 min
      CPU-time   8:31.86 min
      Singular   0:48.71 min
      Gap-time   0:19.50 min

Here is what I get with the current version of the package (i.e., with embedsignature=True and more elaborate helper functions):

sage: from sage.misc.sageinspect import sage_getargspec
sage: from pGroupCohomology.cochain import COCH
sage: sage_getargspec(COCH.setname)
ArgSpec(args=['self', 's', 'is_polyrep'], varargs=None, keywords=None, defaults=(False,))

Here, inspect.getargspec won't work at all, but sage_getargspec gives a correct answer, which can also be seen in the published package documentation.

And concerning performance:

sage: from pGroupCohomology.factory import unit_test_64
sage: L,t = unit_test_64()
...
#267: Walltime   7:50.20 min
      CPU-time   6:59.89 min
      Singular   0:46.04 min
      Gap-time   0:20.58 min

Conclusion

The loss in performance is quite noticeable, and moreover, binding=True doesn't help with the documentation. Therefore, I won't use it.

comment:217 follow-up: Changed 2 years ago by jdemeyer

In recent versions of Python and Cython, there is the even newer syntax f"{foo} bar". Unless you specifically want to support older versions, I would recommend that.

comment:218 in reply to: ↑ 217 ; follow-up: Changed 2 years ago by SimonKing

Replying to jdemeyer:

In recent versions of Python and Cython, there is the even newer syntax f"{foo} bar". Unless you specifically want to support older versions, I would recommend that.

Is that in Sage already? If it is, I would certainly use that syntax in future. But the question is: Can I upgrade the syntax that I used in the past, without manually editing hundreds of lines of code?

comment:219 in reply to: ↑ 216 ; follow-ups: Changed 2 years ago by jdemeyer

Replying to SimonKing:

and therefore the html documentation would not be built correctly.

Did you correctly apply all steps from http://opendreamkit.org/2017/06/09/CythonSphinx, in particular the patching of inspect.isfunction?

comment:220 in reply to: ↑ 218 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

Replying to jdemeyer:

In recent versions of Python and Cython, there is the even newer syntax f"{foo} bar". Unless you specifically want to support older versions, I would recommend that.

Is that in Sage already?

Yes.

But the question is: Can I upgrade the syntax that I used in the past, without manually editing hundreds of lines of code?

No, but there is also no real reason to do that. The old syntax still works and it's not even deprecated.

comment:221 in reply to: ↑ 219 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

and therefore the html documentation would not be built correctly.

Did you correctly apply all steps from http://opendreamkit.org/2017/06/09/CythonSphinx, in particular the patching of inspect.isfunction?

No. I only took your comment:203 that it "would work out of the box"...

comment:222 in reply to: ↑ 220 ; follow-up: Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Replying to jdemeyer:

In recent versions of Python and Cython, there is the even newer syntax f"{foo} bar". Unless you specifically want to support older versions, I would recommend that.

Is that in Sage already?

Yes.

No, it isn't!

sage: version()
'SageMath version 8.2.beta2, Release Date: 2018-01-01'
sage: s = "foo"
sage: f"{s} bar"
  File "<ipython-input-17-b4ba52efd1d0>", line 1
    f"{s} bar"
             ^
SyntaxError: invalid syntax

comment:223 in reply to: ↑ 219 ; follow-up: Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

and therefore the html documentation would not be built correctly.

Did you correctly apply all steps from http://opendreamkit.org/2017/06/09/CythonSphinx, in particular the patching of inspect.isfunction?

Now I am puzzled. The monkey-patch you suggested is:

     def isfunction(obj):
         return hasattr(type(obj), "__code__")

     import inspect
     inspect.isfunction = isfunction

However, when you inspect `inspect.isfunction, you'll find

Signature: inspect.isfunction(obj)
Source:   
def isfunction(obj):
    """
    ...
    """
    # We use type(obj) instead of just obj to avoid __getattr__().
    # Some types, like methods, will return the __code__ of the
    # underlying function in __getattr__() but we don't want to
    # detect those as functions.
    return hasattr(type(obj), "__code__")

So, the patch seems pointless.

comment:224 in reply to: ↑ 222 Changed 2 years ago by jdemeyer

Replying to SimonKing:

No, it isn't!

Sorry, I meant that the Cython-in-Sage supports it. The Python 2 language does not support it.

comment:225 in reply to: ↑ 223 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

So, the patch seems pointless.

That is because the monkey-patch is already applied in Sage.

But I just realized that you are right: even with binding=True, default values are not displayed in the docs. So you found an actual bug!

comment:226 in reply to: ↑ 225 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

So, the patch seems pointless.

That is because the monkey-patch is already applied in Sage.

I see.

But I just realized that you are right: even with binding=True, default values are not displayed in the docs. So you found an actual bug!

Hooray!!

comment:227 follow-up: Changed 2 years ago by SimonKing

The binding=True option led to 20% performance loss, that's why I wouldn't use it even if the bug with default options was fixed.

But here's something for p_group_cohomology-3.1: Replace the pexpect interfaces to gap and singular by calls to libgap and libsingular. After all, meanwhile all Singular data structures relevant for cohomology computation (namely: commutative and super-commutative algebras with degree weights and arbitrary matrix ordering) are wrapped in Sage. But the transition would be a lot of work, that's why I wouldn't do it in version 3.0.

comment:228 in reply to: ↑ 227 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

The binding=True option led to 20% performance loss, that's why I wouldn't use it even if the bug with default options was fixed.

Yes, I understood that. And this is not easy to fix, it would require changes to Python itself.

comment:229 Changed 23 months ago by SimonKing

For reference: In order to achieve global dominance (i.e., to make the future package version p_group_cohomology-3.1 a standard package of Sage), the following would be needed in preparation:

  1. Make SharedMeatAxe standard. I think it should be standard anyway, as it offers matrix arithmetic over small non-prime fields of odd characteristics that is vastly superior compared with the currently used Matrix_generic_dense.
  2. Remove the dependency on the SmallGroups library. In fact, the current cohomology package is able to create the cohomology ring of a group as long as it can be expressed as a permutation group in GAP; one doesn't necessarily need to know the SmallGroups address of it.

Hence, I could re-write the relevant parts of the code so that IdGroup() (a function to determine the SmallGroups address) is used when available for efficiency, and to work around it (e.g., create an elementary abelian group without referring to SmallGroup(p^n, NumberSmallGroups(p^n)-1)) otherwise.

In that way, the cohomology package would only depend on standard packages, and thus the Cython parts of it could be moved into the Sage src tree.

However, it would be good if p_group_cohomology-3.0 (which this ticket is about) could be reviewed first.

comment:230 follow-up: Changed 22 months ago by SimonKing

  • Status changed from needs_review to needs_work
  • Work issues set to Cope with backwards incompatible changes - again.

It is a pity that the spkg hasn't been reviewed yet --- because once again, a backward incompatibly change in SageMath broke the spkg!

Reason this time: _add_ of ModuleElement has changed from cdef (which it still is for Element!) to cpdef.

I must say that those things have happened far too often, and I find it quite annoying. That's one of the reasons why I will try to modify the spkg so that it will not depend on the SmallGroups library, with the aim to make group cohomology a standard package with cython code in the SageMath source tree. In that way, the patchbots would immediately notice if some ticket makes it necessary to change my code.

comment:231 Changed 22 months ago by dimpase

GAP 4.9 (a beta released 1 Feb) has SmallGroups licensed under Artistic License 2.0, which is GPL-compatible. So we should just make database_gap standard, and you should not try to work around this non-problem...

Last edited 22 months ago by dimpase (previous) (diff)

comment:232 in reply to: ↑ 230 ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

Reason this time: _add_ of ModuleElement has changed from cdef (which it still is for Element!) to cpdef.

It was always meant to be used as cpdef _add_ (that is what is documented in element.pyx). So in this case, I would argue that your old code was wrong and should have been fixed anyway.

comment:233 Changed 22 months ago by jdemeyer

  • Dependencies #24359 #24468 deleted

comment:234 follow-up: Changed 22 months ago by jdemeyer

This ticket makes some seemingly unrelated changes to src/sage/libs/meataxe and src/sage/matrix. Is that intentional?

Last edited 22 months ago by jdemeyer (previous) (diff)

comment:235 in reply to: ↑ description ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

  • The spkg-install script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.

If it's a dependency, it should be declared as dependency in build/pkgs/p_group_cohomology/dependencies.

comment:236 in reply to: ↑ 234 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

This ticket makes some changes to src/sage/libs/meataxe. Is that intentional?

The Cython code in the package links against libmtx, and I think it is cleanest to have the relevant pxd-header file in a single central place (src/sage/libs), rather than making a copy of it. About the changes: I need a bit more stuff from meataxe than what sage.matrix.matrix_gfpn_dense needs, therefore the changes.

comment:237 follow-up: Changed 22 months ago by jdemeyer

Is there a public website or repository for this package? If so, it should be put in SPKG.txt. And I would remove the version number from the first line of that file, you'll just forget to update it anyway.

comment:238 in reply to: ↑ 235 Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

  • The spkg-install script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.

If it's a dependency, it should be declared as dependency in build/pkgs/p_group_cohomology/dependencies.

No. I think that was meanwhile discussed enough often.

comment:239 in reply to: ↑ 236 ; follow-up: Changed 22 months ago by jdemeyer

Don't be afraid of Unicode: you can put names like Sikirić and Poincaré in SPKG.txt

Last edited 22 months ago by jdemeyer (previous) (diff)

comment:240 in reply to: ↑ 239 Changed 22 months ago by SimonKing

Replying to jdemeyer:

Don't be afraid of Unicode: you can put names like Sikirić

Ouch, and thank you for spotting that I was misspelling his name as "Sikitic" (with t instead of r)!

comment:241 in reply to: ↑ 237 Changed 22 months ago by SimonKing

Replying to jdemeyer:

Is there a public website or repository for this package? If so, it should be put in SPKG.txt.

I thought the pointer to the package's public documentation would be enough. But of course I can also add information where to download (it is basically the same address).

And I would remove the version number from the first line of that file, you'll just forget to update it anyway.

Good point...

comment:242 in reply to: ↑ 232 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Reason this time: _add_ of ModuleElement has changed from cdef (which it still is for Element!) to cpdef.

It was always meant to be used as cpdef _add_ (that is what is documented in element.pyx). So in this case, I would argue that your old code was wrong and should have been fixed anyway.

In any case, I should cope with that API change. Question: Should it give rise to a new package version 3.0.1, or can it remain 3.0? From my point of view, publication of the package is only finished as soon as it became an optional package in SageMath, hence, I consider the current tar ball not as a "published version".

comment:243 in reply to: ↑ 242 Changed 22 months ago by jdemeyer

Replying to SimonKing:

Question: Should it give rise to a new package version 3.0.1, or can it remain 3.0? From my point of view, publication of the package is only finished as soon as it became an optional package in SageMath, hence, I consider the current tar ball not as a "published version".

I personally don't care much...

comment:244 Changed 22 months ago by git

  • Commit changed from 5f6033d8b17a7915e55733a9161b58a0d00c0048 to ccae2cc49ae941069b96067f9bce45355975da25

Branch pushed to git repo; I updated commit sha1. New commits:

6ab0fe9Merge branch 't/18514/upgrade_of_group_cohomology_spkg' into p_group_cohomology-3.1
ccae2ccp_group_cohomology-3.0: Cope with some API change, improve SPKG.txt

comment:245 Changed 22 months ago by git

  • Commit changed from ccae2cc49ae941069b96067f9bce45355975da25 to 4d696e0a19c1edb4832cc5804be9c92eb21dcaf0

Branch pushed to git repo; I updated commit sha1. New commits:

4d696e0p_group_cohomology-3.0: Fix checksum

comment:246 follow-up: Changed 22 months ago by SimonKing

  • Status changed from needs_work to needs_review

I have modified cochain.pyx so that it builds with the current pxd header in sage.structure.element. The SPKG.txt is modified as requested. The checksum is updated. The tarball is updated as well (still at the location stated in the ticket description), still with version number 3.0. And I just checked that the package still installs.

Concerning the dependencies, I think it was consented at some point that the dependencies files list build time dependencies, so that all packages can be built in a correct order. spkg-install does test whether the SmallGroups library is available and prints a warning if it isn't, but this won't prevent the package from being installable.

Needs review, then...

comment:247 in reply to: ↑ 246 ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

Concerning the dependencies, I think it was consented at some point that the dependencies files list build time dependencies, so that all packages can be built in a correct order.

Well, that advice is meant for dependencies on standard packages. Standard packages are built anyway, so at runtime you can count on the fact that they will be present. This is not true for optional packages, so runtime dependencies should be put in dependencies too (as order-only dependency).

comment:248 follow-up: Changed 22 months ago by jdemeyer

[p_group_cohomology-3.0] Installing p_group_cohomology-3.0
[p_group_cohomology-3.0]
[p_group_cohomology-3.0] Error compiling Cython file:
[p_group_cohomology-3.0] ------------------------------------------------------------
[p_group_cohomology-3.0] ...   
[p_group_cohomology-3.0]             if (len(L)!=R.Data.projrank[n]):
[p_group_cohomology-3.0]                 raise IndexError("Last parameter must be of length %d"%(R.Data.projrank[n]))
[p_group_cohomology-3.0]             self.Resl = R
[p_group_cohomology-3.0]             self.Deg = n
[p_group_cohomology-3.0]             self.Name = str(Nick)
[p_group_cohomology-3.0]             self.Data = makeMTX(rawMatrix(R.G_Alg.Data.p, [L]))
[p_group_cohomology-3.0]                                ^
[p_group_cohomology-3.0] ------------------------------------------------------------
[p_group_cohomology-3.0]
[p_group_cohomology-3.0] pGroupCohomology/cochain.pyx:491:32: undeclared name not builtin: rawMatrix
[p_group_cohomology-3.0]
[p_group_cohomology-3.0] Error compiling Cython file:
[p_group_cohomology-3.0] ------------------------------------------------------------
[p_group_cohomology-3.0] ...   
[p_group_cohomology-3.0]             if (len(L)!=R.Data.projrank[n]):
[p_group_cohomology-3.0]                 raise IndexError("Last parameter must be of length %d"%(R.Data.projrank[n]))
[p_group_cohomology-3.0]             self.Resl = R
[p_group_cohomology-3.0]             self.Deg = n
[p_group_cohomology-3.0]             self.Name = str(Nick)
[p_group_cohomology-3.0]             self.Data = makeMTX(rawMatrix(R.G_Alg.Data.p, [L]))
[p_group_cohomology-3.0]                                         ^
[p_group_cohomology-3.0] ------------------------------------------------------------
[p_group_cohomology-3.0]
[p_group_cohomology-3.0] pGroupCohomology/cochain.pyx:491:41: Cannot convert Python object to 'Matrix_t *'
[p_group_cohomology-3.0] Traceback (most recent call last):
[p_group_cohomology-3.0]   File "setup.py", line 96, in <module>
[p_group_cohomology-3.0]     ext_modules=cythonize(ext_mods, compiler_directives={'embedsignature': True}),
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1039, in cythonize
[p_group_cohomology-3.0]     cythonize_one(*args)
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1161, in cythonize_one
[p_group_cohomology-3.0]     raise CompileError(None, pyx_file)
[p_group_cohomology-3.0] Cython.Compiler.Errors.CompileError: pGroupCohomology/cochain.pyx
[p_group_cohomology-3.0] Error: could not determine package name
[p_group_cohomology-3.0] ********************************************************************************
[p_group_cohomology-3.0] Error installing p_group_cohomology-3.0
[p_group_cohomology-3.0] ********************************************************************************

I guess you are missing a dependency on the Sage library. But to be completely honest, I don't really know how to declare that dependency. This is the first package like this.

comment:249 Changed 22 months ago by jdemeyer

  • Status changed from needs_review to needs_work

After first building the Sage library, it still doesn't work:

[p_group_cohomology-3.0]     building 'pGroupCohomology.resolution' extension
[p_group_cohomology-3.0]     creating build/temp.linux-ppc64le-2.7
[p_group_cohomology-3.0]     creating build/temp.linux-ppc64le-2.7/pGroupCohomology
[p_group_cohomology-3.0]     gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/cpython -I/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/libs/ntl -I/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/cysignals -I/home/jdemeyer/sage-test/local/include -I/home/jdemeyer/sage-test/local/include/python2.7 -I/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/numpy/core/include -I/home/jdemeyer/sage-test/local/lib/python2.7/site-packages -I/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/ext -I/home/jdemeyer/sage-test/local/include/python2.7 -c pGroupCohomology/resolution.c -o build/temp.linux-ppc64le-2.7/pGroupCohomology/resolution.o
[p_group_cohomology-3.0]     pGroupCohomology/resolution.c:564:10: fatal error: modular_resolution/pgroup.h: No such file or directory
[p_group_cohomology-3.0]      #include "modular_resolution/pgroup.h"
[p_group_cohomology-3.0]               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[p_group_cohomology-3.0]     compilation terminated.
[p_group_cohomology-3.0]     error: command 'gcc' failed with exit status 1
[p_group_cohomology-3.0]     Running setup.py install for pGroupCohomology: finished with status 'error'

comment:250 follow-up: Changed 22 months ago by jdemeyer

And the smallgroups warning is broken too:

./spkg-install: line 33: [: too many arguments

comment:251 Changed 22 months ago by jdemeyer

The modular_resolution installation got broken by #24106.

comment:252 follow-up: Changed 22 months ago by jdemeyer

A solution for #24106 would be to use plain $MAKE install instead of sdh_make_install.

comment:253 in reply to: ↑ 248 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

[p_group_cohomology-3.0] Installing p_group_cohomology-3.0
[p_group_cohomology-3.0]
[p_group_cohomology-3.0] Error compiling Cython file:
[p_group_cohomology-3.0] ------------------------------------------------------------
[p_group_cohomology-3.0] ...   
[p_group_cohomology-3.0]             if (len(L)!=R.Data.projrank[n]):
[p_group_cohomology-3.0]                 raise IndexError("Last parameter must be of length %d"%(R.Data.projrank[n]))
[p_group_cohomology-3.0]             self.Resl = R
[p_group_cohomology-3.0]             self.Deg = n
[p_group_cohomology-3.0]             self.Name = str(Nick)
[p_group_cohomology-3.0]             self.Data = makeMTX(rawMatrix(R.G_Alg.Data.p, [L]))
[p_group_cohomology-3.0]                                ^
[p_group_cohomology-3.0] ------------------------------------------------------------
[p_group_cohomology-3.0]
[p_group_cohomology-3.0] pGroupCohomology/cochain.pyx:491:32: undeclared name not builtin: rawMatrix

It seems that you didn't use the branch from this ticket. That's exactly one of the functions that is added to sage.libs.meataxe, because logically it belongs there.

comment:254 in reply to: ↑ 253 ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

It seems that you didn't use the branch from this ticket.

You didn't understand my comment. I did use this branch, it's just that the Sage library was not rebuilt. But since I don't see an easy fix, let's ignore this issue for now.

comment:255 in reply to: ↑ 252 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

A solution for #24106 would be to use plain $MAKE install instead of sdh_make_install.

Now seriously??? I just got used to using yet another Sage build tool. So then, why not use plain pip install, too?

Also, what seems to be the problem? modular_resolution is an autotoolized package, and I'd rather say that if sdh_make_install doesn't work on it then the fix should be in Sage, not in the package.

But since I don't intend to wait for yet another ticket (namely one for fixing sdh_make_install), I'll switch back to using $MAKE and $MAKE install.

comment:256 in reply to: ↑ 254 ; follow-ups: Changed 22 months ago by SimonKing

Replying to jdemeyer:

You didn't understand my comment. I did use this branch, it's just that the Sage library was not rebuilt. But since I don't see an easy fix, let's ignore this issue for now.

I don't think that there is an issue here. As soon as the branch from this ticket is merged, it will of course be built before installing the group cohomology package. So, what happened to you will not happen to the users.

comment:257 in reply to: ↑ 256 Changed 22 months ago by jdemeyer

Replying to SimonKing:

I don't think that there is an issue here. As soon as the branch from this ticket is merged, it will of course be built before installing the group cohomology package. So, what happened to you will not happen to the users.

+1

comment:258 in reply to: ↑ 255 Changed 22 months ago by jdemeyer

Replying to SimonKing:

Also, what seems to be the problem?

The problem is that Sage is trying to separate the build and install steps of packages. This is in general a good thing to do. But in your package, the build of p_group_cohomology relies on the installation of modular_resolution.

I'll switch back to using $MAKE and $MAKE install.

Only $MAKE install should be affected.

comment:259 in reply to: ↑ 250 Changed 22 months ago by SimonKing

Replying to jdemeyer:

And the smallgroups warning is broken too:

./spkg-install: line 33: [: too many arguments

Line 33? That's puzzling:

$ wc -l build/pkgs/p_group_cohomology/spkg-install 
27 build/pkgs/p_group_cohomology/spkg-install

Anyway, I guess the problem arises in these lines:

if [ `echo "SmallGroup(13,1); quit;" | gap -b -T | grep "13"` = "" ]; then
   echo "=================================================================="
   echo "| Warning: It seems that GAP's SmallGroups library is missing.   |"
   echo "| It is a runtime dependency for the p_group_cohomology package. |"
   echo "| One way to install it is as part of the database_gap package.  |"
   echo "=================================================================="
fi

I was once told that this is how to test whether SmallGroups is installed. But now I see that executing the test (in a Sage shell and without SmallGroups installed) gives this:

(sage-sh) king@king-C70-B:sage$ echo "SmallGroup(13,1); quit;" | gap -b -T | grep "13"
Error, the Small Groups library is required but not installed

Not ideal.

Two potential ways out:

  1. Put database_gap into the dependencies, which it seems is what you want me to do anyway. This would ensure that SmallGroups is installed. But technically my package does not depend on the full database-gap but on SmallGroups only, which theoretically could be installed in different ways. By consequence, I would also need to adapt some parts of the documentation and I guess of SPKG.txt.
  2. Find a different way of testing whether SmallGroups is installed or not.

comment:260 in reply to: ↑ 256 ; follow-up: Changed 22 months ago by SimonKing

Replying to SimonKing:

Replying to jdemeyer:

You didn't understand my comment. I did use this branch, it's just that the Sage library was not rebuilt. But since I don't see an easy fix, let's ignore this issue for now.

I don't think that there is an issue here. As soon as the branch from this ticket is merged, it will of course be built before installing the group cohomology package. So, what happened to you will not happen to the users.

OTOH: Is it possible and does it make sense to put sagelib into the dependencies of the package? After all, it does link against stuff from the sage library.

comment:261 in reply to: ↑ 247 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Concerning the dependencies, I think it was consented at some point that the dependencies files list build time dependencies, so that all packages can be built in a correct order.

Well, that advice is meant for dependencies on standard packages. Standard packages are built anyway, so at runtime you can count on the fact that they will be present. This is not true for optional packages, so runtime dependencies should be put in dependencies too (as order-only dependency).

PS: See your comment:93

comment:262 in reply to: ↑ 260 Changed 22 months ago by jdemeyer

Replying to SimonKing:

OTOH: Is it possible to put sagelib into the dependencies of the package?

I don't think so.

comment:263 in reply to: ↑ 261 Changed 22 months ago by jdemeyer

Replying to SimonKing:

PS: See your comment:93

That comment is from the time when you had a very different packaging in mind, so it's not relevant now. That was talking about the Cython module, not the package.

Last edited 22 months ago by jdemeyer (previous) (diff)

comment:264 Changed 22 months ago by git

  • Commit changed from 4d696e0a19c1edb4832cc5804be9c92eb21dcaf0 to a4fa76230828ae8b3e771f968f0665a2048c06eb

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a4fa762p_group_cohomology-3.0: Fix checksum, dependencies and SPKG.txt

comment:265 Changed 22 months ago by git

  • Commit changed from a4fa76230828ae8b3e771f968f0665a2048c06eb to 98e6c213914bfe601a54e66640c5425da03bc161

Branch pushed to git repo; I updated commit sha1. New commits:

98e6c21p_group_cohomology-3.0: Remove useless test in spkg-install

comment:266 follow-up: Changed 22 months ago by SimonKing

  • Status changed from needs_work to needs_review
  • Work issues Cope with backwards incompatible changes - again. deleted

I think it is now again ready for review. Summary of what I did today:

  • Cope with the changes in src/sage/structure/element.pxd
  • Add database_gap as a dependency. I am not totally happy about it, but as it seems the dependency will soon be GPL compatible.
  • Remove the test for SmallGroup from spkg-install, as it is now tested by the dependency list.
  • Change accents (Sikirić, Poincaré) both in SPKG.txt and in the index.rst file of the documentation (in the docstrings, I did use accents already).
  • Fix the information on where it find the local documentation (it used to tell that it can be found in SAGE_LOCAL/share/p_group_cohomology/html/, but it is SAGE_LOCAL/share/p_group_cohomology/.
  • Fix the checksum and update both the documentation and the tarball on the project's web pages.
Last edited 22 months ago by SimonKing (previous) (diff)

comment:267 Changed 22 months ago by jdemeyer

  • Milestone changed from sage-6.8 to sage-8.2
  • Status changed from needs_review to needs_work
[p_group_cohomology-3.0] Invalid checksum for cached file /home/jdemeyer/sage-test/upstream/p_group_cohomology-3.0.tar.gz, deleting

comment:268 follow-ups: Changed 22 months ago by jdemeyer

The Python package is packaged badly. It contains a left-over build directory. Really, the proper way to package this is to use python setup.py sdist for the Python part and make dist for the autotools part.

Also, given that the package is rather large, you could consider using .xz compression. Note that xz is a standard package in Sage, just add | xz to the dependencies.

comment:269 in reply to: ↑ 266 ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

  • Add database_gap as a dependency.

Again a detail, but it would be better to have this as order-only dependency (| database_gap)

comment:270 in reply to: ↑ 268 Changed 22 months ago by SimonKing

Replying to jdemeyer:

The Python package is packaged badly. It contains a left-over build directory. Really, the proper way to package this is to use python setup.py sdist for the Python part and make dist for the autotools part.

OK, good point. For the autotools part, I did already. But for the Python part I didn't.

Also, given that the package is rather large, you could consider using .xz compression. Note that xz is a standard package in Sage, just add | xz to the dependencies.

OK, as soon as I read the xz manual, I'll try it.

comment:271 in reply to: ↑ 269 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

  • Add database_gap as a dependency.

Again a detail, but it would be better to have this as order-only dependency (| database_gap)

What does that mean? Are you saying that the dependencies file should contain this:

meataxe | database_gap

?

comment:272 in reply to: ↑ 271 Changed 22 months ago by jdemeyer

Replying to SimonKing:

Replying to jdemeyer:

Replying to SimonKing:

  • Add database_gap as a dependency.

Again a detail, but it would be better to have this as order-only dependency (| database_gap)

What does that mean? Are you saying that the dependencies file should contain this:

meataxe | database_gap

?

Well really

meataxe | database_gap xz

comment:273 in reply to: ↑ 268 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

The Python package is packaged badly. It contains a left-over build directory. Really, the proper way to package this is to use python setup.py sdist for the Python part and make dist for the autotools part.

Now I recall why I didn't use python setup.py sdist --- because actually I tried it before.

python setup.py sdist creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.

comment:274 in reply to: ↑ 273 ; follow-ups: Changed 22 months ago by jdemeyer

Replying to SimonKing:

python setup.py sdist creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.

Well, you need to add those file to MANIFEST.in. See for example https://github.com/defeo/cypari2/blob/master/MANIFEST.in

comment:275 in reply to: ↑ 274 Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

python setup.py sdist creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.

Well, you need to add those file to MANIFEST.in. See for example https://github.com/defeo/cypari2/blob/master/MANIFEST.in

... and exclude the .c files, right?

comment:276 in reply to: ↑ 274 Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

python setup.py sdist creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.

Well, you need to add those file to MANIFEST.in. See for example https://github.com/defeo/cypari2/blob/master/MANIFEST.in

I put Manifest.in in the same folder as setup.py, and it contains this:

include COPYING 
global-include pGroupCohomology/*.py pGroupCohomology/*.pyx pGroupCohomology/*.pxd pGroupCohomology/*.pxi

global-exclude pGroupCohomology/*.c

prune dist

Before that, I tried to be less specific and only wrote *.py instead of pGroupCohomology/*.py. However, the effect is that all .pxd files and all .c files are INCLUDED and all .py and .pyx files are missing.

So, what did I do wrong?

Last edited 22 months ago by SimonKing (previous) (diff)

comment:277 Changed 22 months ago by SimonKing

PS:

include pGroupCohomology/*.py pGroupCohomology/*.pyx pGroupCohomology/*.pxd pGroupCohomology/*.pxi

exclude pGroupCohomology/*.c

prune dist

didn't work either.

comment:278 follow-up: Changed 22 months ago by jdemeyer

I think that global-include is meant to specify files, not paths.

See https://docs.python.org/2/distutils/sourcedist.html#commands

comment:279 in reply to: ↑ 278 Changed 22 months ago by SimonKing

Replying to jdemeyer:

See https://docs.python.org/2/distutils/sourcedist.html#commands

AHAAA, now I see what went wrong. I called it Manifest.in, not MANIFEST.in.

comment:280 Changed 22 months ago by git

  • Commit changed from 98e6c213914bfe601a54e66640c5425da03bc161 to f06de90c639233c6219cf13b60f302ee5c864e3a

Branch pushed to git repo; I updated commit sha1. New commits:

f06de90p_group_cohomology-3.0: Use xz, not gzip

comment:281 Changed 22 months ago by SimonKing

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Note that the upstream tarball name has changed from ...tar.gz to ...tar.xz. It was a good suggestion to use xz, as the new tarball's size is only about 54% of the old.

I hope the packaging is now cleaner, thanks to using MANIFEST.in. I took the liberty to use SPKG.txt as README file in the Python part of the package (python setup.py sdist expects some kind of README). Hope that's OK.

I checked that the package installs fine and correctly builds the docs. In particular, the checksum provided in the latest commit should be fine.

comment:282 follow-up: Changed 22 months ago by jdemeyer

  • Status changed from needs_review to needs_work

There is still something wrong with the installation:

sage -t pyxsources/pGroupCohomology/auxiliaries.py
**********************************************************************
File "pyxsources/pGroupCohomology/auxiliaries.py", line 56, in pGroupCohomology.auxiliaries.safe_save
Failed example:
    from pGroupCohomology.auxiliaries import safe_save
Exception raised:
    Traceback (most recent call last):
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 534, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 937, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.auxiliaries.safe_save[0]>", line 1, in <module>
        from pGroupCohomology.auxiliaries import safe_save
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/pGroupCohomology/__init__.py", line 2481, in <module>
        from pGroupCohomology.factory import CohomologyRing
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/pGroupCohomology/factory.py", line 43, in <module>
        from pGroupCohomology.auxiliaries import coho_options, coho_logger, safe_save, _gap_init, gap, singular
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/pGroupCohomology/auxiliaries.py", line 488, in <module>
        _gap_init()
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/pGroupCohomology/auxiliaries.py", line 469, in __call__
        G.eval('Read("{}");'.format(os.path.join(SAGE_ROOT,'local','share','sage','ext','gap','modular_cohomology','GapMaxels.g')))
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 581, in eval
        result = Expect.eval(self, input_line, **kwds)
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1297, in eval
        for L in code.split('\n') if L != ''])
      File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 777, in _eval_line
        raise RuntimeError(message)
    RuntimeError: Gap produced error output
    Error, file "/home/jdemeyer/sage-test/local/share/sage/ext/gap/modular_cohomology/GapMaxels\
    .g" must exist and be readable

       executing Read("/home/jdemeyer/sage-test/local/share/sage/ext/gap/modular_cohomology/GapMaxels.g");
**********************************************************************

The file is installed in $SAGE_SHARE/sage/ext/gap/GapMaxels.g instead of $SAGE_SHARE/sage/ext/gap/modular_cohomology/GapMaxels.g

comment:283 follow-up: Changed 22 months ago by jdemeyer

I noticed several serious compiler warnings during the build of the csources, for example:

slice.c: In function 'loadBlock':
slice.c:325:8: warning: implicit declaration of function 'readhdrplus' [-Wimplicit-function-declaration]
   fp = readhdrplus(storedProductFile(ngs, ngs->dimLoaded), NULL, &totrows,
        ^~~~~~~~~~~
slice.c:325:6: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
   fp = readhdrplus(storedProductFile(ngs, ngs->dimLoaded), NULL, &totrows,
      ^

comment:284 in reply to: ↑ 282 Changed 22 months ago by SimonKing

Replying to jdemeyer:

The file is installed in $SAGE_SHARE/sage/ext/gap/GapMaxels.g instead of $SAGE_SHARE/sage/ext/gap/modular_cohomology/GapMaxels.g

Interestingly, I have the file installed in both locations. That's probably an artefact of a previous installation. I'll delete them and see where they will be installed.

comment:285 in reply to: ↑ 283 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

I noticed several serious compiler warnings during the build of the csources, for example:

slice.c: In function 'loadBlock':
slice.c:325:8: warning: implicit declaration of function 'readhdrplus' [-Wimplicit-function-declaration]
   fp = readhdrplus(storedProductFile(ngs, ngs->dimLoaded), NULL, &totrows,
        ^~~~~~~~~~~
slice.c:325:6: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
   fp = readhdrplus(storedProductFile(ngs, ngs->dimLoaded), NULL, &totrows,
      ^

Thanks for spotting it. readhdrplus is declared in fp_decls.h, but the header is not included in slice.c.

I too noticed some warnings. But which ones are serious and which aren't? What is the criterion?

comment:286 Changed 22 months ago by git

  • Commit changed from f06de90c639233c6219cf13b60f302ee5c864e3a to b6b55f18b852bd2024c7d55974858ef20fb7af82

Branch pushed to git repo; I updated commit sha1. New commits:

b6b55f1p_group_cohomology-3.0: Upstream tarball with fixed compiler warnings

comment:287 Changed 22 months ago by SimonKing

  • Status changed from needs_work to needs_review

The current version of modular_resolution (as sub-package of p_group_cohomology) should now install without warning. The only warnings left in the installation of the package seem harmless to me: Some cython variables may be used uninitialised (but in fact I checked at some point that they will definitely be used initialised; their initialisation is in some "if-then-else" construct), some probably harmless warning from python setup.py ...:

    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.pxi' under directory 'pGroupCohomology'
    warning: no previously-included files matching '*.o' found under directory 'pGroupCohomology'
    warning: no previously-included files matching '*.so' found under directory 'pGroupCohomology'
    warning: no previously-included files matching '*' found under directory 'doc/build'

and a warning concerning docs:

copying static files... WARNING: html_static_path entry u'/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.0/src/pyxsources/doc/source/_static' does not exist

I would expect that graft doc/source in MANIFEST.in would also copy the empty folder doc/source/_static, but apparently it doesn't. Since the documenation builds fine, I'd say one can ignore that warning.

comment:288 in reply to: ↑ 285 ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

But which ones are serious and which aren't? What is the criterion?

I can't say that there is an absolute criterion, but the -Wimplicit-function-declaration is actually an error in both C++ and C99. So it's barely standard-compliant code.

comment:289 in reply to: ↑ 288 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

So it's barely standard-compliant code.

Or rather: It was. The warnings about implicit function declaration are gone now.

comment:290 in reply to: ↑ 289 Changed 22 months ago by jdemeyer

Replying to SimonKing:

Or rather: It was. The warnings about implicit function declaration are gone now.

Yes, I understood that. I was just replying to that comment.

comment:291 follow-up: Changed 22 months ago by jdemeyer

On a ppc64le machine, I'm getting some doctest failures:

[p_group_cohomology-3.0] sage -t pyxsources/pGroupCohomology/__init__.py
[p_group_cohomology-3.0] **********************************************************************
[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/__init__.py", line 674, in pGroupCohomology
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     H.8.massey_power()
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 534, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 937, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology[100]>", line 1, in <module>
[p_group_cohomology-3.0]         H.gen(8).massey_power()
[p_group_cohomology-3.0]       File "pGroupCohomology/cochain.pyx", line 2041, in pGroupCohomology.cochain.COCH.massey_power
[p_group_cohomology-3.0]       File "pGroupCohomology/cochain.pyx", line 4522, in pGroupCohomology.cochain.YCOCH.find_cobounding_yoneda_cochains
[p_group_cohomology-3.0]       File "sage/structure/element.pyx", line 3673, in sage.structure.element.Matrix.__mul__ (build/cythonized/sage/structure/element.c:23460)
[p_group_cohomology-3.0]         return coercion_model.bin_op(left, right, mul)
[p_group_cohomology-3.0]       File "sage/structure/coerce.pyx", line 1113, in sage.structure.coerce.CoercionModel_cache_maps.bin_op (build/cythonized/sage/structure/coerce.c:9671)
[p_group_cohomology-3.0]         action = self.get_action(xp, yp, op, x, y)
[p_group_cohomology-3.0]       File "sage/structure/coerce.pyx", line 1656, in sage.structure.coerce.CoercionModel_cache_maps.get_action (build/cythonized/sage/structure/coerce.c:16889)
[p_group_cohomology-3.0]         action = self.discover_action(R, S, op, r, s)
[p_group_cohomology-3.0]       File "sage/structure/coerce.pyx", line 1806, in sage.structure.coerce.CoercionModel_cache_maps.discover_action (build/cythonized/sage/structure/coerce.c:18411)
[p_group_cohomology-3.0]         action = (<Parent>R).get_action(S, op, True, r, s)
[p_group_cohomology-3.0]       File "sage/structure/parent.pyx", line 2495, in sage.structure.parent.Parent.get_action (build/cythonized/sage/structure/parent.c:21664)
[p_group_cohomology-3.0]         action = self.discover_action(S, op, self_on_left, self_el, S_el)
[p_group_cohomology-3.0]       File "sage/structure/parent.pyx", line 2606, in sage.structure.parent.Parent.discover_action (build/cythonized/sage/structure/parent.c:23017)
[p_group_cohomology-3.0]         if parent_is_integers(S) and not self.has_coerce_map_from(S):
[p_group_cohomology-3.0]       File "sage/structure/parent.pyx", line 2024, in sage.structure.parent.Parent.has_coerce_map_from (build/cythonized/sage/structure/parent.c:17925)
[p_group_cohomology-3.0]         return self._internal_coerce_map_from(S) is not None
[p_group_cohomology-3.0]       File "sage/structure/parent.pyx", line 2166, in sage.structure.parent.Parent._internal_coerce_map_from (build/cythonized/sage/structure/parent.c:18944)
[p_group_cohomology-3.0]         mor = self.discover_coerce_map_from(S)
[p_group_cohomology-3.0]       File "sage/structure/parent.pyx", line 2303, in sage.structure.parent.Parent.discover_coerce_map_from (build/cythonized/sage/structure/parent.c:19396)
[p_group_cohomology-3.0]         user_provided_mor = self._coerce_map_from_(S)
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 341, in sage.structure.parent_old.Parent._coerce_map_from_ (build/cythonized/sage/structure/parent_old.c:7549)
[p_group_cohomology-3.0]         return self.coerce_map_from_c(S)
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 159, in sage.structure.parent_old.Parent.coerce_map_from_c (build/cythonized/sage/structure/parent_old.c:4044)
[p_group_cohomology-3.0]         mor = self.coerce_map_from_c(sage_type)
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 143, in sage.structure.parent_old.Parent.coerce_map_from_c (build/cythonized/sage/structure/parent_old.c:3768)
[p_group_cohomology-3.0]         mor = self.coerce_map_from_c_impl(S)
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 191, in sage.structure.parent_old.Parent.coerce_map_from_c_impl (build/cythonized/sage/structure/parent_old.c:4710)
[p_group_cohomology-3.0]         if self.has_coerce_map_from_c(S):
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 277, in sage.structure.parent_old.Parent.has_coerce_map_from_c (build/cythonized/sage/structure/parent_old.c:6312)
[p_group_cohomology-3.0]         x = self.has_coerce_map_from_c_impl(S)
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 286, in sage.structure.parent_old.Parent.has_coerce_map_from_c_impl (build/cythonized/sage/structure/parent_old.c:6489)
[p_group_cohomology-3.0]         self._coerce_c((<parent.Parent>S).an_element())
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 219, in sage.structure.parent_old.Parent._coerce_c (build/cythonized/sage/structure/parent_old.c:5382)
[p_group_cohomology-3.0]         return self._coerce_impl(x)
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/matrix/matrix_space.py", line 1102, in _coerce_impl
[p_group_cohomology-3.0]         return self._coerce_try(x, self.base_ring())
[p_group_cohomology-3.0]       File "sage/structure/parent_old.pyx", line 257, in sage.structure.parent_old.Parent._coerce_try (build/cythonized/sage/structure/parent_old.c:5902)
[p_group_cohomology-3.0]         return self(y)
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/matrix/matrix_space.py", line 875, in __call__
[p_group_cohomology-3.0]         return self.matrix(entries, coerce, copy)
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/matrix/matrix_space.py", line 1888, in matrix
[p_group_cohomology-3.0]         return MC(self, x, copy=copy, coerce=coerce)
[p_group_cohomology-3.0]       File "sage/matrix/matrix_gfpn_dense.pyx", line 425, in sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense.__init__ (build/cythonized/sage/matrix/matrix_gfpn_dense.c:5278)
[p_group_cohomology-3.0]         raise ValueError("Cannot initialise non-square matrix from {}".format(data))
[p_group_cohomology-3.0]     ValueError: Cannot initialise non-square matrix from 1
[p_group_cohomology-3.0] **********************************************************************
[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/__init__.py", line 676, in pGroupCohomology
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     H.element_as_polynomial(_)
[p_group_cohomology-3.0] Expected:
[p_group_cohomology-3.0]     b_2_0^2*a_1_1*a_3_5+b_2_0^2*a_1_0*a_3_4+b_2_0*c_6_8: 8-Cocycle in H^*(E27; GF(3))
[p_group_cohomology-3.0] Got:
[p_group_cohomology-3.0]     a_3_4: 3-Cocycle in H^*(E27; GF(3))
[p_group_cohomology-3.0] **********************************************************************

and various similar failures.

comment:292 in reply to: ↑ 291 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

On a ppc64le machine, I'm getting some doctest failures:

...
[p_group_cohomology-3.0]     ValueError: Cannot initialise non-square matrix from 1

That's very odd. Why should there be a non-square matrix involved? Can you please try on that machine whether you get a different result than I in the following code:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_user_db(tmp_dir())
sage: H = CohomologyRing(27,3)
sage: H.make()
sage: print H

Cohomology ring of Extraspecial 3-group of order 27 and exponent 3 with coefficients in GF(3)

Computation complete
Minimal list of generators:
[b_2_0: 2-Cocycle in H^*(E27; GF(3)),
 b_2_1: 2-Cocycle in H^*(E27; GF(3)),
 b_2_2: 2-Cocycle in H^*(E27; GF(3)),
 b_2_3: 2-Cocycle in H^*(E27; GF(3)),
 c_6_8: 6-Cocycle in H^*(E27; GF(3)),
 a_1_0: 1-Cocycle in H^*(E27; GF(3)),
 a_1_1: 1-Cocycle in H^*(E27; GF(3)),
 a_3_4: 3-Cocycle in H^*(E27; GF(3)),
 a_3_5: 3-Cocycle in H^*(E27; GF(3))]
Minimal list of algebraic relations:
[a_1_0*a_1_1,
 b_2_1*a_1_0-b_2_0*a_1_1,
 b_2_2*a_1_1+b_2_1*a_1_1-b_2_0*a_1_1,
 b_2_2*a_1_0-b_2_1*a_1_1+b_2_0*a_1_1,
 b_2_3*a_1_0-b_2_0*a_1_1,
 -b_2_1^2+b_2_0*b_2_2+b_2_0*b_2_1,
 -b_2_2^2+b_2_1*b_2_3+b_2_1*b_2_2-b_2_1^2,
 -b_2_1*b_2_2-b_2_1^2+b_2_0*b_2_3,
 -b_2_2^2+b_2_1*b_2_2+a_1_1*a_3_4,
 -b_2_1*b_2_2-b_2_1^2+b_2_0*b_2_1+a_1_0*a_3_4,
 -b_2_2*b_2_3+b_2_2^2+a_1_1*a_3_5,
 -b_2_2^2-b_2_1^2+b_2_0*b_2_1+a_1_0*a_3_5,
 b_2_3*a_3_4-b_2_1*a_3_4,
 -b_2_2*a_3_4+b_2_1*a_3_5+b_2_1*a_3_4+b_2_0*b_2_1*a_1_1-b_2_0^2*a_1_1,
 -b_2_1*a_3_4+b_2_0*a_3_5-b_2_0*a_3_4-b_2_0*b_2_1*a_1_1+b_2_0^2*a_1_1,
 b_2_2*a_3_5+b_2_0*b_2_1*a_1_1-b_2_0^2*a_1_1,
 a_3_4*a_3_5+b_2_0*a_1_1*a_3_5+b_2_0*a_1_0*a_3_5-b_2_0*a_1_0*a_3_4]
            
sage: print H.8
3-Cocycle in H^*(E27; GF(3)),
represented by
[0 0 0 0 1 0]
rdeg = 0
ydeg = 1

and various similar failures.

All in .massey_power, or in very different kinds of tests? What is particular about ppc64le? If I recall correctly, there have been some issues on that machine with meataxe matrices (that we have solved, though).

comment:293 Changed 22 months ago by SimonKing

  • Status changed from needs_review to needs_work
  • Work issues set to Investigate failing tests

Ouch, sorry. I just realise that I get the same error.

Actually I did successfully run the test suite in a previous version of the package, and I didn't expect that the changes I did since the last test could possibly affect the computations: The changes have been in the build system and in avoiding compiler warnings, IIRC.

Last edited 22 months ago by SimonKing (previous) (diff)

comment:294 Changed 22 months ago by SimonKing

Sorry, but in the next few days I will very probably not find the time to fix it.

comment:295 in reply to: ↑ 292 ; follow-up: Changed 22 months ago by jdemeyer

Replying to SimonKing:

If I recall correctly, there have been some issues on that machine with meataxe matrices (that we have solved, though).

Yes, the char != signed char issue.

comment:296 in reply to: ↑ 295 ; follow-up: Changed 22 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

If I recall correctly, there have been some issues on that machine with meataxe matrices (that we have solved, though).

Yes, the char != signed char issue.

Yes, that was it. Under what circumstances would it be a problem? Only if one treats a char like a byte, i.e., if one does arithmetic computations with it, right? I briefly hat a look at the code of the modular_resolution subpackage, and it seems that char is really only used for letters, which should be fine, isn't it?

In any case, I see the same error on my computer, too. So, there must be a recent fix that broke things...

comment:297 in reply to: ↑ 296 Changed 22 months ago by jdemeyer

Replying to SimonKing:

Yes, that was it. Under what circumstances would it be a problem?

Typically, when using char for arithmetic. For strings, it probably does not matter.

comment:298 Changed 22 months ago by SimonKing

I didn't find the time to work on the current bug (see comment:292), but here is something else.

Note to self: Use #22611 for the docstrings of instances (here: the @permanent_result and @temporary_result decorators)

comment:299 Changed 21 months ago by SimonKing

Meanwhile I got the impression that the reason for ValueError: Cannot initialise non-square matrix from 1 stems from a change in Sage and not from a change in my code. Namely, I went back to a version in which I know that all doctests used to pass, changed some cdef _mul_ into cpdef _mul_ and tested again. But I still got the same ValueError.

So, apparently I need to initialise the matrix differently.

comment:300 Changed 21 months ago by SimonKing

As it turns out, the whole optional meataxe backend somehow got broken. There have been some matrix related changes in Sage recently - and apparently they haven't been tested with optional packages installed. I'll open a ticket for that issue.

comment:301 Changed 21 months ago by SimonKing

To be more specific: Multiplication of a non-square matrix with the integer 1 is not calling _lmul_ (which it is supposed to!) but tries to convert the integer 1 into the parent of the matrix (which is idiotic since it isn't an algebra), and of course fails as the integer 1 cannot be represented by a non-square matrix.

comment:302 follow-up: Changed 21 months ago by SimonKing

  • Dependencies set to #25076

It is #25076

comment:303 in reply to: ↑ 302 Changed 21 months ago by SimonKing

  • Dependencies changed from #25076 to #25079

Replying to SimonKing:

It is #25076

... and became #25079

comment:304 Changed 21 months ago by git

  • Commit changed from b6b55f18b852bd2024c7d55974858ef20fb7af82 to 138814b1c5fac5c150379e778bd9876d376860c6

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

f0e97afModify the test that ensures that #24742 remains fixed
31f085eEnable _mul_long for matrices
cae999aMore stuff in the meataxe interface, and a meataxe helper function
71a546bFix reading MeatAxe matrix from a file
b9986d5Add p_group_cohomology-3.0 spkg
55b1364p_group_cohomology-3.0: Cope with some API change, improve SPKG.txt
e63bc23p_group_cohomology-3.0: Fix checksum, dependencies and SPKG.txt
58858a3p_group_cohomology-3.0: Remove useless test in spkg-install
14d28e2p_group_cohomology-3.0: Use xz, not gzip
138814bp_group_cohomology-3.0: Upstream tarball with fixed compiler warnings

comment:305 Changed 21 months ago by SimonKing

  • Status changed from needs_work to needs_review
  • Work issues Investigate failing tests deleted

With the current upstream tarball (posted at the location stated in the ticket description) and the current branch (and the dependency), the package builds and passes its test suite. Finally it can be reverted to "needs review"!

comment:306 follow-ups: Changed 21 months ago by jhpalmieri

On OS X, two issues: first, I couldn't get it to install at all because

/bin/bash: .././install-sh: Permission denied

Once I changed permissions on src/csources/install-sh to make it executable, the package installed.

Second, I get lots of doctest failures:

**********************************************************************
File "pyxsources/pGroupCohomology/auxiliaries.py", line 56, in pGroupCohomology.auxiliaries.safe_save
Failed example:
    from pGroupCohomology.auxiliaries import safe_save
Exception raised:
    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.auxiliaries.safe_save[0]>", line 1, in <module>
        from pGroupCohomology.auxiliaries import safe_save
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/__init__.py", line 2481, in <module>
        from pGroupCohomology.factory import CohomologyRing
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.py", line 43, in <module>
        from pGroupCohomology.auxiliaries import coho_options, coho_logger, safe_save, _gap_init, gap, singular
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/auxiliaries.py", line 45, in <module>
        import sage.libs.meataxe
    ImportError: No module named meataxe

I just ran sage -b and now I'm trying again...

comment:307 follow-up: Changed 21 months ago by jhpalmieri

To clarify, I only ran ./sage -i -c p_group_cohomology, at which point Sage installed database_gap and meataxe and then p_group_cohomology, running the test suite for each. Running ./sage -b and then repeating the ./sage -i -c ... command seems to have worked: doctests are passing.

A question: why not run long tests, since I see that a few tests are marked # long time? I haven't tried that; how much time would it add to the tests?

comment:308 in reply to: ↑ 306 ; follow-up: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

On OS X, two issues: first, I couldn't get it to install at all because

/bin/bash: .././install-sh: Permission denied

I am a bit puzzled, as on my machine I have

$ ls -l install-sh 
-rw-r--r-- 1 king king 13997 Jan  7  2016 install-sh

and thus it isn't executable either, but the package does install.

But if it is needed on some machines to change the permission, then I should do it.

Second, I get lots of doctest failures:

**********************************************************************
File "pyxsources/pGroupCohomology/auxiliaries.py", line 56, in pGroupCohomology.auxiliaries.safe_save
Failed example:
    from pGroupCohomology.auxiliaries import safe_save
Exception raised:
    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.auxiliaries.safe_save[0]>", line 1, in <module>
        from pGroupCohomology.auxiliaries import safe_save
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/__init__.py", line 2481, in <module>
        from pGroupCohomology.factory import CohomologyRing
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.py", line 43, in <module>
        from pGroupCohomology.auxiliaries import coho_options, coho_logger, safe_save, _gap_init, gap, singular
      File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/auxiliaries.py", line 45, in <module>
        import sage.libs.meataxe
    ImportError: No module named meataxe

I just ran sage -b and now I'm trying again...

Hm. Now I am even more puzzled.

meataxe is declared as a dependency of the package. Thus, installation of the cohomology package should fail if meataxe isn't installed - and installation of meataxe does involve sage -b, as there are optional cython modules in the Sage library.

Maybe I should ask on sage-devel what exactly is the meaning of a dependency of a package, and why install.sh works on some machines and on others not.

comment:309 in reply to: ↑ 307 ; follow-up: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

To clarify, I only ran ./sage -i -c p_group_cohomology, at which point Sage installed database_gap and meataxe

Hm. I believe the dependencies shouldn't be automatically installed. I expected it to fail with an error message asking the user to install both dependencies.

and then p_group_cohomology, running the test suite for each. Running ./sage -b and then repeating the ./sage -i -c ... command seems to have worked: doctests are passing.

Yes, sage -b is essential for installing meataxe.

A question: why not run long tests, since I see that a few tests are marked # long time? I haven't tried that; how much time would it add to the tests?

I think at some point I have considered that. I will see what it gives. The "short" tests already have a cumulative wall time of 2082.1 seconds on my machine.

comment:310 Changed 21 months ago by SimonKing

Actually the time for sage -t --long happened to be shorter than the time for sage -t (so, perhaps my laptop was more busy when I installed and tested the package in its current version and had less load when I re-tested with long tests).

That indicates that the long tests should be part of the test suite. I'll change it accordingly, but will wait for input of sage-devel concerning the other issues.

comment:311 in reply to: ↑ 308 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

installation of meataxe does involve sage -b, as there are optional cython modules in the Sage library.

No, it does not and this is why it fails.

This is not easy to solve because make does not allow to specify optional dependencies (*). Ideally, we would like to say that sagelib depends on meataxe if meataxe has been installed but I don't know how to do that.

(*) interestingly, it does allow the exact opposite: you can specify that A depends on B only if B has not been installed yet.

comment:312 in reply to: ↑ 309 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

I believe the dependencies shouldn't be automatically installed.

Really? Why not? That's exactly the whole point of specifying dependencies.

comment:313 in reply to: ↑ 306 ; follow-ups: Changed 21 months ago by jdemeyer

If the testsuite depends on running Sage, then $(SAGERUNTIME) must be specified as dependency: just append $(SAGERUNTIME) to build/pkgs/p_group_cohomology/dependencies.

comment:314 Changed 21 months ago by jdemeyer

  • Status changed from needs_review to needs_work

comment:315 in reply to: ↑ 311 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

installation of meataxe does involve sage -b, as there are optional cython modules in the Sage library.

No, it does not and this is why it fails.

Maybe I should have been clearer: There are optional Cython modules in the Sage library depending on meataxe, and the cohomology package uses both meataxe as a c-library libmtx and uses the optional Cython modules in the Sage library.

But while the dependency on libmtx is a build time dependency, IIUC the dependency on the optional Cython modules is a runtime dependency (that's because the .pxd headers are present anyway).

This is not easy to solve because make does not allow to specify optional dependencies (*). Ideally, we would like to say that sagelib depends on meataxe if meataxe has been installed but I don't know how to do that.

It was suggested on sage-devel that meataxe's spkg-install should end with sage -b. So, if it is OK that an optional package triggers building the optional modules in the sage library, that'll be the cleanest solution, I think.

comment:316 in reply to: ↑ 312 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

I believe the dependencies shouldn't be automatically installed.

Really? Why not? That's exactly the whole point of specifying dependencies.

If I recall correctly, it is totally fine for a GPL package to have a runtime dependency that isn't GPL (but it is not OK if it is a build time dependency). Is it OK if installation of a GPL package automatically triggers the installation of a non-GPL package that will be used at runtime?

And that's the situation here. database_gap is still non-GPL, although IIRC its authors finally agreed to relicense it. And p_group_cohomology uses database_gap at run time (i.e., also its test suite depends on it).

comment:317 in reply to: ↑ 313 Changed 21 months ago by SimonKing

Replying to jdemeyer:

If the testsuite depends on running Sage, then $(SAGERUNTIME) must be specified as dependency: just append $(SAGERUNTIME) to build/pkgs/p_group_cohomology/dependencies.

Cool thing!

Would it only trigger sage -b if the test suite is run, or also when just building the package without running tests?

comment:318 in reply to: ↑ 313 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

If the testsuite depends on running Sage, then $(SAGERUNTIME) must be specified as dependency: just append $(SAGERUNTIME) to build/pkgs/p_group_cohomology/dependencies.

I couldn't find a documentation of $(SAGERUNTIME).

Where exactly should $(SAGERUNTIME) be placed? I.e., should build/pkgs/p_group_cohomology/dependencies be

meataxe | database_gap xz $(SAGERUNTIME)

or rather

meataxe $(SAGERUNTIME) | database_gap xz

?

Note that p_group_cohomology links against the Sage library. So, I'd guess it should be the latter version, correct?

comment:319 follow-up: Changed 21 months ago by jhpalmieri

Doctest failures:

Test ran for 290.75 s
**********************************************************************
File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 134, in pGroupCohomology.modular_cohomology._IdGroup
Failed example:
    sorted(D.items())
Expected:
    [(2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)]
Got:
    [('prime', 3), (2520, {0: H^*(8gp3_2520_0; GF(3))})]
**********************************************************************
File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 155, in pGroupCohomology.modular_cohomology._IdGroup
Failed example:
    sorted(D.items())
Expected:
    [(360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)]
Got:
    [('prime', 3), (360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))})]
**********************************************************************

and

File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 159, in pGroupCohomology.modular_cohomology._IdGroup
Failed example:
    sorted(D.items())
Expected:
    [(360, {118: H^*(SmallGroup(360,118); GF(3))}),
     (2520, {0: H^*(8gp3_2520_0; GF(3))}),
     ('prime', 3)]
Got:
    [('prime', 3),
     (360, {118: H^*(SmallGroup(360,118); GF(3))}),
     (2520, {0: H^*(8gp3_2520_0; GF(3))})]
**********************************************************************

and

**********************************************************************
File "pyxsources/pGroupCohomology/resolution.pyx", line 1017, in pGroupCohomology.resolution.RESL
Failed example:
    0.5 < Tu.stats[3]*D.get(Tu.stats[4],1)/(Ta.stats[3]*D.get(Ta.stats[4],1)) < 2
Expected:
    True
Got:
    False
**********************************************************************

comment:320 in reply to: ↑ 319 ; follow-ups: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

Doctest failures:

Test ran for 290.75 s
**********************************************************************
File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 134, in pGroupCohomology.modular_cohomology._IdGroup
Failed example:
    sorted(D.items())
Expected:
    [(2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)]
Got:
    [('prime', 3), (2520, {0: H^*(8gp3_2520_0; GF(3))})]
**********************************************************************
File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 155, in pGroupCohomology.modular_cohomology._IdGroup
Failed example:
    sorted(D.items())
Expected:
    [(360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)]
Got:
    [('prime', 3), (360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))})]
**********************************************************************

and

File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 159, in pGroupCohomology.modular_cohomology._IdGroup
Failed example:
    sorted(D.items())
Expected:
    [(360, {118: H^*(SmallGroup(360,118); GF(3))}),
     (2520, {0: H^*(8gp3_2520_0; GF(3))}),
     ('prime', 3)]
Got:
    [('prime', 3),
     (360, {118: H^*(SmallGroup(360,118); GF(3))}),
     (2520, {0: H^*(8gp3_2520_0; GF(3))})]
**********************************************************************

Interesting. Apparently these are the same dictionaries, but sorting works differently (i.e., is machine dependent).

So, question: How to test that a returned dictionary is what one has expected, if sorting the items doesn't work? Perhaps I should just remove the key prime, assuming that sorting integers still works.

**********************************************************************
File "pyxsources/pGroupCohomology/resolution.pyx", line 1017, in pGroupCohomology.resolution.RESL
Failed example:
    0.5 < Tu.stats[3]*D.get(Tu.stats[4],1)/(Ta.stats[3]*D.get(Ta.stats[4],1)) < 2
Expected:
    True
Got:
    False
**********************************************************************

Interesting. That test is supposed to compare the efficiency of different algorithms, and when I created the tests I took care that the quotients of timings are well within the range [0.5, 2].

Is it possible for you to manually recreate the test, to see in what range the quotient of timings is located? But perhaps I should just mark this test "random".

comment:321 in reply to: ↑ 320 ; follow-up: Changed 21 months ago by jhpalmieri

Replying to SimonKing:

Interesting. Apparently these are the same dictionaries, but sorting works differently (i.e., is machine dependent).

So, question: How to test that a returned dictionary is what one has expected, if sorting the items doesn't work? Perhaps I should just remove the key prime, assuming that sorting integers still works.

You can test sorted(D.items()) == sorted([actual list]) or test whether blah in D.items() is True. Sorting integers should work, so removing prime would also work.

comment:322 in reply to: ↑ 321 Changed 21 months ago by SimonKing

Replying to jhpalmieri:

You can test sorted(D.items()) == sorted([actual list])

Right. Or I could actually manually create a copy of the expected dictionary and compare with the actual output. Good idea.

comment:323 in reply to: ↑ 320 ; follow-up: Changed 21 months ago by jhpalmieri

Replying to SimonKing:

Interesting. That test is supposed to compare the efficiency of different algorithms, and when I created the tests I took care that the quotients of timings are well within the range [0.5, 2].

Is it possible for you to manually recreate the test, to see in what range the quotient of timings is located? But perhaps I should just mark this test "random".

I spent a few minutes trying to recreate it, but I got tired of tracking down all of the definitions to do it by hand. When I run the test (by doing ./sage -t --force-lib /path/to/resolution.pyx), it fails every time. I added a statement printing the quantity in the inequality, and it comes out around between 10 and 11 each time. Maybe marking it # random is the right thing to do.

comment:324 in reply to: ↑ 318 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

It was suggested on sage-devel that meataxe's spkg-install should end with sage -b.

A very bad idea. That will badly break parallel builds: you might end up with two instances of make both building the Sage library.

Replying to SimonKing:

Would it only trigger sage -b if the test suite is run, or also when just building the package without running tests?

It will trigger sage -b unconditionally.

Note that it is not guaranteed that it will run sage -b after building meataxe (this is the part that cannot be fixed). In a serial build make uses the order as specified in the dependencies, so that is why it is better to put $(SAGERUNTIME) after meataxe. But in a parallel build, make can build meataxe and sagelib in any order.

Replying to SimonKing:

Where exactly should $(SAGERUNTIME) be placed? I.e., should build/pkgs/p_group_cohomology/dependencies be

meataxe | database_gap xz $(SAGERUNTIME)

This is the right one. The other alternative would rebuild p_group_cohomology every time.

comment:325 in reply to: ↑ 323 ; follow-up: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

I spent a few minutes trying to recreate it, but I got tired of tracking down all of the definitions to do it by hand. When I run the test (by doing ./sage -t --force-lib /path/to/resolution.pyx), it fails every time. I added a statement printing the quantity in the inequality, and it comes out around between 10 and 11 each time. Maybe marking it # random is the right thing to do.

Thank you! That's very interesting, and I'll have a look what these numbers mean.

comment:326 in reply to: ↑ 324 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Where exactly should $(SAGERUNTIME) be placed? I.e., should build/pkgs/p_group_cohomology/dependencies be

meataxe | database_gap xz $(SAGERUNTIME)

This is the right one. The other alternative would rebuild p_group_cohomology every time.

What do you mean by "every time"? Do you mean that a change in the Sage library followed by make would trigger rebuilding p_group_cohomology? In that case I'd actually prefer the alternative, as I've seen too often that a change in sagelib leaves the cohomology spkg broken, making it necessary to rebuild it.

Ideally, it should of course only be rebuilt if critical headers change, say, sage/structure/element.pxd. But I guess that's not possible, unless all pyx files are put into the sage library.

comment:327 in reply to: ↑ 325 Changed 21 months ago by SimonKing

Replying to SimonKing:

Replying to jhpalmieri:

I spent a few minutes trying to recreate it, but I got tired of tracking down all of the definitions to do it by hand. When I run the test (by doing ./sage -t --force-lib /path/to/resolution.pyx), it fails every time. I added a statement printing the quantity in the inequality, and it comes out around between 10 and 11 each time. Maybe marking it # random is the right thing to do.

Thank you! That's very interesting, and I'll have a look what these numbers mean.

I see now. What is compared are two ways to lift a chain map. One method ("autolift") is used by default in small degrees, as it is fast but tends to consume much memory in higher degrees; thus, the other method ("Urbild Gröbnerbasis") is used in higher degrees.

It is stated in the comment before the text that autolift used to be 250 times faster than Urbild-GB, but after some recent optimisations they became comparable. They still are comparable, on my machine (since the test passes), but on your machine autolift is about 10 times faster than Urbild-GB.

I will change the test so that I give the output on my machine, mark the test as "random", and include in the comment that the ratio is highly machine dependent.

comment:328 Changed 21 months ago by SimonKing

Here is the test:

sage: from pGroupCohomology.cochain import COCH
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_user_db(tmp_dir())
sage: H = CohomologyRing(8,3, from_scratch=True)
sage: R = H.resolution()
sage: C = COCH(H,2,'C',[1,0,1])
sage: c0 = R.CochainToChainmap(2, C.MTX())
sage: c1 = R.liftChainMap(c0)
sage: R.makeAutolift(2)
sage: timeit.eval('cX = R.liftChainMap(c1)')
625 loops, best of 3: 230 µs per loop
sage: timeit.eval('cX = R.ugb_liftChainMap(c1[0]+1,c1[1]+1,c1[2])')
625 loops, best of 3: 297 µs per loop

What do you think: Is it better to have a test that checks that the ratio of execution times is at least a certain number (I am sure I'll find an example where the timings consistently differ by a factor of at least 5)? Or is it better to have a random test and just give the raw figures?

comment:329 Changed 21 months ago by jhpalmieri

I would prefer a random test and just the raw figures. That will be more robust.

comment:330 in reply to: ↑ 326 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

Do you mean that a change in the Sage library followed by make would trigger rebuilding p_group_cohomology?

Yes, even without a change in the Sage library.

comment:331 follow-up: Changed 21 months ago by SimonKing

Concerning autolift: I am currently playing with the default degree up to which the autolift method is used. Let n be a number, and then I do

sage: from pGroupCohomology.factory import unit_test_64
sage: from pGroupCohomology.auxiliaries import coho_options
sage: coho_options['autolift'] = n
sage: unit_test_64()

Here are the figures I'm getting (the first item should be [], and the second item is a tuple "wall time, cpu time of Sage, cpu time of Singular, cpu time of GAP":

  • n=0
    ([], [396.6325900554657, 352.84, 38.83, 18.056])
    
  • n=1, which is the current default
    ([], [391.3751630783081, 350.676, 37.89, 17.484])
    
  • n=2, which used to be the default in an older version of the spkg
    ([], [386.2917420864105, 345.348, 37.52, 17.628])
    
  • n=3
    ([], [386.8177170753479, 346.348, 37.6, 18.056])
    
  • n=4
    ([], [385.62233090400696, 346.06399999999996, 37.18, 17.368])
    
  • n=5
    ([], [400.68896293640137, 356.97999999999996, 39.72, 18.344])
    

Of course each test was in a separate Sage session.

I think the numbers indicate that I should return to the old default, n=2. But, John, since the two lifting methods differ a lot more on your machine than on mine, could you perhaps do these tests as well and tell me the results?

comment:332 in reply to: ↑ 330 ; follow-up: Changed 21 months ago by SimonKing

@John, OK, I will use the robust method to expose the timings in tests.

Replying to jdemeyer:

Replying to SimonKing:

Do you mean that a change in the Sage library followed by make would trigger rebuilding p_group_cohomology?

Yes, even without a change in the Sage library.

Ouch, that's not what I want. So, I'll put $(SAGERUNTIME) to the very end of the dependencies.

And who knows, perhaps in the future it will be possible to put everything into the sage library. My plan for version 3.1 is:

  • Make meataxe standard (rationale: It is A LOT faster than the generic implementation of matrices over non-prime fields of odd characteristic)
  • Make p_group_cohomology work without the SmallGroups library. There are only few places where I really use the SmallGroups identifiers.

If I understand correctly, these used to be the two main obstructions against putting p_group_cohomology into the sage library.

comment:333 in reply to: ↑ 332 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

If I understand correctly, these used to be the two main obstructions against putting p_group_cohomology into the sage library.

Neither of those is actually an obstruction for moving the code to the Sage library. We already have code in sagelib depending on meataxe and other code depending on smallgroups.

comment:334 follow-up: Changed 21 months ago by dimpase

I do not think it is worthwhile to spend any nontrivial amount of time on "work without the SmallGroups library". The new GAP release, 4.9, with the changed SmallGroups library license will be released very soon. While it might take some time to get 4.9 into Sage (!libGAP, you know...), we can trivially (I think) backport the "new" SmallGroups, if needed.

comment:335 in reply to: ↑ 333 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

If I understand correctly, these used to be the two main obstructions against putting p_group_cohomology into the sage library.

Neither of those is actually an obstruction for moving the code to the Sage library. We already have code in sagelib depending on meataxe and other code depending on smallgroups.

If you look at the development of this ticket, at some point I did put all cython and python modules into the sage library. I was told at some point that OptionalExtension should only be used when the optional extension provides a different implementation of a functionality that is also provided by the default sage library. In contrast, one shouldn't use OptionalExtension for functionality that would simply not be available without an optional spkg.

My dream for p_group_cohomology-3.1 is to make it a standard package, and in that case all the python and cython modules should of course be in the sage library.

comment:336 in reply to: ↑ 334 ; follow-up: Changed 21 months ago by SimonKing

Replying to dimpase:

I do not think it is worthwhile to spend any nontrivial amount of time on "work without the SmallGroups library". The new GAP release, 4.9, with the changed SmallGroups library license will be released very soon. While it might take some time to get 4.9 into Sage (!libGAP, you know...), we can trivially (I think) backport the "new" SmallGroups, if needed.

Making database_gap a standard spkg?

comment:337 in reply to: ↑ 335 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

I was told at some point that OptionalExtension should only be used when the optional extension provides a different implementation of a functionality that is also provided by the default sage library. In contrast, one shouldn't use OptionalExtension for functionality that would simply not be available without an optional spkg.

I don't know where you got this idea. There is nothing wrong with truly providing optional functionality (not available otherwise) this way.

Last edited 21 months ago by jdemeyer (previous) (diff)

comment:338 in reply to: ↑ 337 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

I was told at some point that OptionalExtension should only be used when the optional extension provides a different implementation of a functionality that is also provided by the default sage library. In contrast, one shouldn't use OptionalExtension for functionality that would simply not be available without an optional spkg.

I don't know where you got this idea. There is nothing wrong with truly providing optional functionality (not available otherwise) this way.

OK, in that case I could indeed return to the code layout that I had in some intermediate stage of development (i.e.: Have an optional module_resolution-1.0 spkg and have everything else in the Sage library).

Potential issues:

  • If I recall correctly, building documentation of an optional extension module is a problem. Can you confirm? What I'd really like to have is that the documentation of the optional extension in the Sage library is built even if the optional spkg is not installed.
  • The cohomology computations need some helper files that rightfully belong to SAGE_SHARE (namely: code for Singular and GAP), which IIRC are not under version control. So, how to install them? Should modular_resolution-1.0 be install them?
  • Making the cohomology code optional extensions would mean to mark hundreds of doc tests as "optional: modular_resolution", which is highly awkward.

Would anything of the above be a show-stopper? Do you see further problems with that code layout?

comment:339 in reply to: ↑ 336 Changed 21 months ago by dimpase

Replying to SimonKing:

Making database_gap a standard spkg?

yep.

comment:340 in reply to: ↑ 338 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

  • If I recall correctly, building documentation of an optional extension module is a problem. Can you confirm?

Indeed, that does not work.

  • The cohomology computations need some helper files that rightfully belong to SAGE_SHARE (namely: code for Singular and GAP), which IIRC are not under version control. So, how to install them?

Just install them. Any package can install stuff in SAGE_SHARE.

Should modular_resolution-1.0 be install them?

If they are logically considered part of that package, then yes.

  • Making the cohomology code optional extensions would mean to mark hundreds of doc tests as "optional: modular_resolution", which is highly awkward.

Awkward but acceptable.

comment:341 Changed 21 months ago by jdemeyer

Note that I'm not suggesting that optional modules in the Sage library is the best solution. I'm just saying what the possibilities are.

comment:342 in reply to: ↑ 340 Changed 21 months ago by SimonKing

Jeroen, I am also not suggesting that either of the possibilities is clearly best.

Replying to jdemeyer:

Replying to SimonKing:

  • If I recall correctly, building documentation of an optional extension module is a problem. Can you confirm?

Indeed, that does not work.

I find documentation extremely important. How else could a user learn how to use to cohomology code? So, for now I'd prefer to keep the current layout (i.e., not using optional Cython extensions in the Sage library).

comment:343 in reply to: ↑ 331 ; follow-up: Changed 21 months ago by jhpalmieri

Replying to SimonKing:

Concerning autolift: I am currently playing with the default degree up to which the autolift method is used. Let n be a number, and then I do

sage: from pGroupCohomology.factory import unit_test_64
sage: from pGroupCohomology.auxiliaries import coho_options
sage: coho_options['autolift'] = n
sage: unit_test_64()

I think the numbers indicate that I should return to the old default, n=2. But, John, since the two lifting methods differ a lot more on your machine than on mine, could you perhaps do these tests as well and tell me the results?

On two different machines. First, a slowish laptop:

  • n=0:
    ([], [922.0159947872162, 690.2828870000001, 58.54, 22.397])
    
  • n=1:
    ([], [889.0557789802551, 667.6387410000001, 57.38, 21.683])
    
  • n=2:
    ([], [890.1577479839325, 666.102922, 58.01, 22.42])
    
  • n=3:
    ([], [881.2708470821381, 658.588641, 57.66, 22.383])
    
  • n=4:
    ([], [887.1860530376434, 667.076394, 57.81, 21.924])
    
  • n=5:
    ([], [877.5219340324402, 655.3260490000001, 57.6, 22.078])
    

Next, a faster desktop:

  • n=0:
    ([], [401.840313911438, 262.332941, 24.72, 10.603])
    
  • n=1:
    ([], [378.8941731452942, 261.727588, 24.61, 10.553])
    
  • n=2:
    ([], [391.03160214424133, 263.670184, 24.92, 10.487])
    
  • n=3:
    ([], [392.65900707244873, 260.529421, 24.41, 10.477])
    
  • n=4:
    ([], [383.2508010864258, 262.07234, 24.52, 10.461])
    
  • n=5:
    ([], [386.03561902046204, 261.946665, 24.81, 10.523])
    

comment:344 in reply to: ↑ 343 ; follow-up: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

On two different machines.

Thank you!

I don't see a very clear winner, but there is some tendency towards n=2 or n=3. I am now running some tests with groups of order 128. Since I am doing some changes in the package anyway, I can as well change the default behaviour of lifting, although it matters only slightly.

BTW, should I change the permission of install.sh?

comment:345 in reply to: ↑ 344 ; follow-up: Changed 21 months ago by jhpalmieri

Replying to SimonKing:

BTW, should I change the permission of install.sh?

It doesn't build for me otherwise, so either change it in the tar file or I suppose you could change spkg-install:

  • build/pkgs/p_group_cohomology/spkg-install

    diff --git a/build/pkgs/p_group_cohomology/spkg-install b/build/pkgs/p_group_cohomology/spkg-install
    index d7ff1ee75b..82f940cecf 100644
    a b cd src 
    22
    33# building modular_resolution-1.0
    44cd csources
     5chmod a+x install-sh
    56sdh_configure
    67sdh_make
    78# sdh_make install got broken by trac ticket #24106

(That's what I did, because then I didn't have to prepare a new tar file and change the checksums. It seems cleaner to change it in the actual tar file, though.)

Last edited 21 months ago by jhpalmieri (previous) (diff)

comment:346 in reply to: ↑ 345 ; follow-up: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

(That's what I did, because then I didn't have to prepare a new tar file and change the checksums. It seems cleaner to change it in the actual tar file, though.)

On the other hand, changing it in the tar file makes it more difficult to maintain the spkg - as changing the permission could be easily forgotten.

comment:347 in reply to: ↑ 346 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

On the other hand, changing it in the tar file makes it more difficult to maintain the spkg - as changing the permission could be easily forgotten.

Actually, I'm guessing that you did something wrong in creating this tar file, or that the automake on your system is subtly broken. Normally, install-sh is created executable.

comment:348 in reply to: ↑ 347 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing: Actually, I'm guessing that you did something wrong in creating this tar file, or that the automake on your system is subtly broken. Normally, install-sh is created executable.

When I do make dist for the autotoolized modular_resolution-1.0 sub-package, then I get a .gz and a .bz2 tarball. For these, I get

$ tar -xpf modular_resolution-1.0.tar.gz 
$ ls -l modular_resolution-1.0/install-sh 
-rw-r--r-- 1 king king 13997 Jan  7  2016 modular_resolution-1.0/install-sh
$ rm -r modular_resolution-1.0
$ tar -xpf modular_resolution-1.0.tar.bz2 
$ ls -l modular_resolution-1.0/install-sh 
-rw-r--r-- 1 king king 13997 Jan  7  2016 modular_resolution-1.0/install-sh

ltmain.sh isn't executable either. And I think tar -xpf should preserve permissions (even tar -xf should, right?).

Moreover, when I do make distdir instead, then even in the uncompressed source tree the permissions are wrong. So, I suppose the error is in my autotools.

I tried some duckduckgo search with autotools install.sh wrong permissions, which pointed me to https://github.com/healpy/healpy/issues/154. There, the first comments seem to indicate that inspite of install.sh not being executable, installation works on linux but not on OS X. However, I am not sure if that helps here. BTW, a search where I replace install.sh with install-sh revealed even less.

Is there really no way around manually editing the permissions? Perhaps letting spkg-install do the job is best, after all.

comment:349 in reply to: ↑ 348 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

When I do make dist

I'm guessing that the problem is not with make dist, that just packages things up. What matters is the permission of install-sh in the sources, before you even run make dist. That install-sh is normally created by automake.

comment:350 in reply to: ↑ 349 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

When I do make dist

I'm guessing that the problem is not with make dist, that just packages things up. What matters is the permission of install-sh in the sources, before you even run make dist. That install-sh is normally created by automake.

The permissions are wrong even in the sources.

So, I guess the maintenance problem can be solved by changing the permissions in the sources. For all *sh files, right?

comment:351 follow-up: Changed 21 months ago by jdemeyer

The following works for me:

cd csources
autoreconf -fiv

comment:352 in reply to: ↑ 351 Changed 21 months ago by SimonKing

Replying to jdemeyer:

The following works for me:

cd csources
autoreconf -fiv

Great, it does work! Thank you. By the way, it also slightly changed the INSTALL file.

comment:353 Changed 21 months ago by jdemeyer

It's generally a good idea to run autoreconf -fiv before packaging a release as it ensures that all autotools-generated files are up-to-date. This includes also config.guess which has regularly been an issue in the past (see #19714 for example).

comment:354 Changed 21 months ago by git

  • Commit changed from 138814b1c5fac5c150379e778bd9876d376860c6 to 441ee4250588968bd122c8ff54989e91c75344f8

Branch pushed to git repo; I updated commit sha1. New commits:

29a5fdap_group_cohomology-3.0: Let test suite include long tests
c2541fbp_group_cohomology-3.0: Add SAGERUNTIME to the dependencies
1103493Use classify_elements more efficiently
e41b514Merge branch 't/25079/use__mul_long_matrix_int' into p_group_cohomology-3.0
441ee42p_group_cohomology-3.0: Change upstream tarball (permissions, doctest changes)

comment:355 Changed 21 months ago by SimonKing

I have updated the upstream tarball and added changes to Sage, as discussed above.

However, there is a surprising problem: sage -f -c p_group_cohomology now hangs in the tests, after compiling the spkg. top shows that Sage works for some time, but then it simply shows no activity. I tried three times with no success.

What could be wrong??

comment:356 follow-up: Changed 21 months ago by SimonKing

Strangely, the tests eventually are executed. But certainly Sage shouldn't be sleeping during tests.

comment:357 in reply to: ↑ 356 Changed 21 months ago by jdemeyer

Replying to SimonKing:

But certainly Sage shouldn't be sleeping during tests.

It would help to run the tests with --verbose so you can see what is going on.

comment:358 Changed 21 months ago by SimonKing

Also, when I interrupt the tests with Ctrl-c, I don't get a bash. So, somehow Sage hangs even when it is interrupted. I will now re-try with verbose tests.

comment:359 follow-up: Changed 21 months ago by SimonKing

It hangs for a long time at lines such as

H1 = CohomologyRing(8,3)

Perhaps the program is trying to access the internet. I am now in my office, where I often have problems with internet connection. At home, internet works fine, perhaps that is why the tests did work there.

However, I thought that I had disallowed internet access during tests. In any case, if an internet connection cannot be established, the program should compute from scratch.

Hm. On second thought: CohomologyRing(8,3) is part of the database that comes with the package. Thus, it shouldn't even attempt to use internet! Is perhaps my hard disc drive broken?

comment:360 Changed 21 months ago by SimonKing

I just tested in a new Sage session: Yes, H1 = CohomologyRing(8,3) does what I expected. It just takes the ring out of the data base and is done in a fraction of a second.

comment:361 Changed 21 months ago by SimonKing

During the tests, basically each cohomology ring definition hangs for several minutes before it suddenly works.

Can some of you please run the tests, too?

comment:362 Changed 21 months ago by jdemeyer

  • Dependencies changed from #25079 to #25079, #25106
[p_group_cohomology-3.0] Traceback (most recent call last):
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/src/bin/sage-runtests", line 127, in <module>
[p_group_cohomology-3.0]     err = DC.run()
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/control.py", line 1172, in run
[p_group_cohomology-3.0]     self.run_doctests()
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/control.py", line 895, in run_doctests
[p_group_cohomology-3.0]     self.dispatcher = DocTestDispatcher(self)
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1485, in __init__
[p_group_cohomology-3.0]     init_sage()
[p_group_cohomology-3.0]   File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 116, in init_sage
[p_group_cohomology-3.0]     import matplotlib.font_manager
[p_group_cohomology-3.0] ImportError: No module named matplotlib.font_manager

I created #25106 for this.

comment:363 in reply to: ↑ 359 ; follow-ups: Changed 21 months ago by jdemeyer

Replying to SimonKing:

Thus, it shouldn't even attempt to use internet!

Perhaps this part of the package/installation is broken?

comment:364 in reply to: ↑ 363 ; follow-ups: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Thus, it shouldn't even attempt to use internet!

Perhaps this part of the package/installation is broken?

Some tests expose logging. I guess in these tests it would be visible whether or not the package attempts to access internet.

ANd has matplotlib.font_manager (previous comment) really something to do with this ticket?? If not, please remove the additional dependency.

comment:365 in reply to: ↑ 363 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Thus, it shouldn't even attempt to use internet!

Perhaps this part of the package/installation is broken?

I don't see what change could cause this. There haven't been any changes to the code in the last few commits.

comment:366 Changed 21 months ago by jdemeyer

I confirm that the testsuite is connecting to the internet.

comment:367 in reply to: ↑ 365 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

I don't see what change could cause this. There haven't been any changes to the code in the last few commits.

Maybe the bug exists for a long time. Why do you specifically say "in the last few commits"?

comment:368 follow-up: Changed 21 months ago by jdemeyer

Your package is trying to read database files that do not exist. For example pGroupCohomology/4gp2/H4gp2.sobj but there is no 4gp2 directory.

comment:369 in reply to: ↑ 364 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

ANd has matplotlib.font_manager (previous comment) really something to do with this ticket??

No, but it does cause doctest failures. So it must be a dependency.

comment:370 follow-up: Changed 21 months ago by jdemeyer

Is it intentional that the tarball pGroupCohomology/8gp3.tar.gz is installed? That looks like a bug to me.

comment:371 follow-up: Changed 21 months ago by jdemeyer

There is also this doctest failure:

[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 4513, in pGroupCohomology.cohomology.COHO.__getattr__
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     import sagenb.misc.support as s
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology.cohomology.COHO.__getattr__[9]>", line 1, in <module>
[p_group_cohomology-3.0]         import sagenb.misc.support as s
[p_group_cohomology-3.0]     ImportError: No module named sagenb.misc.support

Why do you need sagenb?

comment:372 in reply to: ↑ 367 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

I don't see what change could cause this. There haven't been any changes to the code in the last few commits.

Maybe the bug exists for a long time. Why do you specifically say "in the last few commits"?

Because there have been changes in that aspect in the past. Feels like a year ago or so. But certainly that has been a long time before the version that passed tests easily (which was 4 days ago in commit:305).

comment:373 in reply to: ↑ 370 ; follow-up: Changed 21 months ago by jhpalmieri

Replying to jdemeyer:

Is it intentional that the tarball pGroupCohomology/8gp3.tar.gz is installed? That looks like a bug to me.

The bug is probably that 8gp3.tar.gz is in group_cohomology_database.tar.

comment:374 Changed 21 months ago by jdemeyer

Do you still have the previous version which didn't have the "internet connection" problem?

comment:375 in reply to: ↑ 373 ; follow-up: Changed 21 months ago by SimonKing

When I fixed the internet problem (I DID fix it!), I made sure that all tests pass, regardless whether internet connection is available or not. Even the tests that were explicitly about accessing the database in the web has been modified so that only local file access is needed.

Replying to jhpalmieri:

Replying to jdemeyer:

Is it intentional that the tarball pGroupCohomology/8gp3.tar.gz is installed? That looks like a bug to me.

The bug is probably that 8gp3.tar.gz is in group_cohomology_database.tar.

If I recall correctly, 8gp3.tar.gz was specifically added in order to be able to simulate a web database by local file access in doctests. So, rather not a bug.

comment:376 in reply to: ↑ 371 ; follow-up: Changed 21 months ago by jhpalmieri

Replying to jdemeyer:

There is also this doctest failure:

[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 4513, in pGroupCohomology.cohomology.COHO.__getattr__
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     import sagenb.misc.support as s
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/jdemeyer/sage-test/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology.cohomology.COHO.__getattr__[9]>", line 1, in <module>
[p_group_cohomology-3.0]         import sagenb.misc.support as s
[p_group_cohomology-3.0]     ImportError: No module named sagenb.misc.support

Why do you need sagenb?

See #24998 for a fix for this (a replacement for the completions method from sagenb.misc.support).

comment:377 Changed 21 months ago by jdemeyer

  • Dependencies changed from #25079, #25106 to #25079, #25106, #24998

comment:378 in reply to: ↑ 368 ; follow-up: Changed 21 months ago by SimonKing

  • Dependencies changed from #25079, #25106, #24998 to #25079, #25106

Replying to jdemeyer:

Your package is trying to read database files that do not exist. For example pGroupCohomology/4gp2/H4gp2.sobj but there is no 4gp2 directory.

Unless requested otherwise: The program first looks into the current private database (in DOT_SAGE). If the file doesn't exist then the program looks into the current public database (in SAGE_SHARE) and creates links from the private database to relevant files of the public database (and if the user does computations with the ring, then new files are added only to the private database, not to the public database, since the user may not have write permission there). And if the program doesn't find it in the public database either, it tries to access a repository in the internet. It would download a .tar.gz-file, unpack it in the private database, and load the relevant data. If that fails (and it is supposed to fail very quickly, since Sage is supposed to block internet access during tests --- it certainly did in the past!) then the ring is computed from scratch, with files residing in the private database.

Hence, the missing file should simply be created. It shouldn't give rise to an error.

comment:379 in reply to: ↑ 376 ; follow-up: Changed 21 months ago by SimonKing

Replying to jhpalmieri:

Why do you need sagenb?

To simulate tab completion.

See #24998 for a fix for this (a replacement for the completions method from sagenb.misc.support).

Has there been a deprecation warning at some point?

comment:380 in reply to: ↑ 379 Changed 21 months ago by jdemeyer

Replying to SimonKing:

Has there been a deprecation warning at some point?

A deprecation for what? Sagenb is sort-of deprecated but still supported.

comment:381 in reply to: ↑ 378 ; follow-ups: Changed 21 months ago by jdemeyer

Replying to SimonKing:

it is supposed to fail very quickly

It doesn't. That's your problem.

comment:382 in reply to: ↑ 369 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

ANd has matplotlib.font_manager (previous comment) really something to do with this ticket??

No, but it does cause doctest failures. So it must be a dependency.

You mean it causes doctest failures in the cohomology package? Why should that possibly happen? I mean, there is plotting in the barcode.py module, but I don't think it uses matplotlib (unless matplotlib is the default in Sage when calling the plot() function).

But perhaps you see my frustration:

  • 3 years ago, after comment:52, I fixed the problem with internet access. At that point, Sage decided to generally block internet access during tests. So, I fixed it, now it comes up again.
  • 18 months ago (comment:149) I got all tests to pass, but I had some questions, so, I left it as "needs_info".
  • Later, changes broke even installation of the package. I fixed it.
  • 4 days ago, all tests passed. And all of the sudden everything became foobar again, since all of a sudden doctests that easily pass interactively won't reasonably work during the test suite, and since some module from the Sage source tree got relocated.
Last edited 21 months ago by SimonKing (previous) (diff)

comment:383 in reply to: ↑ 381 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

it is supposed to fail very quickly

It doesn't. That's your problem.

But what has changed in the last 4 days that could explain it? Sage blocks internet access during tests since at least 3 years, and I coped with it 3 years ago. Is it perhaps the case that Sage can block using Wifi but cannot properly block ethernet? Because that's the only difference I can see: At home, I have Wifi, in my office I have ethernet.

comment:384 Changed 21 months ago by SimonKing

BTW, what I didn't understand: Do you currently see the same problem than I (namely: hanging tests)? Or is it just on my machine?

I'll repeat the tests at home, and IMHO it would be fine to say that it is required to run the tests without connecting the computer to the ethernet of Friedrich-Schiller-Universität Jena.

comment:385 Changed 21 months ago by SimonKing

PS: Still it is "hooray".

[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] All tests passed!
[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] Total time for all tests: 12883.2 seconds
[p_group_cohomology-3.0]     cpu time: 641.8 seconds
[p_group_cohomology-3.0]     cumulative wall time: 22076.6 seconds

The time to run the tests is of course a disaster, but still all tests pass. In particular the tests pass that check against a specific log. The fact that the log is as expected demonstrates that internet access was off, as it should be. So, perhaps it is just Sage not realising that the internet was off.

comment:386 follow-ups: Changed 21 months ago by SimonKing

Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage. Could even be file access.

comment:387 in reply to: ↑ 386 ; follow-up: Changed 21 months ago by dimpase

Replying to SimonKing:

Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage. Could even be file access.

in your office, do you perhaps have ~/.sage/ on an NFS volume? As e.g. GAP workspaces are kept there, it could be a performance killer.

comment:388 in reply to: ↑ 387 Changed 21 months ago by SimonKing

Replying to dimpase:

Replying to SimonKing:

Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage. Could even be file access.

in your office, do you perhaps have ~/.sage/ on an NFS volume? As e.g. GAP workspaces are kept there, it could be a performance killer.

No, it is just my usual laptop plugged to ethernet cable.

comment:389 in reply to: ↑ 386 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage.

It's the internet connection, see 381.

comment:390 follow-up: Changed 21 months ago by jhpalmieri

Here is a problem. First I deleted (from my .sage/pGroupCohomology/db directory) the directories 8gp1, 8gp2, 8gp3, 8gp4. Then I started Sage and did this:

sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(8,1)
sage: H = CohomologyRing(8,2)
sage: H = CohomologyRing(8,3)
sage: H = CohomologyRing(8,4)

This recreated those directories. I quit Sage and started again:

sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(8,1)
BOOM
sage: H = CohomologyRing(8,2)
BOOM
sage: H = CohomologyRing(8,3)
sage: H = CohomologyRing(8,4)

The BOOM means that the first two commands produce errors, but the last two seem to work fine. The error looks like:

IOError                                   Traceback (most recent call last)
<ipython-input-2-887a1ff3a72e> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(1))

/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds)
   1490         if q.is_prime_power():
   1491             CacheKey = (KEY, os.path.join(root_user_db,GStem,'dat','State'))
-> 1492             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1493             return OUT
   1494 

/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1118                 if coho_options.has_key('@use_this_root@'):
   1119                     del coho_options['@use_this_root@']
-> 1120                 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_user_db,file_name), msg))
   1121         ## 3. Link with public db and load from there
   1122         elif os.access(os.path.join(root_public_db,file_name), os.R_OK) and not from_scratch:

IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/8gp1/H8gp1.sobj are not readable: Singular error:
   ? ...parse error
   ? error occurred in or before STDIN line 18: `ideal COHO1I;`
   ? wrong type declaration. type 'help ideal;'

It's weird: I just quit and started again, and CohomologyRing(8,1) gave me this error the first time, but worked (in the same Sage session) after that, while CohomologyRing(8,2) gave an error every time I used it.

comment:391 Changed 21 months ago by jhpalmieri

If my error is not reproducible for anyone else, it may be due to breakage with the most recent Xcode on OS X: that has at least caused some problems which might be related to Singular. People should test it on other platforms, though.

comment:392 in reply to: ↑ 389 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage.

It's the internet connection, see 381.

OK. Could one tell the user that the test suite should be run without being connected to internet (I mean: REALLY plugging WLAN and ethernet off)?

comment:393 in reply to: ↑ 390 Changed 21 months ago by SimonKing

  • Work issues set to Allow (un)pickling of empty rings

Dear John,

Replying to jhpalmieri:

Here is a problem. First I deleted (from my .sage/pGroupCohomology/db directory) the directories 8gp1, 8gp2, 8gp3, 8gp4. Then I started Sage and did this:

sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(8,1)
sage: H = CohomologyRing(8,2)
sage: H = CohomologyRing(8,3)
sage: H = CohomologyRing(8,4)

This recreated those directories. I quit Sage and started again:

sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(8,1)
BOOM

}}}

Nice catch!

Singular error:
   ? ...parse error
   ? error occurred in or before STDIN line 18: `ideal COHO1I;`
   ? wrong type declaration. type 'help ideal;'

It's weird: I just quit and started again, and CohomologyRing(8,1) gave me this error the first time, but worked (in the same Sage session) after that, while CohomologyRing(8,2) gave an error every time I used it.

I tried twice, and I got the same error twice.

Here is how to work around the problem:

  • Delete the folders
  • Start Sage and create the rings
  • COMPUTE the rings (with H.make())
  • quit and restart sage
  • create the rings again, and they are correctly loaded.

I also tried to just save the empty rings instead of computing the ring structure, but this didn't help.

So, I can reproduce it. I guess the reason is that the pickle of a freshly created (i.e., empty) ring structure constitutes a corner case in which certain autogenerated definitions in Singular become syntax errors. I'll investigate what is happening here. Certainly it SHOULD work!

comment:394 follow-up: Changed 21 months ago by jdemeyer

Would it not be possible to put all files needed for the testsuite in the package? That way, you avoid the internet connection.

comment:395 in reply to: ↑ 381 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

it is supposed to fail very quickly

It doesn't. That's your problem.

I still don't understand what is your exact statement. So, let me ask again.

  • Do you observe similar slowness when you run the test on your machine?
  • Or do you just observe that internet is accessed, but without slowness?

By the way, I do have reason to believe that there is something weird going on in the combination of my laptop with my office: If my laptop is connected with the ethernet in my office and I try to shut the laptop down, then the shutdown of ubuntu sometimes takes ages. So, before shutting down my laptop I have to disconnect from ethernet, wait a few seconds till the network manager confirms that it is disconnected, and then I can start to shut down.

At the moment I am trying to run the tests at home.

comment:396 in reply to: ↑ 394 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Would it not be possible to put all files needed for the testsuite in the package? That way, you avoid the internet connection.

Theoretically I could put more rings into the "public database" that is shipped with the spkg.

However, I have reason to believe that this will not solve the problem.

Namely, in my office, I have seen that even H = CohomologyRing(8,3) hangs. Now the point is that this particular ring is contained in the "public database" (this is something else than the tarball used to simulate web access in some tests). Hence, H = CohomologyRing(8,3) is supposed to NEVER access internet,vwith only one exception: If CohomologyRing.set_public_db() was used to redefine the location of the public database.

So, if this hangs then it isn't the internet. And if this hangs then putting more stuff into the public database will not solve it.

By the way: While I was writing these lines, the spkg installed and passed all its tests in a total of less than 20 minutes wall-time.

comment:397 follow-up: Changed 21 months ago by SimonKing

Let me review how the package behaves in different internet settings. It is all on the same ubuntu laptop.

  • With no internet connection whatsoever:
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 729.6 seconds
        cpu time: 1127.5 seconds
        cumulative wall time: 2119.5 seconds
    
  • With W-Lan at my home:
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 624.6 seconds
        cpu time: 850.7 seconds
        cumulative wall time: 1764.2 seconds
    
  • With ethernet in my office (note that in that setting interactive use of the package does not show slowness!)
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 12883.2 seconds
        cpu time: 641.8 seconds
        cumulative wall time: 22076.6 seconds
    

Conclusions

  1. The tests pass under all circumstances. Therefore, it is not the case that the tests "depend on internet" or "are broken by internet".
  2. The comparison "no internet" vs. "W-Lan" shows that the tests work faster if data can be found in internet. I don't see that as a problem. What matters is that the eventual outcome is the same.
  3. Very likely the package is not to blame for the slowness of the tests with ethernet connection. Evidence: In that setting I have all sorts of trouble (slowness in shutting down my computer, and also thunderbird sometimes doesn't properly work in my office).
  4. Therefore, I do not think that the internet issue needs to be addressed.

TODO

  1. I need to check that pre-existing user data have no influence on the doctests. I.e., I will remove ~/.sage/pGroupCohomology and SAGE_SHARE/pGroupCohomology before re-installing and re-testing the package. Both with and without internet connection.
  2. The bug found by John ("creation of empty rings breaks pickling") needs to be fixed. But other than that, I still think the package is fine.
Last edited 21 months ago by SimonKing (previous) (diff)

comment:398 in reply to: ↑ 395 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

  • Do you observe similar slowness when you run the test on your machine?

Yes:

[p_group_cohomology-3.0] Total time for all tests: 11383.1 seconds
[p_group_cohomology-3.0]     cpu time: 1155.4 seconds
[p_group_cohomology-3.0]     cumulative wall time: 23145.5 seconds
  • Or do you just observe that internet is accessed, but without slowness?

It's trying to access the internet but failing since (as you correctly noted) internet is blocked during package installation.

comment:399 in reply to: ↑ 397 Changed 21 months ago by jdemeyer

Replying to SimonKing:

  1. Therefore, I do not think that the internet issue needs to be addressed.

I strongly disagree. The testsuite spends 3 hours doing absolutely nothing (waiting for an internet connection that will never succeed).

comment:400 follow-up: Changed 21 months ago by jdemeyer

Note that there are several easy ways to overcome the internet problem:

  1. Never access the internet for easy groups: just do the computation locally.
  1. Simply add more data to the database such that internet access won't be needed during the tests.
  1. Have some option (environment variable or whatever) to disable internet lookup and use that during testing.

comment:401 in reply to: ↑ 375 ; follow-up: Changed 21 months ago by jdemeyer

Replying to SimonKing:

If I recall correctly, 8gp3.tar.gz was specifically added in order to be able to simulate a web database by local file access in doctests. So, rather not a bug.

If that file is only needed for the testsuite, it shouldn't be installed. You could ship it in the sources in some testdata subdirectory or whatever.

comment:402 in reply to: ↑ 228 Changed 21 months ago by jdemeyer

Replying to SimonKing:

The binding=True option led to 20% performance loss, that's why I wouldn't use it even if the bug with default options was fixed.

Off-topic, but I'm working on this: https://www.python.org/dev/peps/pep-0575/

comment:403 in reply to: ↑ 398 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

  • Do you observe similar slowness when you run the test on your machine?

Yes:

Thank you. You hadn't been clear enough (for me at least) about that before.

  • Or do you just observe that internet is accessed, but without slowness?

It's trying to access the internet but failing since (as you correctly noted) internet is blocked during package installation.

Aha! Actually I incorrectly noted that internet was blocked during doctests. So, it is about package installation, not about doctests.

comment:404 in reply to: ↑ 401 Changed 21 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

If I recall correctly, 8gp3.tar.gz was specifically added in order to be able to simulate a web database by local file access in doctests. So, rather not a bug.

If that file is only needed for the testsuite, it shouldn't be installed. You could ship it in the sources in some testdata subdirectory or whatever.

Good idea!

comment:405 in reply to: ↑ 400 ; follow-up: Changed 21 months ago by SimonKing

Replying to jdemeyer:

Note that there are several easy ways to overcome the internet problem:

  1. Never access the internet for easy groups: just do the computation locally.

I have to think about it. "Small" probably means "up to order 125". In orders 1 to 127, only the groups of order 64 take a serious amount of time to compute, but order 64 is provided by the public database, so they are available locally anyway.

  1. Have some option (environment variable or whatever) to disable internet lookup and use that during testing.

Or:

  1. Have a function that quickly determines whether sage -t is running. Use internet if that function returns "False", otherwise do not use internet.

Would it be difficult to check for sage -t?

comment:406 Changed 21 months ago by SimonKing

  • Work issues changed from Allow (un)pickling of empty rings to Internet; (un)pickling of empty rings; test altering private database

All tests pass when one first removes all existing cohomology data.

However, I noticed that after running the test suite there are 274 rings in the user's private database DOT_SAGE/pGroupCohomology/db/. That's another issue to be fixed.

comment:407 Changed 21 months ago by SimonKing

  • Work issues changed from Internet; (un)pickling of empty rings; test altering private database to Internet; empty rings; tests alter private database; do not install test data files

comment:408 Changed 21 months ago by SimonKing

As I stated on sage-devel, I consider adding a function CohomologyRing.doctest_setup() that I would insert at the beginning of each doctest. It would

  • disallow internet access
  • setup a temporary directory for cohomology computations (I am doing this in most tests anyway)
  • clear the cache (I am doing that in some tests already to prevent side-effects of tests)

In that way, two work issues ("internet" and "tests alter private database") would be resolved.

comment:409 Changed 21 months ago by SimonKing

  • Work issues changed from Internet; empty rings; tests alter private database; do not install test data files to Internet; empty rings; tests alter private database; do not install test data files; use #24998

comment:410 Changed 21 months ago by SimonKing

  • Dependencies changed from #25079, #25106 to #25079, #25106, #24998

comment:411 Changed 21 months ago by jdemeyer

  • Summary changed from Upgrade of group cohomology spkg to Upgrade of p_group_cohomology spkg

comment:412 follow-ups: Changed 21 months ago by SimonKing

Can PLEASE some expert for trac finally make it so that the browser isn't insanely jumping its focus while typing a comment?!?

Lectures are starting tomorrow, hence I currently only have little time to work on the spkg.

But at least I'd like to recall some design choices and then outline the next steps I'm planning.

So far I distinguished user_db, public_db and web_db, where db in all cases means "database". However, I think I should distinguish the roles that all these databases are playing:

  • A user_db is located in DOT_SAGE and is where new cohomology data are by default written to. It is formed by a bunch of folders, one folder for one ring. ALL computations are done there, even if the computation starts with data obtained from a public_db.
  • A public_db is a source of cohomology data located in SAGE_SHARE; it is formed by folders located in a common root directory, one folder for one cohomology ring. Loading a ring from a public_db means to create links from user_db to public_db.
  • A web_db is some url providing cohomology rings in the form of .tar.gz files, one file for one ring, that when unpacked results in a folder suitable for user_db.

Now I plan to do a little relabelling:

  1. A more suitable label for user_db would be workspace. In most cases a single cohomology ring relies on computing various other cohomology rings. It makes sense to have a single workspace for all rings that the user is doing computations with. So, there should be a global option defining the location of the workspace (such a global option exists already).
  2. In contrast, both user_db and web_db are local resp. web based data sources. It should be totally fine to have several data sources simultaneously. So, there should be global options local_sources and web_sources providing a tuple of paths resp. URLs (the current global options only allow a single location). Or, with the unifications outlined below, it would be enough to have a single global option sources providing both kinds of sources.
  3. It should be possible to declare the optional data sources in the cohomology ring constructor, so that it will be used for the ring-to-be-constructed and the other rings that it depends on.

The relabelled global options would be used as follows:

  1. The cohomology ring constructor first tries to find a ring in workspace. If it isn't there, it would try to load it (in order) from the available local_sources. If it isn't in any of them, it would try to download it (in order) from the available web_sources.
  2. I mentioned a function CohomologyRing.doctest_setup(). That function would set workspace=tmp_dir() and web_sources=(), thus solving the "internet in doctests" issue.

All of the above could be done soonish (modulo the time that I need to prepare lectures). In particular, once the above is done, it would probably be in a sufficient shape for review.

After that (or when the review takes too long), I plan to rework the internal logic by which the file locations are managed. It currently works, but I find it messy.

And now I have a question for you: I have mentioned above that local databases provide one folder per ring, whereas web databases provide the .tar.gz versions of such folders. It would be tempting to unify the format, so that I can unify the process of (down)loading a cohomology ring. So, what shall I do?

  • Change the web databases so that they provide folders? Downside would be that downloading a ring would need more bandwith.
  • Change the local databases (not the workspace, though!) so that they provide .tar.gz balls? Downsides: Loading would take longer, and it wouldn't be possible to preserve storage space on disk by using links from workspace to local database.
  • Keep the current formats, but still unify the routine that loads a cohomology ring? I.e., if there is a folder in the database, then use it, otherwise resort to the tar-ball.
  • None of the above?

comment:413 in reply to: ↑ 405 ; follow-up: Changed 20 months ago by jdemeyer

Replying to SimonKing:

I have to think about it. "Small" probably means "up to order 125". In orders 1 to 127, only the groups of order 64 take a serious amount of time to compute, but order 64 is provided by the public database, so they are available locally anyway.

What kind of time are we talking about to compute one group? If it's less than 100ms, I would compute it locally because opening an internet connection typically takes longer than that. That would not only help doctests, but also actual users.

comment:414 in reply to: ↑ 412 ; follow-up: Changed 20 months ago by jdemeyer

Replying to SimonKing:

  1. It should be possible to declare the optional data sources in the cohomology ring constructor

This sounds strange to me, a single global option makes more sense. Is there any reason why you would want to use different data sources for different rings?

comment:415 in reply to: ↑ 412 ; follow-up: Changed 20 months ago by jdemeyer

Replying to SimonKing:

  • Change the web databases so that they provide folders? Downside would be that downloading a ring would need more bandwith.
  • Change the local databases (not the workspace, though!) so that they provide .tar.gz balls? Downsides: Loading would take longer, and it wouldn't be possible to preserve storage space on disk by using links from workspace to local database.
  • Keep the current formats, but still unify the routine that loads a cohomology ring? I.e., if there is a folder in the database, then use it, otherwise resort to the tar-ball.
  • None of the above?

I would go for for "None of the above" and keep the current implementation. Is there any downside to that?

comment:416 in reply to: ↑ 413 Changed 20 months ago by SimonKing

Replying to jdemeyer:

What kind of time are we talking about to compute one group? If it's less than 100ms, I would compute it locally because opening an internet connection typically takes longer than that. That would not only help doctests, but also actual users.

Hi Jeroen,

it very much depends on the group. The cohomology of group 299 of order 256 couldn't be computed in one month.

But 150 ms for a computation from scratch is probably impossible for any group.

One should also consider what cohomology rings can really be found in the web database, and currently the smallest groups are those of order 128 (except for the dihedral group of order 8, but that's in the local database, too). Also, it is only prime power order.

From that point of view, it really makes no sense to access the web for something that isn't prime power order and is less than order 128.

Last edited 20 months ago by SimonKing (previous) (diff)

comment:417 in reply to: ↑ 414 ; follow-up: Changed 20 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

  1. It should be possible to declare the optional data sources in the cohomology ring constructor

This sounds strange to me, a single global option makes more sense. Is there any reason why you would want to use different data sources for different rings?

It was a suggestion that I recently got on sage-devel. And it makes sense because a computation can very well simultaneously involve a group of order 128 (to be found in the web) and order 64 (to be found in the public database) and of order 32 (computed locally). So, they already are different sources used simultaneously. So, the only change would be that theoretically you could use several web sources where currently you can use only one.

It wouldn't make the logic of the ring constructor more complicated.

And last but not least: The location that you'll do computations with is the workspace --- regardless of the original source of the ring. So, the source really doesn't matter for the computations, i.e., there is no need to make the source globally unique.

comment:418 in reply to: ↑ 415 Changed 20 months ago by SimonKing

Replying to jdemeyer:

I would go for for "None of the above" and keep the current implementation. Is there any downside to that?

A unification of the database formats would mean that I'd only need a single function to load a ring where I currently need two. But that's all. And I pointed out that a unification of formats would have its downsides, too (bandwidth in web resp. time to unpack stuff locally). So, I'll keep that.

To be clear: That point is orthogonal to whether or not to allow multiple web sources simultaneously.

comment:419 in reply to: ↑ 417 ; follow-up: Changed 20 months ago by jdemeyer

Replying to SimonKing:

And it makes sense because a computation can very well simultaneously involve a group of order 128 (to be found in the web) and order 64 (to be found in the public database) and of order 32 (computed locally). So, they already are different sources used simultaneously.

But weren't you talking about allowing to specify a tuple of sources? In that case, this example uses a fixed tuple of sources, namely (local database, public database, web database).

comment:420 in reply to: ↑ 419 Changed 20 months ago by SimonKing

Replying to jdemeyer:

But weren't you talking about allowing to specify a tuple of sources? In that case, this example uses a fixed tuple of sources, namely (local database, public database, web database).

Right, it would be a fixed tuple of sources, and the program would search them in the order given by the tuple.

Hm. On the one hand, it would be easy to allow overriding the global tuple of sources by an optional argument. On the other hand, personally I would probably never use that, except in doctests.

So, better keep it simple, i.e., with just a global option.

comment:421 Changed 20 months ago by jdemeyer

Let's say that there should at least be a global option to specify the sources. I let you decide whether you want an additional argument when creating the ring.

comment:422 Changed 20 months ago by SimonKing

I am basically done with the changes and will soon post an update of the package. I have yet to run the tests with an ethernet connection in my office (recall that this used to be a problem), but will post the new package before that.

  • I decided to keep a single local source (formerly known as "public db"), which on the one hand is due to details in old pickles and which on the other hand is reasonable: I think no sane user will create additional local databases (i.e., in addition to the one in SAGE_SHARE/pGroupCohomolog/).
  • What was formerly known as "private db" became "workspace" --- I guess that's more descriptive.
  • The remote sources are defined as a tuple of URLs. In particular, if that tuple is empty, there will be no web access.
  • There is a new function doctest_setup() which clears the cache, puts the workspace into a temporary directory and defines "remote sources" as empty tuple.

The above is all about global options. I decided to keep the possibility to define a websource for a single cohomology ring, temporarily overriding the global setting.

  • The file 8gp3.tar.gz, that is used to provide a local version of a web repository, will not be installed. Instead, it is shipped in a folder test_data in the package.
  • I have fixed the bug on reading data for an "empty" cohomology ring (i.e., no generators, no relations). The problem was that I tried to create an empty ideal in Singular, which failed as there was no basering. I also added a test for that fix.

(MINOR) PROBLEM:

There is the following warning:

[p_group_cohomology-3.0] Warning: This package has a badly-behaved setup.py which outputs
[p_group_cohomology-3.0] more than the package name for 'setup.py --name'; using the last

I tried to google for that warning, but couldn't find what's wrong with my setup.py. Can you help me with it?

comment:423 follow-up: Changed 20 months ago by tscrim

That is apparently something sage-specific as part of sage-pip-install. This has only apparently appeared only on one other ticket #22756, but that doesn't seem to have any useful information about the warning message. Might be worthwhile to look at what the precise output of your setup.py.

comment:424 in reply to: ↑ 423 ; follow-up: Changed 20 months ago by SimonKing

Replying to tscrim:

That is apparently something sage-specific as part of sage-pip-install. This has only apparently appeared only on one other ticket #22756, but that doesn't seem to have any useful information about the warning message. Might be worthwhile to look at what the precise output of your setup.py.

$ python setup.py --name
Compiling pGroupCohomology/resolution.pyx because it changed.
Compiling pGroupCohomology/cochain.pyx because it changed.
Compiling pGroupCohomology/cohomology.pyx because it changed.
Compiling pGroupCohomology/modular_cohomology.pyx because it changed.
Compiling pGroupCohomology/dickson.pyx because it changed.
[1/5] Cythonizing pGroupCohomology/cochain.pyx
[2/5] Cythonizing pGroupCohomology/cohomology.pyx
[3/5] Cythonizing pGroupCohomology/dickson.pyx
[4/5] Cythonizing pGroupCohomology/modular_cohomology.pyx
[5/5] Cythonizing pGroupCohomology/resolution.pyx
pGroupCohomology

So, the problem is that the innocent python setup.py --name isn't just returning the name but results in a compilation. Why is that?

comment:425 in reply to: ↑ 424 ; follow-up: Changed 20 months ago by tscrim

Replying to SimonKing:

$ python setup.py --name
Compiling pGroupCohomology/resolution.pyx because it changed.
Compiling pGroupCohomology/cochain.pyx because it changed.
Compiling pGroupCohomology/cohomology.pyx because it changed.
Compiling pGroupCohomology/modular_cohomology.pyx because it changed.
Compiling pGroupCohomology/dickson.pyx because it changed.
[1/5] Cythonizing pGroupCohomology/cochain.pyx
[2/5] Cythonizing pGroupCohomology/cohomology.pyx
[3/5] Cythonizing pGroupCohomology/dickson.pyx
[4/5] Cythonizing pGroupCohomology/modular_cohomology.pyx
[5/5] Cythonizing pGroupCohomology/resolution.pyx
pGroupCohomology

So, the problem is that the innocent python setup.py --name isn't just returning the name but results in a compilation. Why is that?

I have no idea as I have never dealt with these things. Jeroen?

comment:426 Changed 20 months ago by git

  • Commit changed from 441ee4250588968bd122c8ff54989e91c75344f8 to 6a65dec4648d10bbeed67d876a4f8890e7259b73

Branch pushed to git repo; I updated commit sha1. New commits:

cdfe056incorporate one function from sagenb
039799badding a doctest
da202c8simplify the completion method
9afe56fMerge branch 'u/chapoton/24998' in 8.2.rc1
dc2e6f0trac 24998 removing ignored argument
1277686Merge branch 't/24998/24998' into p_group_cohomology-3.0
72671adMerge branch 'develop' into p_group_cohomology-3.0
6a65decp_group_cohomology-3.0: Fix unpickling the empty cohomology ring; cleaner doctest setup

comment:427 Changed 20 months ago by SimonKing

  • Status changed from needs_work to needs_review

I have updated the documentation at http://users.minet.uni-jena.de/cohomology/documentation/ and the source tarball at http://users.minet.uni-jena.de/cohomology/p_group_cohomology-3.0.tar.xz

The with the latest changes, the doctests pass in a reasonable time:

[p_group_cohomology-3.0] Doctesting 12 files using 3 threads.
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/cochain.pxd
[p_group_cohomology-3.0]     [0 tests, 0.00 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/modular_cohomology.pyx
[p_group_cohomology-3.0]     [450 tests, 460.34 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/cochain.pyx
[p_group_cohomology-3.0]     [1459 tests, 525.24 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/factory.py
[p_group_cohomology-3.0]     [216 tests, 80.46 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/cohomology.pyx
[p_group_cohomology-3.0]     [1371 tests, 606.90 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/resolution.pyx
[p_group_cohomology-3.0]     [892 tests, 18.69 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/resolution.pxd
[p_group_cohomology-3.0]     [0 tests, 0.00 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/barcode.py
[p_group_cohomology-3.0]     [143 tests, 23.43 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/dickson.pyx
[p_group_cohomology-3.0]     [28 tests, 2.07 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/resolution_bindings.pxd
[p_group_cohomology-3.0]     [0 tests, 0.00 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/auxiliaries.py
[p_group_cohomology-3.0]     [69 tests, 3.29 s]
[p_group_cohomology-3.0] sage -t --long pyxsources/pGroupCohomology/__init__.py
[p_group_cohomology-3.0]     [349 tests, 189.75 s]
[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] All tests passed!
[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] Total time for all tests: 650.5 seconds
[p_group_cohomology-3.0]     cpu time: 965.3 seconds
[p_group_cohomology-3.0]     cumulative wall time: 1910.2 seconds
[p_group_cohomology-3.0] 
[p_group_cohomology-3.0] real	10m57.949s
[p_group_cohomology-3.0] user	16m44.248s
[p_group_cohomology-3.0] sys	2m30.648s
[p_group_cohomology-3.0] Successfully installed p_group_cohomology-3.0

I have yet to test for the "web access problem", that I only encountered in my office. But I am confident that the remote sources are set to the empty tuple in all tests that might possibly access the internet.

So, for now it is "needs review"!

comment:428 Changed 20 months ago by git

  • Commit changed from 6a65dec4648d10bbeed67d876a4f8890e7259b73 to 1cf4d6c5a812af021ae26f88f36592481191a383

Branch pushed to git repo; I updated commit sha1. New commits:

1cf4d6cp_group_cohomology-3.0: Some fixes

comment:429 Changed 20 months ago by SimonKing

In the latest changes, I fix some typos in the documentation, change some method names to something more descriptive (from_workspace, from_local_sources, from_remote_sources), and fix from_workspace so that it behaves according to what one would expect, namely: The ring is taken from the workspace if it is there and is computed from scratch if it isn't, but certainly it is not copied resp. downloaded from other sources.

And tests pass, so, it remains "needs review".

comment:430 in reply to: ↑ 425 Changed 20 months ago by jdemeyer

Replying to tscrim:

I have no idea as I have never dealt with these things. Jeroen?

Personally, I would just ignore that warning for now. There is no official proper way to package Cython projects.

comment:431 Changed 20 months ago by jhpalmieri

As with earlier versions, if I take a vanilla copy of Sage and do ./sage -i -c p_group_cohomology, I get test failures because of

Exception raised:
   [snip]
   import sage.libs.meataxe
ImportError: No module named meataxe

I don't care about this. Note that when I ran ./sage -i -c ..., meataxe was installed and then the Sage library was built, so I don't know why the meataxe extension wasn't built then.

After running ./sage -b, all tests pass, and they do so pretty quickly (11 minutes on a slowish computer).

Last edited 20 months ago by jhpalmieri (previous) (diff)

comment:432 follow-up: Changed 20 months ago by tscrim

That is a general Sage issue: we do not compile optional extensions after /sage -i.

comment:433 in reply to: ↑ 432 ; follow-up: Changed 20 months ago by jhpalmieri

Replying to tscrim:

That is a general Sage issue: we do not compile optional extensions after /sage -i.

Making $(SAGERUNTIME) an order-only dependency was supposed to address this (see the earlier discussion here).

comment:434 Changed 20 months ago by tscrim

Right...hmm...IDK. I don't have my laptop in front of me to check in more detail. I will do so later today.

comment:435 in reply to: ↑ 433 ; follow-up: Changed 20 months ago by jdemeyer

Replying to jhpalmieri:

Making $(SAGERUNTIME) an order-only dependency was supposed to address this (see the earlier discussion here).

No, that was to address a different issue.

I don't see a solution for the sage -b problem, but suggestions are welcome...

Last edited 20 months ago by jdemeyer (previous) (diff)

comment:436 in reply to: ↑ 435 Changed 20 months ago by SimonKing

Replying to jdemeyer:

Replying to jhpalmieri:

Making $(SAGERUNTIME) an order-only dependency was supposed to address this (see the earlier discussion here).

No, that was to address a different issue.

I don't see a solution for the sage -b problem, but suggestions are welcome...

I have a suggestion for the problem here (but it wouldn't solve the general problem): p_group_cohomology's spkg-install should check whether sage.libs.meataxe can be imported. If it can't then either the installation should be aborted with a message advising the user to do sage -b, or spkg-install should run sage -b automatically.

comment:437 follow-up: Changed 20 months ago by jhpalmieri

The problem is in running the tests, so would it make more sense for p_group_cohomology's spkg-check to test whether meataxe can be imported?

comment:438 Changed 20 months ago by jhpalmieri

In any case, I'm not sure that this is an obstacle to giving a positive review here.

comment:439 in reply to: ↑ 437 Changed 20 months ago by SimonKing

Replying to jhpalmieri:

The problem is in running the tests, so would it make more sense for p_group_cohomology's spkg-check to test whether meataxe can be imported?

The problem is that without sage -b, even the simplest operations in the package won't work. So, if sage -b was forgotten then the package isn't properly installed. That's why I'd be in favour of solving it in spkg-install. Note that not all users will run the tests, and for them a solution involving spkg-check won't be a solution.

comment:440 follow-ups: Changed 20 months ago by jhpalmieri

If there are no good solutions, I am willing to give this a positive review. (Although I like the idea of testing to see if sage.libs.meataxe can be imported at the end of spkg-install and printing a helpful message if not.)

comment:441 in reply to: ↑ 440 ; follow-up: Changed 20 months ago by SimonKing

Replying to jhpalmieri:

If there are no good solutions, I am willing to give this a positive review. (Although I like the idea of testing to see if sage.libs.meataxe can be imported at the end of spkg-install and printing a helpful message if not.)

Do I understand correctly that you only needed to do sage -b and did not need to re-install the package in order to make it work?

In that case, I think a warning should be given at the end of spkg-install and an error schould be raised at the beginning of spkg-check, if sage.libs.meataxe cannot be imported.

But if a re-installation of the package is needed after sage -b, then I'd prefer an error being raised at the beginning of spkg-install.

comment:442 in reply to: ↑ 440 Changed 20 months ago by jdemeyer

Replying to jhpalmieri:

printing a helpful message if not

Unfortunately, my experience is that users typically don't read such helpful messages.

comment:443 in reply to: ↑ 441 ; follow-up: Changed 20 months ago by jdemeyer

Replying to SimonKing:

Do I understand correctly that you only needed to do sage -b and did not need to re-install the package in order to make it work?

Which "the package"?

You need to build the Sage library, that's all.

comment:444 in reply to: ↑ 443 Changed 20 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Do I understand correctly that you only needed to do sage -b and did not need to re-install the package in order to make it work?

Which "the package"?

p_group_cohomology

You need to build the Sage library, that's all.

So, you install meataxe, forget to do sage -b, install p_group_cohomology, then finally do sage -b, and all is good?

You said that helpful messages are often not read. In that case, I'd suggest that at the end of p_group_cohomology's spkg-install it is checked whether sage.libs.meataxe can be imported, and if it can't then sage -b is done automatically.

comment:445 Changed 20 months ago by git

  • Commit changed from 1cf4d6c5a812af021ae26f88f36592481191a383 to 4d13b9e9257f58dafaf475c627f5f6a19aa0342b

Branch pushed to git repo; I updated commit sha1. New commits:

4d13b9ep_group_cohomology-3.0: Do sage -b when meataxe can't be imported

comment:446 follow-up: Changed 20 months ago by SimonKing

Is my change to spkg-install acceptable? I cannot test it at the moment, as I am running a computation that I wouldn't like to interrupt (it will fun for a week or so).

PS: I'm doing the test for proper installation of meataxe the beginning of spkg-install.

Last edited 20 months ago by SimonKing (previous) (diff)

comment:447 in reply to: ↑ 446 ; follow-up: Changed 20 months ago by jdemeyer

  • Milestone changed from sage-8.2 to sage-8.3
  • Status changed from needs_review to needs_work

Replying to SimonKing:

Is my change to spkg-install acceptable?

No, this was already discussed in 324

comment:448 Changed 20 months ago by git

  • Commit changed from 4d13b9e9257f58dafaf475c627f5f6a19aa0342b to 843545fffbc89961d90890b5131ea46b1a8aaa23

Branch pushed to git repo; I updated commit sha1. New commits:

843545fp_group_cohomology-3.0: Do not do sage -b, but let the user do it

comment:449 in reply to: ↑ 447 Changed 20 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Is my change to spkg-install acceptable?

No, this was already discussed in 324

OK, I hope the next version is better: Now installation simply aborts, and asks the user to do "sage -b" before retrying to install p_group_cohomology.

comment:450 follow-up: Changed 20 months ago by jdemeyer

Can you please suggest make sagelib instead of sage -b? The comand sage -b is really a more low-level command that you should only use if you know what you are doing. I would never recommend it like that.

Last edited 20 months ago by jdemeyer (previous) (diff)

comment:451 Changed 20 months ago by jdemeyer

Also the error message MeatAxe cannot be imported is not really right. It should be the Sage interface to MeatAxe cannot be imported or something like that.

comment:452 Changed 20 months ago by git

  • Commit changed from 843545fffbc89961d90890b5131ea46b1a8aaa23 to a33b1b26a3010495099aa454de097bc2c8f3d9b4

Branch pushed to git repo; I updated commit sha1. New commits:

a33b1b2p_group_cohomology-3.0: Recommend 'make sagelib' instead of 'sage -b'

comment:453 in reply to: ↑ 450 ; follow-up: Changed 20 months ago by SimonKing

Replying to jdemeyer:

Can you please suggest make sagelib instead of sage -b? The comand sage -b is really a more low-level command that you should only use if you know what you are doing. I would never recommend it like that.

Actually I thought sage -b is the recommended thing to do. Is it really better to do make sagelib? I was totally unaware of it. Anyway I changed it.

Replying to jdemeyer:

Also the error message MeatAxe cannot be imported is not really right. It should be the Sage interface to MeatAxe cannot be imported or something like that.

Done. What cannot be imported is sage.libs.meataxe

comment:454 in reply to: ↑ 453 Changed 20 months ago by jdemeyer

Replying to SimonKing:

Actually I thought sage -b is the recommended thing to do.

In this specific case, sage -b does make sense. But I don't want to recommend it because the various make targets like make sagelib or make build should be what people use by default. The reason is that those do dependency checking while sage -b is more low-level command which does not.

comment:455 Changed 20 months ago by jdemeyer

For me personally, the standard command that I use is make build because it's fast (unlike plain make which also builds the doc) and it works well. The only time that I run sage -b is indirectly by running sage -bt ... (combining sage -b and sage -t).

comment:456 follow-up: Changed 20 months ago by jhpalmieri

The error message is not properly formatted: you don't want Do `make sagelib` and retry – the backquotes should just be regular quotes: Do 'make sagelib' and retry. (In fact I just ran ./sage -i p_group_cohomology twice in a row without make sagelib first and the second time it first built sagelib, so it succeeded in building p_group_cohomology.)

comment:457 in reply to: ↑ 456 ; follow-up: Changed 20 months ago by SimonKing

Replying to jhpalmieri:

The error message is not properly formatted: you don't want Do `make sagelib` and retry – the backquotes should just be regular quotes:

I don't see why it is a problem, but I'll change it in a few minutes.

(In fact I just ran ./sage -i p_group_cohomology twice in a row without make sagelib first and the second time it first built sagelib, so it succeeded in building p_group_cohomology.)

How came that? Do you say that the second sage -i p_group_cohomology did automatically do "make sagelib"? It shouldn't!

comment:458 Changed 20 months ago by git

  • Commit changed from a33b1b26a3010495099aa454de097bc2c8f3d9b4 to a38b9bbec1f0c05dba20fef8520f65af563269d9

Branch pushed to git repo; I updated commit sha1. New commits:

a38b9bbp_group_cohomology-3.0: Change backtick into quote

comment:459 in reply to: ↑ 457 Changed 20 months ago by jhpalmieri

Replying to SimonKing:

Replying to jhpalmieri:

The error message is not properly formatted: you don't want Do `make sagelib` and retry – the backquotes should just be regular quotes:

I don't see why it is a problem, but I'll change it in a few minutes.

It's a problem because when the shell prints a string, it tries to evaluate whatever is in backquotes: try echo `ls` compared to echo 'ls'.

(In fact I just ran ./sage -i p_group_cohomology twice in a row without make sagelib first and the second time it first built sagelib, so it succeeded in building p_group_cohomology.)

How came that? Do you say that the second sage -i p_group_cohomology did automatically do "make sagelib"? It shouldn't!

This may be a mistake on my part. I'm trying again (is there a quicker way than make distclean and rebuilding from scratch?).


New commits:

a38b9bbp_group_cohomology-3.0: Change backtick into quote

comment:460 Changed 20 months ago by SimonKing

  • Status changed from needs_work to needs_review
  • Work issues Internet; empty rings; tests alter private database; do not install test data files; use #24998 deleted

Done.

Probably the backtick explains why the second sage -i p_group_cohomology has built sagelib: The backticked make sagelib got evaluated!

comment:461 follow-up: Changed 20 months ago by jdemeyer

Typo: sage.lib.meataxe -> sage.libs.meataxe

comment:462 Changed 20 months ago by git

  • Commit changed from a38b9bbec1f0c05dba20fef8520f65af563269d9 to ca212cdaab90cc21cc85d4268fcc4bc21bd4a923

Branch pushed to git repo; I updated commit sha1. New commits:

ca212cdp_group_cohomology-3.0: Fix a typo in an error message of spkg-install

comment:463 in reply to: ↑ 461 Changed 20 months ago by SimonKing

Replying to jdemeyer:

Typo: sage.lib.meataxe -> sage.libs.meataxe

In the error message, you mean. Fixed!

comment:464 follow-up: Changed 20 months ago by jdemeyer

Something I just realized after reading the recent comments here: you can actually simplify

Do 'make sagelib' and retry installation of p_group_cohomology

to

Retry installation of p_group_cohomology

This will work because the re-installation of p_group_cohomology will trigger a rebuild of the Sage library because of the $(SAGERUNTIME) dependency.

comment:465 follow-up: Changed 20 months ago by jdemeyer

  • Status changed from needs_review to needs_work

comment:466 in reply to: ↑ 465 ; follow-up: Changed 20 months ago by SimonKing

Replying to jdemeyer:

Why did you change it to "needs_work"? I thought that I had addressed all remaining points (considering comment:464 as a non-issue).

comment:467 in reply to: ↑ 464 ; follow-up: Changed 20 months ago by SimonKing

Replying to jdemeyer:

Something I just realized after reading the recent comments here: you can actually simplify

Do 'make sagelib' and retry installation of p_group_cohomology

to

Retry installation of p_group_cohomology

This will work because the re-installation of p_group_cohomology will trigger a rebuild of the Sage library because of the $(SAGERUNTIME) dependency.

Why is that the case? I.e., what is different in the second installation attempt?

comment:468 in reply to: ↑ 466 Changed 20 months ago by jdemeyer

Replying to SimonKing:

considering comment:464 as a non-issue

It seems strange to tell a user "do A and B" when "do B" is sufficient.

comment:469 in reply to: ↑ 467 Changed 20 months ago by jdemeyer

Replying to SimonKing:

what is different in the second installation attempt?

The difference is that meataxe is installed before starting the build of p_group_cohomology.

Maybe I should explain the problem again in full detail: there are 3 relevant packages: meataxe (M), sagelib (S) and p_group_cohomology (P). The fact that S is a special package doesn't really matter much: the build system mostly treats S like any other package. One exception is that it is never considered up-to-date: it will always rebuild S if it occurs as a dependency.

Whenever S is built, it will compile the optional module sage.libs.meataxe provided that M was built before building S. So we need some build of S after the build of M.

There is a dependency of P on both M and S. The problem is that M and S are built in an unspecified order. So the build order could be M, S, P (good) or S, M, P (bad).

If we are in the bad case S, M, P and we build P again, then the full build order will be S, M, P, S, P which works because it contains the good subsequence M, S, P.

comment:470 Changed 20 months ago by git

  • Commit changed from ca212cdaab90cc21cc85d4268fcc4bc21bd4a923 to d77ab39269bf5525c1a8cc154b3adb333d0b61e3

Branch pushed to git repo; I updated commit sha1. New commits:

d77ab39p_group_cohomology-3.0: Simplify build instructions in case of failing import

comment:471 Changed 20 months ago by SimonKing

  • Status changed from needs_work to needs_review

Thank you for the explanation. I have changed the error message accordingly.

comment:472 follow-up: Changed 19 months ago by SimonKing

I am a bit worried: From a discussion on sage-devel, I got the impression that a recently merged ticket broke several optional packages, including database_gap, which also affects p_group_cohomology. So, what's the status? Is it safe to use the current Sage beta release to work with optional packages?

Also: Is there more than #25079 that prevents completion of the review of this ticket?

comment:473 Changed 19 months ago by dimpase

The corresponding (sub)issue is fixed in #25323, and the whole (hopefully) problem in #25332, should be in the coming beta.

comment:474 in reply to: ↑ 472 Changed 19 months ago by jdemeyer

Replying to SimonKing:

Also: Is there more than #25079 that prevents completion of the review of this ticket?

You need to decide what to do about #25106.

comment:475 follow-up: Changed 19 months ago by dimpase

what does this ticket have to do with #25106 ?

Last edited 19 months ago by dimpase (previous) (diff)

comment:476 in reply to: ↑ 475 Changed 19 months ago by SimonKing

Replying to dimpase:

what does this ticket have to do with #25106 ?

Nothing whatsoever, I believe.

comment:477 in reply to: ↑ 364 Changed 19 months ago by SimonKing

  • Dependencies changed from #25079, #25106, #24998 to #25079, #24998

Replying to SimonKing:

ANd has matplotlib.font_manager (previous comment) really something to do with this ticket?? If not, please remove the additional dependency.

Since nobody else did, I do. I really don't see why #25106 is in the dependency list.

comment:478 Changed 19 months ago by jdemeyer

  • Status changed from needs_review to needs_work

comment:479 Changed 19 months ago by jdemeyer

You could at least have read 362 which added the dependency and ticket #25106 which explains the problem.

comment:480 follow-up: Changed 19 months ago by jdemeyer

Note that #25106 doesn't need to be dependency but then 362 needs to be fixed in a different way.

comment:481 in reply to: ↑ 480 ; follow-ups: Changed 19 months ago by SimonKing

Replying to jdemeyer:

Note that #25106 doesn't need to be dependency but then 362 needs to be fixed in a different way.

In 362 you did in fact NOT explain the problem. You just showed that something went wrong, but didn't give a hint how to reproduce it.

And since you didn't tell, I can only guess: You tried to install the p_group_cohomology package before you completed building Sage. But that's not what one is supposed to do in the case of an optional package: First, one has to have a FULLY FUNCTIONAL Sage installation (i.e., with all standard packages built and installed) and only then should one start with installation of optional packages.

And if Sage is fully functional, matplotlib is available (it is standard package) and the doctesting framework is available too.

comment:482 follow-up: Changed 19 months ago by jhpalmieri

Since #25106 has a positive review, what difference does it make now? Can we move on?

comment:483 in reply to: ↑ 482 Changed 19 months ago by SimonKing

Replying to jhpalmieri:

Since #25106 has a positive review, what difference does it make now? Can we move on?

+1

I'll take care of #25079 soon.

comment:484 in reply to: ↑ 481 Changed 19 months ago by jdemeyer

  • Dependencies changed from #25079, #24998 to #25079, #24998, #25106
  • Status changed from needs_work to needs_review

Replying to SimonKing:

You tried to install the p_group_cohomology package before you completed building Sage.

Indeed and there is (despite what you think) nothing wrong with that.

comment:485 in reply to: ↑ 481 ; follow-up: Changed 19 months ago by jdemeyer

Replying to SimonKing:

In 362 you did in fact NOT explain the problem.

That is true but I linked to #25106 where I did explain. Maybe I explained badly, but I certainly tried to be helpful (even replying to your comments on that ticket).

comment:486 in reply to: ↑ 485 ; follow-ups: Changed 19 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

In 362 you did in fact NOT explain the problem.

That is true but I linked to #25106 where I did explain. Maybe I explained badly, but I certainly tried to be helpful (even replying to your comments on that ticket).

Thank you. The missing bit of information for me was indeed that optional packages are supposed to be buildable before Sage is fully functional (as long as the package's explicitly listed dependencies are available, of course).

Anyway, since #25106 has positive review, I suppose we can consider that issue as being fixed, right?

Do I understand correctly that I do not need to post here a merge commit with #25106?

comment:487 in reply to: ↑ 486 ; follow-up: Changed 19 months ago by jdemeyer

Replying to SimonKing:

Anyway, since #25106 has positive review, I suppose we can consider that issue as being fixed, right?

Indeed.

Do I understand correctly that I do not need to post here a merge commit with #25106?

Indeed. Nothing needs to change, just keep the dependency.

Apologies if my posts from yesterday were rude, I was a bit frustrated by the sudden removal of that dependency.

comment:488 in reply to: ↑ 486 Changed 19 months ago by jdemeyer

Replying to SimonKing:

The missing bit of information for me was indeed that optional packages are supposed to be buildable before Sage is fully functional (as long as the package's explicitly listed dependencies are available, of course).

You really should think of "standard" packages and "optional" as being exactly the same, except that optional packages are not installed by default. In particular, they follow the same rules about dependencies.

This rule makes sense because it makes it easy to change optional packages to standard (in theory at least).

comment:489 in reply to: ↑ 487 Changed 19 months ago by SimonKing

Replying to jdemeyer:

Apologies if my posts from yesterday were rude, I was a bit frustrated by the sudden removal of that dependency.

In fact I am sorry because my own posts have been too rude. My frustration: There have been too many occasions where the old "official" package versions or the "inofficial" new releases of the package broke because of changes in Sage that were more or less trivial but nonetheless made it necessary to change the package, with each change involving a lengthy peer review process (this ticket is 3 years old) during which further trivial changes in Sage led to further necessary adaptions of the package. And I thought we would have yet another iteration for it, under circumstances that from my biased perspective are close to "wrong usage".

comment:490 Changed 19 months ago by jhpalmieri

So one problem (solved by #25106) is that, although $(SAGERUNTIME) is a dependency for this package, $(SAGERUNTIME) does not require matplotlib, so it is possible that the Sage library will build before matplotlib. And pre #25106, the doctesting framework required matplotlib, leading to problems when testing the p-group cohomology package before completing a top-level make command. Do I have all that right?

comment:491 follow-up: Changed 19 months ago by SimonKing

I just tested that both #25079 and #25490 cleanly merge with the current branch from here. So, I see no need to post a merge commit here. Do you?

Last edited 19 months ago by SimonKing (previous) (diff)

comment:492 in reply to: ↑ 491 Changed 19 months ago by SimonKing

Replying to SimonKing:

I just tested that both #25079 and #25490 cleanly merge with the current branch from here. So, I see no need to post a merge commit here. Do you?

PS: Not only is there no merge conflict. Also there is no error in the package's test suite.

comment:493 Changed 19 months ago by SimonKing

  • Dependencies changed from #25079, #24998, #25106 to #25079, #24998, #25106, #25511

With #25511, it will certainly be needed to change p_group_cohomology again.

In its current shape, the package can be reviewed. But unless a quick review is possible, I'd provide a version 3.1 that incorporates the to-be-provided changes in #25511.

comment:494 follow-up: Changed 19 months ago by jdemeyer

Simon: do you mind if I rewrite git history on this ticket and its dependencies? The dependencies are becoming a mess with various tickets depending on each other. I'd like to make the dependency graph linear:

#25476 -> #25079 -> #25511 -> #18514

comment:495 in reply to: ↑ 494 ; follow-up: Changed 19 months ago by SimonKing

Replying to jdemeyer:

Simon: do you mind if I rewrite git history on this ticket and its dependencies?

Why would it be needed to rewrite the git history? I thought "dependency of a ticket" means that there is no need to post an explicit merge commit, as long as the branches from the two tickets merge cleanly.

The dependencies are becoming a mess with various tickets depending on each other. I'd like to make the dependency graph linear:

#25476 -> #25079 -> #25511 -> #18514

Is that currently not the case? Here, when we omit the already-merged tickets #24998 and #25106, we have dependency #25511, which currently depends on #25079, which depends on #25476.

So, it should be fine to state on #25511 that the ONLY dependency is #25079 and to state here that the ONLY dependency is #25511. But I don't see what that has to do with changing git history.

Last edited 19 months ago by SimonKing (previous) (diff)

comment:496 in reply to: ↑ 495 ; follow-up: Changed 19 months ago by jdemeyer

Replying to SimonKing:

Why would it be needed to rewrite the git history?

Just to make things simpler.

I thought "dependency of a ticket" means that there is no need to post an explicit merge commit, as long as the branches from the two tickets merge cleanly.

Right, it is not strictly required. But then you need to do those merges anyway whenever working on that ticket or a dependency of it.

comment:497 follow-up: Changed 19 months ago by jdemeyer

I just saw this commit here:

  • src/sage/matrix/matrix_gfpn_dense.pyx

    diff --git a/src/sage/matrix/matrix_gfpn_dense.pyx b/src/sage/matrix/matrix_gfpn_dense.pyx
    index e8490e5..25a4d6c 100644
    a b cdef class Matrix_gfpn_dense(Matrix_dense): 
    390390            self.Data = MatLoad(filename)
    391391            FfSetField(self.Data.Field)
    392392            B = GF(self.Data.Field, 'z')
    393             parent = MatrixSpace(B, self.Data.Nor, self.Data.Noc)
     393            parent = MatrixSpace(B, self.Data.Nor, self.Data.Noc, implementation=Matrix_gfpn_dense)
    394394            self._is_immutable = False
    395395            self._parent = parent
    396396            self._base_ring = B

I guess that should be moved to #25511.

comment:498 in reply to: ↑ 496 ; follow-up: Changed 19 months ago by SimonKing

Replying to jdemeyer:

Right, it is not strictly required. But then you need to do those merges anyway whenever working on that ticket or a dependency of it.

Then I'd say: Go ahead. I.e., rebase this ticket on top of #25511, making it the only dependency of this ticket. But then please also tell me how to smoothly pull from the git branch of this ticket after a forced push: Is there a "forced pull" command?

comment:499 in reply to: ↑ 498 Changed 19 months ago by jdemeyer

Replying to SimonKing:

But then please also tell me how to smoothly pull from the git branch of this ticket after a forced push: Is there a "forced pull" command?

I typically do as follows:

$ git fetch
$ git trac checkout 18514
$ git reset --hard COMMIT_HASH_ON_TICKET

where COMMIT_HASH_ON_TICKET is the value of the "Commit" field on Trac.

Last edited 19 months ago by jdemeyer (previous) (diff)

comment:500 in reply to: ↑ 497 Changed 19 months ago by SimonKing

Replying to jdemeyer:

I just saw this commit here: ... I guess that should be moved to #25511.

Indeed.

comment:501 Changed 19 months ago by jdemeyer

There is also a conflict of sorts with #25490 because both add the MPB field.

comment:502 Changed 19 months ago by jdemeyer

  • Branch changed from u/SimonKing/upgrade_of_group_cohomology_spkg to u/jdemeyer/upgrade_of_group_cohomology_spkg

comment:503 Changed 19 months ago by jdemeyer

  • Commit changed from d77ab39269bf5525c1a8cc154b3adb333d0b61e3 to 3959b5049e299282708950770b8741a2a4b9b675
  • Dependencies changed from #25079, #24998, #25106, #25511 to #25511
  • Status changed from needs_review to needs_work

New commits:

4eae803Making matrices use the new _echelon_in_place method.
e2f0550Specify that _echelon_in_place shall return the pivots
3c4a06dEnable _mul_long for matrices
1eaed37More stuff in the meataxe interface, and a meataxe helper function
8a06c0fFix docstring formatting
13da208Clean up creating Matrix_gfpn_dense matrices
3959b50Add p_group_cohomology-3.0 spkg

comment:504 Changed 19 months ago by jdemeyer

Rebased everything and added an intermediate ticket #25518. The dependency chain is now

#25476 -> #25079 -> #25518 -> #25511 -> #18514

I'm done for now. Simon, feel free to continue working on this.

comment:505 follow-up: Changed 18 months ago by SimonKing

I now try to rework the package on top of #25511. One question arose: Why does the following import statement in resolution.pyx does not work:

from sage.cpython.string cimport str_to_bytes
from sage.cpython.string import FS_ENCODING

?

It is the same import statement as in sage.matrix.matrix_gfpn_dense - so, it SHOULD work, shouldn't it?

EDIT: This is the error I get when compiling it:

[1/1] Cythonizing pGroupCohomology/resolution.pyx

Error compiling Cython file:
------------------------------------------------------------
...
    # Missing from cpython.unicode in Cython 0.27.3
    char* PyUnicode_AsUTF8(object s)


cdef inline str char_to_str(const char* c, encoding=None, errors=None):
    IF PY_MAJOR_VERSION <= 2:
      ^
------------------------------------------------------------

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/cpython/string.pxd:25:7: Compile-time name 'PY_MAJOR_VERSION' not defined

Error compiling Cython file:
------------------------------------------------------------
...
        TypeError: expected bytes, list found
    """
    if not isinstance(b, bytes):
        raise TypeError(f"expected bytes, {type(b).__name__} found")

    IF PY_MAJOR_VERSION <= 2:
      ^
------------------------------------------------------------

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/cpython/string.pxd:70:7: Compile-time name 'PY_MAJOR_VERSION' not defined

Error compiling Cython file:
------------------------------------------------------------
...
        TypeError: expected str ... list found
    """
    cdef const char* err
    cdef const char* enc

    IF PY_MAJOR_VERSION <= 2:
      ^
------------------------------------------------------------

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/cpython/string.pxd:106:7: Compile-time name 'PY_MAJOR_VERSION' not defined


Error compiling Cython file:
------------------------------------------------------------
...
from pGroupCohomology.cochain cimport COCH

from libc.string cimport memcpy
from cysignals.memory cimport sig_free
from cysignals.signals cimport sig_check, sig_on, sig_off
from sage.cpython.string cimport str_to_bytes
^
------------------------------------------------------------

pGroupCohomology/resolution.pyx:58:0: 'sage/cpython/string/str_to_bytes.pxd' not found
Last edited 18 months ago by SimonKing (previous) (diff)

comment:506 Changed 18 months ago by SimonKing

I also get this very strange attribute error (unfortunately without a good traceback):

...
pGroupCohomology/cochain.pyx in pGroupCohomology.cochain.COCH._mul_()

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/structure/element.pyx in sage.structure.element.Element.__getattr__ (build/cythonized/sage/structure/element.c:4519)()
    491             AttributeError: 'LeftZeroSemigroup_with_category.element_class' object has no attribute 'blah_blah'
    492         """
--> 493         return self.getattr_from_category(name)
    494 
    495     cdef getattr_from_category(self, name):

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/structure/element.pyx in sage.structure.element.Element.getattr_from_category (build/cythonized/sage/structure/element.c:4628)()
    504         else:
    505             cls = P._abstract_element_class
--> 506         return getattr_from_other_class(self, cls, name)
    507 
    508     def __dir__(self):

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/cpython/getattr.pyx in sage.cpython.getattr.getattr_from_other_class (build/cythonized/sage/cpython/getattr.c:2536)()
    392         dummy_error_message.cls = type(self)
    393         dummy_error_message.name = name
--> 394         raise AttributeError(dummy_error_message)
    395     attribute = <object>attr
    396     # Check for a descriptor (__get__ in Python)

AttributeError: 'Singular' object has no attribute '_strip_prompt'

But '_strip_prompt' appears nowhere in the cohomology package! So, what the heck is going on here?

comment:507 Changed 18 months ago by SimonKing

I don't understand why Singular gets involved, but the error in the previous comment occurs when trying to call Matrix_gfpn_dense._new, which apparently got removed in #25511. So, I now know how to work around. But the compiler error in comment:505 still needs explanation.

comment:508 in reply to: ↑ 505 Changed 18 months ago by jdemeyer

Replying to SimonKing:

It is the same import statement as in sage.matrix.matrix_gfpn_dense - so, it SHOULD work, shouldn't it?

I guess the difference is between compiling in Sage and compiling externally. I certainly do consider it a bug.

comment:509 follow-up: Changed 18 months ago by jdemeyer

I opened #25549 for that.

comment:510 in reply to: ↑ 509 ; follow-ups: Changed 18 months ago by SimonKing

Replying to jdemeyer:

I opened #25549 for that.

Thank you! For now, I am able to work around that problem, because at the moment it still works to pass a str to MatLoad, but I guess if #25549 gets fixed soon I should use it.

A related question: Can you point me to an explanation on how to deal with strings in a "Python-3-approved" way? I still do if isinstance(...,basestring):, cdef str FOO, and even return <str>PyBytes_FromString(...), but I guess that should change.

EDIT: In fact, return <str>PyBytes_FromString(...) wasn't I. That line is in sage.cpython.string.

Last edited 18 months ago by SimonKing (previous) (diff)

comment:511 in reply to: ↑ 510 ; follow-up: Changed 18 months ago by jdemeyer

Replying to SimonKing:

Can you point me to an explanation on how to deal with strings in a "Python-3-approved" way?

That's a very general question with no easy answer. It all depends on what you want to do precisely...

comment:512 in reply to: ↑ 511 ; follow-ups: Changed 18 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Can you point me to an explanation on how to deal with strings in a "Python-3-approved" way?

That's a very general question with no easy answer. It all depends on what you want to do precisely...

Well, my understanding is that in Python 3, bytes takes the role of what in Python 2 is str. And a string in Python 3 always comes with an encoding.

In fact, my questions ARE very general and unprecise, as I heard that there ARE differences between Python 2 and 3 regarding strings, but I do not have the faintest idea how they differ and what it means for coding.

  • How to test whether a function argument is a string? I recall that isinstance(FOO,basestring) is depreciated.
  • Is there a change when talking with a C library function? I.e., currently a Python str is converted into a char*; is there any pitfall in Python 3 (such as "one is supposed to use a proper string with encoding in Python code, but when talking with C one needs to convert to bytes"?
  • Example: What does
            if type(filename) is not bytes:
                filename = str_to_bytes(filename, FS_ENCODING,
                                        'surrogateescape')
    
    do that if not isinstance(filename, basestring): <do some conversion> doesn't

comment:513 in reply to: ↑ 512 Changed 18 months ago by jdemeyer

First of all, the answers are different between Python and Cython. Since you're mostly dealing with Cython code, I'll give the Cython answer.

There are 2 classes which are unambiguous: bytes and unicode. Here, bytes is a sequence of bytes (corresponding to char * in C) and unicode is a sequence of unicode characters. You can also think of unicode as an abstract string, while bytes is an encoded string (say, with UTF-8 encoding).

There is also the str class, which is an alias for bytes on Python 2 but an alias of unicode on Python 3. But it's best to think of bytes, str and unicode as three distinct classes (that's more or less what Cython does internally)

isinstance(foo, basestring) matches str and unicode (so it matches bytes on Python 2 but not on Python 3).

For literals, the following rules apply:

  • b"foo" -> bytes
  • "foo" -> str
  • u"foo" -> unicode

Replying to SimonKing:

  • How to test whether a function argument is a string?

Depends what you mean with "a string".

  • Is there a change when talking with a C library function?

Absolutely, I would say that this is the main reason of existence of sage.cpython.string. C functions take a char * which maps to bytes. But the Python wrappers typically take a str as argument. So there needs to be a conversion between str and bytes (which is a no-op in Python 2).

comment:514 in reply to: ↑ 512 Changed 18 months ago by jdemeyer

Replying to SimonKing:

        if type(filename) is not bytes:
            filename = str_to_bytes(filename, FS_ENCODING,
                                    'surrogateescape')

Let me just explain what this does and why it's the right thing to do. For filenames, many Python functions accept both bytes and str. So it's natural to allow this too in Sage.

If filename has type bytes, then nothing needs to be done because C functions can accept that (Cython does the conversion bytes -> char * for you). If not, str_to_bytes is called which converts either str or unicode to bytes (it raises an error for other input). So that piece of code accepts bytes, str and unicode on all Python versions. The result is that filename is of type bytes.

The arguments FS_ENCODING, 'surrogateescape' are what Python uses by default for converting filenames.

The only thing that slightly puzzles me is why the check is type(filename) is not bytes instead of not isinstance(filename, bytes) (i.e. allowing subclasses of bytes). Maybe custom subclasses of bytes cannot easily be converted to char *?

comment:515 Changed 18 months ago by SimonKing

  • Branch changed from u/jdemeyer/upgrade_of_group_cohomology_spkg to u/SimonKing/upgrade_of_group_cohomology_spkg

comment:516 follow-ups: Changed 18 months ago by SimonKing

  • Commit changed from 3959b5049e299282708950770b8741a2a4b9b675 to 0631029b380acdc82ac195d553b42b5c2bfa42d4
  • Status changed from needs_work to needs_review

I have reworked upstream so that the recent additions from #25511 are used. The advantage is that this ticket's branch only adds changes to build/pkgs, but not to the Sage library; these changes are in the only new commit of the branch that I just force-pushed.

Of course, I have updated the .tar.xz-file at the location given in the ticket description.

Result:

[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] All tests passed!
[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] Total time for all tests: 541.7 seconds
[p_group_cohomology-3.0]     cpu time: 883.3 seconds
[p_group_cohomology-3.0]     cumulative wall time: 1575.8 seconds
[p_group_cohomology-3.0] 
[p_group_cohomology-3.0] real	9m4.813s
[p_group_cohomology-3.0] user	15m25.364s
[p_group_cohomology-3.0] sys	2m18.384s
[p_group_cohomology-3.0] Successfully installed p_group_cohomology-3.0
[p_group_cohomology-3.0] Deleting temporary build directory
[p_group_cohomology-3.0] /home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.0
[p_group_cohomology-3.0] Finished installing p_group_cohomology-3.0.spkg
make[1]: Leaving directory '/home/king/Sage/git/sage/build/make'

real	17m7.539s
user	22m55.884s
sys	2m23.596s
Sage build/upgrade complete!

So, ready for review!

Strangely, only the branch field of this ticket was automatically updated, but not the "commit" field. So, I changed it manually.

comment:517 in reply to: ↑ 516 Changed 18 months ago by jdemeyer

Replying to SimonKing:

Strangely, only the branch field of this ticket was automatically updated, but not the "commit" field.

https://github.com/sagemath/sage_trac_plugin/issues/10

comment:518 in reply to: ↑ 516 ; follow-up: Changed 18 months ago by jhpalmieri

Replying to SimonKing:

Of course, I have updated the .tar.xz-file at the location given in the ticket description.

I am having problems downloading the file: I get

Fehler 403
Zugriff auf /cohomology/p_group_cohomology-3.0.tar.xz verweigert.

Wenn Sie dies für einen Fehler in der Serverkonfiguration halten, so senden Sie bitte eine Nachricht an den Webmaster oder an die Autorin der Seite, über die Sie hierher gelangt sind. Bitte fügen Sie Ihrer Nachricht die folgenden Zeilen an:

Wed Jun 13 19:47:02 2018
(75.172.196.95)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0

Error 403
You don't have permission to access /cohomology/p_group_cohomology-3.0.tar.xz on this server.

If you feel that this is a configuration error, please contact the administrators or the author of the referring page. Please include the following lines in your message:

Wed Jun 13 19:47:02 2018
(75.172.196.95)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0 

comment:519 in reply to: ↑ 518 Changed 18 months ago by SimonKing

  • Description modified (diff)

Replying to jhpalmieri:

Replying to SimonKing:

Of course, I have updated the .tar.xz-file at the location given in the ticket description.

I am having problems downloading the file:

Thanks for spotting it. I have informed our administrator.

Temporarily, I have posted it at my homepage (https://users.fmi.uni-jena.de/~king/p_group_cohomology-3.0.tar.xz), but that will not be permanent, as I have a data limit there.

And can PLEASE finally someone fix trac, so that the browser isn't insanely jumping to odd places (for example it was just centering at comment:223 while I tried to write the new comment)

comment:520 Changed 18 months ago by jhpalmieri

The checksums didn't work for me, but once I fixed those, all tests in the test suite passed. Is there anything else that needs to be done here?

comment:521 Changed 18 months ago by git

  • Commit changed from 0631029b380acdc82ac195d553b42b5c2bfa42d4 to a330d39c12ea918970ca3ac9678e02e3441dd760

Branch pushed to git repo; I updated commit sha1. New commits:

a330d39p_group_cohomology-3.0: Fix checksum

comment:522 Changed 18 months ago by SimonKing

Sorry, I forgot to push the checksum update. So, here it is...

What *could* be reasonable is to check the package on different platforms. But I guess Jeroen has a more elaborate opinion on what needs to be done to finish the review.


New commits:

a330d39p_group_cohomology-3.0: Fix checksum

comment:523 Changed 18 months ago by jdemeyer

  • Description modified (diff)

comment:524 Changed 18 months ago by jdemeyer

I prefer to do a final review after all dependencies have been merged. You can remind me when 8.3.beta6 is released.

comment:525 Changed 18 months ago by jdemeyer

Since there are no longer any changes to Sage here, it's just a matter of checking that the package builds and passes its testsuite, ideally on several platforms.

comment:526 in reply to: ↑ 510 Changed 18 months ago by jdemeyer

Replying to SimonKing:

For now, I am able to work around that problem, because at the moment it still works to pass a str to MatLoad

But that won't work on Python 3, right?

comment:527 Changed 18 months ago by jdemeyer

  • Branch changed from u/SimonKing/upgrade_of_group_cohomology_spkg to u/jdemeyer/upgrade_of_group_cohomology_spkg

comment:528 Changed 18 months ago by jdemeyer

  • Commit changed from a330d39c12ea918970ca3ac9678e02e3441dd760 to 7bf9f76405e664d32ce7c01f43afccc6f4b52a60
  • Dependencies #25511 deleted

Rebased to latest beta.


New commits:

7bf9f76Add optional package p_group_cohomology-3.0

comment:529 follow-up: Changed 18 months ago by tscrim

I tried this on 8.3.beta7 and it was not able to pass its test suite. It seems to be coming from this:

sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(8,1)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-887a1ff3a72e> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(1))

/home/travis/sage-build/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds)
   1531             if q < 128:
   1532                 extras['websource'] = False
-> 1533             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1534             return OUT
   1535 

/home/travis/sage-build/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1244             OUT = COHO(gap(KEY[0]), **extras)
   1245         else:
-> 1246             OUT = COHO(KEY[0],KEY[1], **extras)
   1247         _gap_init(OUT.group().parent())
   1248         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.G_ALG.r_action()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.makeMTX()

/home/travis/sage-build/local/lib/python2.7/site-packages/sage/matrix/matrix0.pyx in sage.matrix.matrix0.Matrix.__cinit__ (build/cythonized/sage/matrix/matrix0.c:3925)()
     95         [2]
     96     """
---> 97     def __cinit__(self, parent, *args, **kwds):
     98         """
     99         The initialization routine of the ``Matrix`` base class ensures

TypeError: __cinit__() takes at least 1 positional argument (0 given)

which was given by a relatively recent change to matrices __cinit__ (I don't remember the exact ticket). Once this is fixed, I am willing to set a positive review.

comment:530 in reply to: ↑ 529 ; follow-up: Changed 18 months ago by SimonKing

Replying to tscrim:

I tried this on 8.3.beta7 and it was not able to pass its test suite. It seems to be coming from this ... relatively recent change to matrices __cinit__ (I don't remember the exact ticket).

Yet again.

Frankly, it happens way too often that the group cohomology package gets broken because of changes in sagelib. So, I believe that the package should eventually be moved to the sage library, too, in order to make it easier to prevent this reoccurring mishap.

comment:531 Changed 18 months ago by jdemeyer

  • Status changed from needs_review to needs_work

This is no longer allowed:

cdef MTX M = MTX.__new__(MTX)

Since #25505, it is mandatory to construct matrices with parent as first argument. So you need

cdef MTX M = MTX.__new__(MTX, MatrixSpace(...))

Even better, use the function new_mtx from src/sage/matrix/matrix_gfpn_dense.pyx which more or less does the same thing.

comment:532 in reply to: ↑ 530 ; follow-up: Changed 18 months ago by tscrim

Replying to SimonKing:

Replying to tscrim:

I tried this on 8.3.beta7 and it was not able to pass its test suite. It seems to be coming from this ... relatively recent change to matrices __cinit__ (I don't remember the exact ticket).

Yet again.

Frankly, it happens way too often that the group cohomology package gets broken because of changes in sagelib. So, I believe that the package should eventually be moved to the sage library, too, in order to make it easier to prevent this reoccurring mishap.

I would be happy with that. I think it would be easier to maintain in the long-run. Is there a particular reason why this is not being done?

comment:533 in reply to: ↑ 532 ; follow-up: Changed 18 months ago by SimonKing

Replying to tscrim:

... So, I believe that the package should eventually be moved to the sage library, too, in order to make it easier to prevent this reoccurring mishap.

I would be happy with that. I think it would be easier to maintain in the long-run. Is there a particular reason why this is not being done?

It would already be possible to implement it with optional extension modules. But it has been argued at some point that optional extension modules should only be used for standard functionality of Sage that can also be provided by a non-standard backend.

So, when I talk about "putting it into the Sage library", I mean "putting it into the Sage library as plain extension modules that depend only on standard packages". Then:

  1. It would be needed to make SharedMeatAxe a standard package. Given that its matrix arithmetic is vastly superior to what vanilla Sage has to offer for matrices over small non-prime fields, this should happen anyway.
  2. It would be needed to make the C-library part of the group cohomology package (aka modular_resolution-1.0) a standard package.
  3. It would be needed to make SmallGroups (via database_gap) standard. This used to be impossible because of licencing, but that's fixed now.

comment:534 Changed 18 months ago by SimonKing

Why does "git trac checkout 18514" followed by "git trac pull" NOT yield the current branch?

comment:535 Changed 18 months ago by SimonKing

I guess using new_mtx instead of makeMTX is a good idea, partly because in many cases there will be a template, so that the creation of the new matrix will be even faster.

comment:536 in reply to: ↑ 533 ; follow-up: Changed 18 months ago by jdemeyer

Replying to SimonKing:

But it has been argued at some point that optional extension modules should only be used for standard functionality of Sage that can also be provided by a non-standard backend.

See 337 and following.

comment:537 in reply to: ↑ 536 ; follow-up: Changed 18 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

But it has been argued at some point that optional extension modules should only be used for standard functionality of Sage that can also be provided by a non-standard backend.

See 337 and following.

OK. But what I forgot to add: I care about documentation, and if I understand correctly, the documentation of an optional extension module will not be built (indeed, the documentation of matrix_gfpn_dense is not available). And I believe that's a show stopper for any complex package.

In any case, I would somehow prefer to postpone this to version 3.1. My plan for version 3.1 is to make the package independent of SmallGroups, and while I am at it, I can put the package into the Sage library --- provided that the documentation problem can be solved.

comment:538 in reply to: ↑ 537 Changed 18 months ago by tscrim

Replying to SimonKing:

OK. But what I forgot to add: I care about documentation, and if I understand correctly, the documentation of an optional extension module will not be built (indeed, the documentation of matrix_gfpn_dense is not available). And I believe that's a show stopper for any complex package.

Well, I am guessing you have the manually build and host the documentation yourself currently. So it would not be so different than being included in Sage. So I do not see it as really being that different than being a separate package. The interactive doc is still the same, and by being included, it will also get picked up by, e.g. search_src.

I am in Zaragoza with Jeroen, and we just a brief conversion about how to get Sphinx and Sage to handle optional extensions. He believes it should be something that Sphinx should be able to do, in which case, we could have the official Sage doc be built with all optional packages. However, this is something that I think will likely not be quick to get done.

In any case, I would somehow prefer to postpone this to version 3.1. My plan for version 3.1 is to make the package independent of SmallGroups, and while I am at it, I can put the package into the Sage library --- provided that the documentation problem can be solved.

If you are able to do that and MeatAxe also becomes standard, then the doc issues will be fixed because it will be a usual pyx file.

comment:539 Changed 18 months ago by SimonKing

  • Branch changed from u/jdemeyer/upgrade_of_group_cohomology_spkg to u/SimonKing/upgrade_of_group_cohomology_spkg

comment:540 Changed 18 months ago by SimonKing

  • Commit changed from 7bf9f76405e664d32ce7c01f43afccc6f4b52a60 to e59e1dc79070fe7ae722e30acfd3629372a87496
  • Description modified (diff)
  • Status changed from needs_work to needs_review

Good news: I managed to replace !makeMTX by !new_mtx.

Bad news:

  1. Strange enough, the "commit" field of the ticket has not been correctly set, I needed to edit it manually.
  2. THE PACKAGE DOES NOT EVEN START TO BUILD. ARRGHH!!!

I have posted the tar ball in its original location (see ticket description). The attempt to install the package ends in an infinite loop:

...
[p_group_cohomology-3.0] Installing p_group_cohomology-3.0
[p_group_cohomology-3.0] Warning: This package has a badly-behaved setup.py which outputs
[p_group_cohomology-3.0] more than the package name for 'setup.py --name'; using the last
[p_group_cohomology-3.0] line as the package name: pGroupCohomology
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
[p_group_cohomology-3.0] Skipping pGroupCohomology as it is not installed.
...

Of course it is not installed! That's why I want to install it, stupid!

Can you explain what is going wrong here?

I noticed that the python setup.py --name results in a compilation of the cython files, which certainly shouldn't happen. How to prevent it?

But if I understand correctly, the installation went past that point and did correctly obtain the package name. So, why does it REPEATEDLY complain that the package to be installed is not installed.

comment:541 follow-up: Changed 18 months ago by SimonKing

Note that today I upgraded my pip installation. It is now

$ pip --version
pip 10.0.1 from /home/king/Sage/git/sage/local/lib/python2.7/site-packages/pip (python 2.7)

Can this be related? I am no totally sure, though, because installing the package directly from the unpacked sources did work.

So, bug in Sage's package builder?

comment:542 in reply to: ↑ 541 ; follow-up: Changed 18 months ago by jdemeyer

Replying to SimonKing:

Note that today I upgraded my pip installation. It is now

$ pip --version
pip 10.0.1 from /home/king/Sage/git/sage/local/lib/python2.7/site-packages/pip (python 2.7)

Can this be related?

Absolutely! I can confirm that Sage does not work with pip 10.

comment:543 in reply to: ↑ 542 Changed 18 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

Note that today I upgraded my pip installation. It is now

$ pip --version
pip 10.0.1 from /home/king/Sage/git/sage/local/lib/python2.7/site-packages/pip (python 2.7)

Can this be related?

Absolutely! I can confirm that Sage does not work with pip 10.

Ouch.

So, at least two questions arise:

  • Why is it possible to install the package from its sources by doing python setup.py build followed by pip install . --upgrade? Note that plainly doing pip install . did not suffice.
  • How to downgrade pip? Note that system-wide I have pip 8. So, if you tell me how to remove pip from the sage installation, I would do so.

comment:544 follow-up: Changed 18 months ago by SimonKing

I tried "pip uninstall pip", which first seemed to work. But now it seems that I have to rebuild Sage from scratch.

comment:545 follow-ups: Changed 18 months ago by SimonKing

After downgrading pip, installing the package works. But alas, there are two independent errors in its test suite:

1.

[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 2457, in pGroupCohomology.cohomology.COHO
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     H4.make()
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology.cohomology.COHO[91]>", line 1, in <module>
[p_group_cohomology-3.0]         H4.make()
[p_group_cohomology-3.0]       File "pGroupCohomology/cohomology.pyx", line 12099, in pGroupCohomology.cohomology.COHO.make
[p_group_cohomology-3.0]       File "pGroupCohomology/cohomology.pyx", line 11559, in pGroupCohomology.cohomology.COHO.next
[p_group_cohomology-3.0]       File "pGroupCohomology/resolution.pyx", line 2194, in pGroupCohomology.resolution.RESL.makeAutolift
[p_group_cohomology-3.0]     TypeError: Cannot convert sage.matrix.matrix_mod2_dense.Matrix_mod2_dense to sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense

If I understand correctly that's when testing the option to use a different matrix backend for some internal computations. Hence, at some point a conversion is needed, which presumably fails because of internal changes in Sage.

2.

[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/barcode.py", line 465, in pGroupCohomology.barcode.BarCode2d.plot
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     plot(B)     # render
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology.barcode.BarCode2d.plot[5]>", line 1, in <module>
[p_group_cohomology-3.0]         plot(B)     # render
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/decorators.py", line 567, in wrapper
[p_group_cohomology-3.0]         return func(*args, **options)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/plot/plot.py", line 1942, in plot
[p_group_cohomology-3.0]         G = funcs.plot(*args, **original_opts)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/barcode.py", line 469, in plot
[p_group_cohomology-3.0]         from sage.plot.plot import circle, line
[p_group_cohomology-3.0]     ImportError: cannot import name circle

If I understand correctly, it means that there is a changed import location, i.e., yet another internal change in Sage that creates trouble.

When I developed the first package version in 2009, Sage was an ideal platform. Frankly, if I would create a similar package today, I wouldn't know what platform to choose.

Last edited 18 months ago by SimonKing (previous) (diff)

comment:546 in reply to: ↑ 545 Changed 18 months ago by SimonKing

Replying to SimonKing:

1.

[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 2457, in pGroupCohomology.cohomology.COHO
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     H4.make()
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology.cohomology.COHO[91]>", line 1, in <module>
[p_group_cohomology-3.0]         H4.make()
[p_group_cohomology-3.0]       File "pGroupCohomology/cohomology.pyx", line 12099, in pGroupCohomology.cohomology.COHO.make
[p_group_cohomology-3.0]       File "pGroupCohomology/cohomology.pyx", line 11559, in pGroupCohomology.cohomology.COHO.next
[p_group_cohomology-3.0]       File "pGroupCohomology/resolution.pyx", line 2194, in pGroupCohomology.resolution.RESL.makeAutolift
[p_group_cohomology-3.0]     TypeError: Cannot convert sage.matrix.matrix_mod2_dense.Matrix_mod2_dense to sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense

If I understand correctly that's when testing the option to use a different matrix backend for some internal computations. Hence, at some point a conversion is needed, which presumably fails because of internal changes in Sage.

No, I was mistaken: When providing a template for new_mtx, I accidentally chose a matrix that with the option "nouseMTX" is not a meataxe matrix.

comment:547 in reply to: ↑ 545 Changed 18 months ago by SimonKing

Replying to SimonKing:

2.

[p_group_cohomology-3.0] File "pyxsources/pGroupCohomology/barcode.py", line 465, in pGroupCohomology.barcode.BarCode2d.plot
[p_group_cohomology-3.0] Failed example:
[p_group_cohomology-3.0]     plot(B)     # render
[p_group_cohomology-3.0] Exception raised:
[p_group_cohomology-3.0]     Traceback (most recent call last):
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
[p_group_cohomology-3.0]         self.compile_and_execute(example, compiler, test.globs)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
[p_group_cohomology-3.0]         exec(compiled, globs)
[p_group_cohomology-3.0]       File "<doctest pGroupCohomology.barcode.BarCode2d.plot[5]>", line 1, in <module>
[p_group_cohomology-3.0]         plot(B)     # render
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/decorators.py", line 567, in wrapper
[p_group_cohomology-3.0]         return func(*args, **options)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/plot/plot.py", line 1942, in plot
[p_group_cohomology-3.0]         G = funcs.plot(*args, **original_opts)
[p_group_cohomology-3.0]       File "/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/barcode.py", line 469, in plot
[p_group_cohomology-3.0]         from sage.plot.plot import circle, line
[p_group_cohomology-3.0]     ImportError: cannot import name circle

If I understand correctly, it means that there is a changed import location, i.e., yet another internal change in Sage that creates trouble.

Apparently the import location HAS changed. However, I am importing "circle" for absolutely no reason. I only use "line", which still can be imported from sage.plot.plot. However, the official import statement is from sage.plot.line import line, and I'll use this.

comment:548 Changed 18 months ago by SimonKing

  • Status changed from needs_review to needs_work

comment:549 Changed 18 months ago by git

  • Commit changed from e59e1dc79070fe7ae722e30acfd3629372a87496 to 0aaeef3f5ce41405e9e1755d62f57046db134db4

Branch pushed to git repo; I updated commit sha1. New commits:

0aaeef3p_group_cohomology: Fixing bugs from previous commit

comment:550 Changed 18 months ago by SimonKing

  • Status changed from needs_work to needs_review

Hooray...

[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] All tests passed!
[p_group_cohomology-3.0] ----------------------------------------------------------------------
[p_group_cohomology-3.0] Total time for all tests: 634.0 seconds
[p_group_cohomology-3.0]     cpu time: 878.8 seconds
[p_group_cohomology-3.0]     cumulative wall time: 1557.3 seconds
[p_group_cohomology-3.0] 
[p_group_cohomology-3.0] real	10m36.533s
[p_group_cohomology-3.0] user	15m26.060s
[p_group_cohomology-3.0] sys	2m23.668s
[p_group_cohomology-3.0] Successfully installed p_group_cohomology-3.0

So, things should now work, with the package that I have posted at the location indicated in the ticket description.

comment:551 follow-up: Changed 18 months ago by tscrim

It now works for me. I am basically ready to set this to a positive review, except I do have a two questions:

  • What are you going to do for documentation?
  • How will people know about this inside of Sage? In particular, I think you should have some link to the spkg doc from Sage's doc, wherever you decide to host the spkg doc. Also, I think it would be useful to have a file in the sage/tests folder that does some of the tests for this package. That way, it would be more likely that when some behavior changes that your spkg is breaking.

comment:552 in reply to: ↑ 544 Changed 18 months ago by jdemeyer

Replying to SimonKing:

I tried "pip uninstall pip", which first seemed to work. But now it seems that I have to rebuild Sage from scratch.

I probably should have said this earlier, but ./sage -f pip should fix your problem.

comment:553 in reply to: ↑ 551 Changed 18 months ago by SimonKing

Replying to tscrim:

  • What are you going to do for documentation?

The documentation is hosted at my home institute.

  • How will people know about this inside of Sage?

I guess that's the same for most optional Sage packages. Or maybe there is one crucial difference: Most optional packages are independent projects that can be installed and used in the Sage ecosystem; so, people become aware without Sage. But p_group_cohomology is a package that *uses* Sage and is thus not independent.

However, is a problem here to solve? I know that some experts for modular cohomology of finite groups know p_group_cohomology, but I doubt that people outside that niche will use that package.

In particular, I think you should have some link to the spkg doc from Sage's doc, wherever you decide to host the spkg doc. Also, I think it would be useful to have a file in the sage/tests folder that does some of the tests for this package. That way, it would be more likely that when some behavior changes that your spkg is breaking.

This would only be executed on a machine that has the package installed in the first place, right?

But actually I think this might be a good idea: The current test suite is only executed if the package is newly installed and the user doesn't forget to provide the -c option. But if one has a Python file with optional tests somewhere in the sage library, then these tests would be executed when doing "make test" provided that the package is installed. And in that way, problems of my package resulting from a change in sagelib would be more likely to become visible.

OK, I'll try to cook something up...

comment:554 follow-up: Changed 18 months ago by jdemeyer

  • Status changed from needs_review to needs_work

There is an undeclared dependency on matplotlib: the testsuite plots stuff, so you should add matplotlib as dependency (order-only, so after |).

comment:555 Changed 18 months ago by git

  • Commit changed from 0aaeef3f5ce41405e9e1755d62f57046db134db4 to 45146d82f5a77402aecbd7ba3ebb30e9dbd76dac

Branch pushed to git repo; I updated commit sha1. New commits:

4b89060p_group_cohomology-3.0: Add matplotlib as dependency
45146d8p_group_cohomology-3.0: Add sage.tests.modular_group_cohomology

comment:556 in reply to: ↑ 554 Changed 18 months ago by SimonKing

  • Status changed from needs_work to needs_review

Replying to jdemeyer:

There is an undeclared dependency on matplotlib: the testsuite plots stuff, so you should add matplotlib as dependency (order-only, so after |).

Is it not part of $SAGE_RUNTIME? In any case, I added it as a dependency.

Also, I added src/sage/tests/modular_group_cohomology.py. The tests there take 9 seconds (I hope that's OK), and they cover a wide range of functionality of the package. Hence, if some change in the sage library breaks p_group_cohomology and a user who has installed the package runs make test, then the problem is very likely to become visible immediately.

I guess it is back to "needs review", right?

comment:557 Changed 18 months ago by jdemeyer

OK, let me test this new version on a few machines. If everything goes well, I agree that positive review can be given.

comment:558 follow-ups: Changed 18 months ago by tscrim

So on my machine, I get

sage -t --warn-long src/sage/tests/modular_group_cohomology.py
**********************************************************************
File "src/sage/tests/modular_group_cohomology.py", line 46, in sage.tests.modular_group_cohomology
Warning, slow doctest:
    H.essential_ideal()                           # optional - p_group_cohomology
Test ran for 1.39 s
**********************************************************************
File "src/sage/tests/modular_group_cohomology.py", line 62, in sage.tests.modular_group_cohomology
Warning, slow doctest:
    H = CohomologyRing(gap(AlternatingGroup(7)), GroupName="A(7)", prime=2, from_scratch=True) # optional - p_group_cohomology
Test ran for 1.51 s
**********************************************************************
File "src/sage/tests/modular_group_cohomology.py", line 79, in sage.tests.modular_group_cohomology
Warning, slow doctest:
    H = CohomologyRing(gap(AlternatingGroup(6)), GroupName="A(6)", prime=3, from_scratch=True) # optional - p_group_cohomology
Test ran for 1.90 s

So while these tests are longer than 1 second on my machine, I think they are not sufficiently long to warrant a # long time considering the purpose of this file.

comment:559 in reply to: ↑ 558 ; follow-up: Changed 18 months ago by SimonKing

Replying to tscrim:

So on my machine, I get

sage -t --warn-long src/sage/tests/modular_group_cohomology.py
**********************************************************************
File "src/sage/tests/modular_group_cohomology.py", line 46, in sage.tests.modular_group_cohomology
Warning, slow doctest:
    H.essential_ideal()                           # optional - p_group_cohomology
Test ran for 1.39 s
**********************************************************************
File "src/sage/tests/modular_group_cohomology.py", line 62, in sage.tests.modular_group_cohomology
Warning, slow doctest:
    H = CohomologyRing(gap(AlternatingGroup(7)), GroupName="A(7)", prime=2, from_scratch=True) # optional - p_group_cohomology
Test ran for 1.51 s
**********************************************************************
File "src/sage/tests/modular_group_cohomology.py", line 79, in sage.tests.modular_group_cohomology
Warning, slow doctest:
    H = CohomologyRing(gap(AlternatingGroup(6)), GroupName="A(6)", prime=3, from_scratch=True) # optional - p_group_cohomology
Test ran for 1.90 s

Would one get these warnings automatically? Or does one need to provide a special option? The complete log for my test run is this:

$ ./sage -t src/sage/tests/modular_group_cohomology.py 
too many failed tests, not using stored timings
Running doctests with ID 2018-06-30-20-33-28-f134a034.
Git branch: t/18514/upgrade_of_group_cohomology_spkg
Using --optional=database_gap,meataxe,mpir,p_group_cohomology,python2,sage
Doctesting 1 file.
sage -t src/sage/tests/modular_group_cohomology.py
    [16 tests, 9.26 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 9.3 seconds
    cpu time: 7.3 seconds
    cumulative wall time: 9.3 seconds

It is of course conceivable that some of the 16 tests actually take longer than one second (given that on average they take 0.58 seconds). But has the policy changed? I seem to remember that when I was young, tests were supposed to be "not longer than a few seconds". But just *one* second seems short to me.

comment:560 in reply to: ↑ 559 Changed 18 months ago by SimonKing

Replying to SimonKing:

Would one get these warnings automatically? Or does one need to provide a special option?

Sorry, I didn't notice that you used sage -t --warn-long.

On my machine, I get warnings for four tests:

    H.essential_ideal()
    ascii_art(H.bar_code('LowerCentralSeries')[2])
    H = CohomologyRing(gap(AlternatingGroup(7)), GroupName="A(7)", prime=2, from_scratch=True)
    H = CohomologyRing(gap(AlternatingGroup(6)), GroupName="A(6)", prime=3, from_scratch=True)

I'll try to come up with shorter tests.

comment:561 Changed 18 months ago by git

  • Commit changed from 45146d82f5a77402aecbd7ba3ebb30e9dbd76dac to d4a612c7ebf08d5bb48479163099671fd880994f

Branch pushed to git repo; I updated commit sha1. New commits:

d4a612cp_group_cohomology-3.0: Simplify tests/modular_group_cohomology.py

comment:562 Changed 18 months ago by SimonKing

I have simplified the tests and still think that they cover a wide range of functionality. This time I didn't get a warning (and I tested 10 times):

$ ./sage -t --warn-long src/sage/tests/modular_group_cohomology.py
Running doctests with ID 2018-07-01-00-21-35-4bcb0f8f.
Git branch: t/18514/upgrade_of_group_cohomology_spkg
Using --optional=database_gap,meataxe,mpir,p_group_cohomology,python2,sage
Doctesting 1 file.
sage -t --warn-long src/sage/tests/modular_group_cohomology.py
    [17 tests, 5.15 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 5.2 seconds
    cpu time: 4.0 seconds
    cumulative wall time: 5.2 seconds

comment:563 in reply to: ↑ 558 ; follow-up: Changed 18 months ago by jdemeyer

Replying to tscrim:

So while these tests are longer than 1 second on my machine, I think they are not sufficiently long to warrant a # long time considering the purpose of this file.

Why not? The official policy is that normal tests should take less than 1 second, # long time tests less than 30 seconds. Remember that 1 second on your machine may be multiple seconds on a slower machine.

comment:564 in reply to: ↑ 563 ; follow-up: Changed 18 months ago by tscrim

Replying to jdemeyer:

Replying to tscrim:

So while these tests are longer than 1 second on my machine, I think they are not sufficiently long to warrant a # long time considering the purpose of this file.

Why not? The official policy is that normal tests should take less than 1 second, # long time tests less than 30 seconds. Remember that 1 second on your machine may be multiple seconds on a slower machine.

2/3 were less than 1.5 seconds, and my machine is not particularly fast. I am happier to have the shorter tests. Yet, I think this is something that will be tested only by the bots (if they even have this optional package installed) and is meant to catch interface breakage, I did not see as much harm in having longer regular tests.

comment:565 Changed 18 months ago by jdemeyer

  • Branch changed from u/SimonKing/upgrade_of_group_cohomology_spkg to u/jdemeyer/upgrade_of_group_cohomology_spkg

comment:566 in reply to: ↑ 564 ; follow-up: Changed 18 months ago by jdemeyer

  • Commit changed from d4a612c7ebf08d5bb48479163099671fd880994f to 65769150a96fcd8a5fcfad88a4c2173dedb11b42
  • Reviewers set to Travis Scrimshaw, Jeroen Demeyer
  • Status changed from needs_review to positive_review

Replying to tscrim:

2/3 were less than 1.5 seconds, and my machine is not particularly fast. I am happier to have the shorter tests. Yet, I think this is something that will be tested only by the bots (if they even have this optional package installed) and is meant to catch interface breakage, I did not see as much harm in having longer regular tests.

On the other hand, those bots typically run tests with --long, so it wouldn't hurt to add # long time in that case.

I just added a few trivial changes to wrap some long lines, so I guess this is finally good to go...


New commits:

10cdf0eAdd optional package p_group_cohomology-3.0
6576915Wrapping some long lines

comment:567 in reply to: ↑ 566 Changed 18 months ago by SimonKing

Replying to jdemeyer:

Replying to tscrim:

2/3 were less than 1.5 seconds, and my machine is not particularly fast. I am happier to have the shorter tests. Yet, I think this is something that will be tested only by the bots (if they even have this optional package installed) and is meant to catch interface breakage, I did not see as much harm in having longer regular tests.

On the other hand, those bots typically run tests with --long, so it wouldn't hurt to add # long time in that case.

Just for the record: The tests in the final version are all less than a second.

In any case: Thank you for the positive review!

comment:568 Changed 18 months ago by vbraun

  • Branch changed from u/jdemeyer/upgrade_of_group_cohomology_spkg to 65769150a96fcd8a5fcfad88a4c2173dedb11b42
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.