Opened 8 years ago
Closed 4 years ago
#18514 closed defect (fixed)
Upgrade of p_group_cohomology spkg
Reported by:  Simon King  Owned by:  

Priority:  major  Milestone:  sage8.3 
Component:  packages: optional  Keywords:  group cohomology 
Cc:  jhpalmiery, Volker Braun, Jeroen Demeyer, Harald Schilly, Han Frederic, david.green@…  Merged in:  
Authors:  Simon King  Reviewers:  Travis Scrimshaw, Jeroen Demeyer 
Report Upstream:  None of the above  read trac for reasoning.  Work issues:  
Branch:  6576915 (Commits, GitHub, GitLab)  Commit:  65769150a96fcd8a5fcfad88a4c2173dedb11b42 
Dependencies:  Stopgaps: 
Description (last modified by )
In the past, p_group_cohomology2.1.4 was an optional spkg. Sage dropped support for old style spkgs, and thus only an experimental spkg remained.
The purpose of this ticket is to provide a new group cohomology package for Sage. I suggest to make it optional (not experimental).
In the experimental version 2.1.5, I have improved the computation of Poincaré series, depth and filter degree type (the latter now uses a Hilbertdriven computation of Gröbner bases in elimination order, which works since in that setting the Hilbert function is easily available), and I added new functionality related with nilpotency.
Package structure
Version 3.0 is refactored as follows:
 It depends on the optional meataxe package (see #24359), which is based on a fork of MeatAxe.
 The spkginstall script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.
 The C code written by David Green has become an autotoolized shared library.
 Gap and Singular code is put into
SAGE_SHARE/
, where GAP and Singular libraries belong.  The rest of the package is formed by python and cython code, and uses pip for installation.
 The documentation is built in an improved way.
Further changes
 The custom logging in the old spkg is replaced by Python's logging module.
 The tests avoid sideeffects by using a reset function. Hence, no custom doc tester is needed:
sage t
is enough.  In the previous spkg, I tried to be as backward compatible as possible. But now I'm dropping support for very old Singular versions.
 Some experimental options for the cohomology computation are removed.
Many tests have changed, since the ring presentations of the computed cohomology rings, specifically for nonprimepower groups, have changed. I do not fully understand why they changed. but it seems that it is a consequence of changes in Singular. The changes did (of course...) not affect the isomorphism types of the rings.
Upstream tar ball
https://users.fmi.unijena.de/cohomology/p_group_cohomology3.0.tar.xz
Change History (568)
comment:1 Changed 8 years ago by
Status:  new → needs_review 

comment:2 followup: 3 Changed 8 years ago by
I see you are compiling some extensions with the csage
library but there is a ticket (#17854) which will make that library disappear. Ideally, you should only compile against csage
only if it actually exists.
comment:3 Changed 8 years ago by
comment:4 followup: 5 Changed 8 years ago by
The sig_on()
framework was in csage
until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link against csage
if the Sage version is <= 6.7
comment:5 followup: 6 Changed 8 years ago by
Replying to jdemeyer:
The
sig_on()
framework was incsage
until 6.7 and got moved to the Sage library in 6.8.beta0 (#18027). So you can probably just link againstcsage
if the Sage version is <= 6.7
OK. What does that mean for the code? Do I need to import anything if I am using sig_on/off
, or would this just work automatically with the new Cython version?
comment:6 Changed 8 years ago by
Replying to SimonKing:
OK. What does that mean for the code? Do I need to import anything if I am using
sig_on/off
, or would this just work automatically with the new Cython version?
You just include sage/ext/interrupt.pxi
, that doesn't change.
The main difference is that you no longer need to add csage
as library. Currently, it doesn't hurt if you add the library anyway, but in the future, csage
might not exist anymore.
And there is one additional caveat which is relevant for your package: since #18027, you must include interrupt.pxi
only in .pyx
files and not .pxd
files (unless you really know what you're doing).
comment:7 followup: 8 Changed 8 years ago by
On the topic of .pxd
files, this is what they should contain:
 The API of your module, that is all
cdef
classes andc(p)def
functions that you want to export for use in other modules (in the same package or by other packages).
 If your module provides bindings to some library (the
mtx
module in your package for example), then add the declarations of the library functions in the.pxd
file.
 Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from
RingElement
. In your.pxd
file, you want to writecdef class MySpecialElement(RingElement)
. This makesRingElement
a dependency, so you need to writefrom ... cimport RingElement
in the.pxd
file.
And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the .pyx
file. This is important in order to minimize dependencies.
NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different .pxd
file. In Sage, we usually put the library bindings in sage/libs
, look at sage/libs/gmp
for a good example.
comment:8 followup: 9 Changed 8 years ago by
Replying to jdemeyer:
On the topic of
.pxd
files, this is what they should contain:
 The API of your module, that is all
cdef
classes andc(p)def
functions that you want to export for use in other modules (in the same package or by other packages).
 If your module provides bindings to some library (the
mtx
module in your package for example), then add the declarations of the library functions in the.pxd
file.
 Any dependencies of the above. Sage example: if you're defining a new kind of element of a ring, you will probably inherit from
RingElement
. In your.pxd
file, you want to writecdef class MySpecialElement(RingElement)
. This makesRingElement
a dependency, so you need to writefrom ... cimport RingElement
in the.pxd
file.And that's it really. Things like interrupts have nothing to do with the API of your module, those are implementation details and belong in the
.pyx
file. This is important in order to minimize dependencies.
OK. That's something that I should change. Actually, until not so long ago, I thought that cimports were only possible in pxd files.
NOTE about 2: if your module provides bindings to some library but also does significant other stuff, then it's best to split up the bindings in a different
.pxd
file. In Sage, we usually put the library bindings insage/libs
, look atsage/libs/gmp
for a good example.
Do I understand correctly: You suggest that I should create a new pxd file for the MeatAxe
bindings, say, mtx_lib.pxd
, which is then cimported in mtx.pxd
?
comment:9 Changed 8 years ago by
Replying to SimonKing:
Do I understand correctly: You suggest that I should create a new pxd file for the
MeatAxe
bindings, say,mtx_lib.pxd
, which is then cimported inmtx.pxd
?
I cannot really speak about this specific case, but it certainly would make sense to split it up.
comment:10 Changed 8 years ago by
Component:  PLEASE CHANGE → packages: optional 

comment:11 Changed 8 years ago by
Status:  needs_review → needs_work 

Work issues:  → Cope with yet another backward incompatible change of Sage 
I just found that the package won't build with the latest beta of Sage. Reason:
pGroupCohomology/cohomology.c:248:36: fatal error: sage/ext/interrupt/pxi.h: No such file or directory #include "sage/ext/interrupt/pxi.h" ^ compilation terminated. error: command 'gcc' failed with exit status 1
So, can you please tell me how to use interrupts, after all these backward incompatible changes? It really sucks.
comment:12 Changed 8 years ago by
Good news! On my machine, I did some refactoring of code according to the hints you gave meand that did the trick.
My plan is to continue refactoring, taking into account what you said about where to import stuff, and then push a new package version to the official page.
comment:13 followup: 24 Changed 8 years ago by
Bad news. It builds, but I can't import it. Which brings up the question again why there are all these incompatible changes in Sage. They didn't use to occur so frequently in earlier versions of Sage, IIRC.
comment:14 Changed 8 years ago by
Status:  needs_work → needs_review 

Work issues:  Cope with yet another backward incompatible change of Sage 
I got it fixed. So, you'll find the updated version of p_group_cohomology2.1.5
on my web page.
Some of your concerns about importing/including stuff are addressed in the new version: I moved some of it from .pxd files into .pyx files. What I did not do is to split off library bindings and put them into separate .pxd files. That would still make sense, but for now I put the ticket back to "needs review".
comment:15 Changed 8 years ago by
Oh, and I also created a new module pGroupCohomology.auxiliaries
, in which I've put some auxiliary functions that have previously been in pGroupCohomology.resolution
.
comment:16 Changed 7 years ago by
Cc:  Han Frederic added 

I have a built failure for
sage i upstream/p_group_cohomology2.1.5.spkg
with sage 6.8beta3+trac18666 (but success with master)
gcc pthread shared L/home/fred/dev/sage/local/lib build/temp.linuxx86_642.7/pGroupCohomology/resolution.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/aufloesung.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/aufnahme.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/djgerr.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/fileplus.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/nBuchberger.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/pgroup.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/pincl.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/slice.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/urbild.o build/temp.linuxx86_642.7/pGroupCohomology/c_sources/uvr.o L/home/fred/dev/sage/local/lib lmtx lpython2.7 o build/lib.linuxx86_642.7/pGroupCohomology/resolution.so cythoning pGroupCohomology/cohomology.pyx to pGroupCohomology/cohomology.c building 'pGroupCohomology.cohomology' extension gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall fPIC I/home/fred/dev/sage/src/c_lib/include I/home/fred/dev/sage/local/share/sage/ext I/home/fred/dev/sage/src/sage/ext Imtx2.2.4/src IpGroupCohomology/c_sources IpGroupCohomology I/home/fred/dev/sage/local/include/python2.7 c pGroupCohomology/cohomology.c o build/temp.linuxx86_642.7/pGroupCohomology/cohomology.o O3 In file included from pGroupCohomology/cohomology.c:247:0: mtx2.2.4/src/meataxe.h:15:0: warning: "_GNU_SOURCE" redefined #define _GNU_SOURCE ^ In file included from pGroupCohomology/cohomology.c:8:0: /home/fred/dev/sage/local/include/python2.7/pyconfig.h:1160:0: note: this is the location of the previous definition #define _GNU_SOURCE 1 ^ pGroupCohomology/cohomology.c:262:37: fatal error: sage/libs/ntl/ntlwrap.cpp: No such file or directory #include "sage/libs/ntl/ntlwrap.cpp" ^ compilation terminated. error: command 'gcc' failed with exit status 1 Error installing Cython code. real 0m43.412s user 0m40.636s sys 0m2.736s ************************************************************************ Error installing package p_group_cohomology2.1.5 ************************************************************************
comment:17 Changed 7 years ago by
You might want to wait for #18494, which should make it easier to install external packages using the Sage library.
comment:18 followup: 19 Changed 7 years ago by
The real questions are: Why is it looking for ntl? I don't think it is explicitly requested by my package, thus, it probably is an indirect dependency, i.e., one that is in fact a dependency of something in Sage that I cimport into my package. But if it is a dependency of something in Sage, then why does it not work?
I suppose that I have to provide an include path to fix it. If the dependency really is via something in the Sage library, then this "have to" is not very niceSage could figure it out without human help.
comment:19 Changed 7 years ago by
Replying to SimonKing:
The real questions are: Why is it looking for ntl?
Because the Sage <> NTL interface is a real mess. In particular, any module which needs just a fragment of NTL automatically cimports essentially all of the NTL interface. In particular, sage.rings.integer
does that.
comment:20 Changed 7 years ago by
In setup.py
, replace
include_dirs = CSAGE_PATH + [ os.path.join(SAGE_SRC,"c_lib","include"), SAGE_EXTCODE, SRC_EXT, os.path.join("mtx2.2.4","src"), os.path.join("pGroupCohomology","c_sources"), "pGroupCohomology"]
by
try: # Sage >= 6.8 from sage.env import sage_include_directories except ImportError: # Sage < 6.8 def sage_include_directories(): return [ os.path.join(SAGE_LOCAL, "include", "csage"), os.path.join(SAGE_SRC, "c_lib", "include"), SAGE_EXTCODE, SRC_EXT] # Not sure that you really need SAGE_EXTCODE ... include_dirs = sage_include_directories() + [ os.path.join("mtx2.2.4","src"), os.path.join("pGroupCohomology","c_sources"), "pGroupCohomology"]
and use #18494.
comment:21 Changed 7 years ago by
Dependencies:  → #18494 

I hope it is safe to make #18494 a dependency. If I understand correctly, there is only a very trivial doctest failure, and so hopefully it will be in Sage soonish.
Meanwhile I almost gave up to make my package backwards compatible with sage3.x...
comment:22 Changed 7 years ago by
With sage_include_directories()
I will not need to construct CSAGE_PATH
, right?
comment:23 Changed 7 years ago by
I have updated the spkg, and successfully installed it on the Sage version that is provided by #18494.
comment:24 Changed 7 years ago by
Replying to SimonKing:
Which brings up the question again why there are all these incompatible changes in Sage.
Essentially, because of #17854 and its subtickets. Removing the lowlevel c_lib
(a.k.a. libcsage
) did indeed have consequences for all packages compiling against Sage.
They didn't use to occur so frequently in earlier versions of Sage, IIRC.
And they probably will not occur anymore in the future, now that #17854 is dealt with. And if there are changes, now we can just change the function sage_include_directories()
from #18494.
comment:25 followup: 26 Changed 7 years ago by
Is there a reason not to make a newstyle packages for p_group_cohomology
?
comment:26 Changed 7 years ago by
Replying to tmonteil:
Is there a reason not to make a newstyle packages for
p_group_cohomology
?
Yes. I need someone who explains to me in ALL detail how to create a newstyle package.
It begins with: How do I create an "upstream" repository for my sources? Note that I will not change the meataxe matrix wrapper that is included in the package into a wrapper for the latest meataxe upstream, as this would require to rewrite most of the c and much of the cythoncode. So, the files in the current package are an old version of meataxe with massive patches (i.e., basically a fork), plus c and cython and singular and gapcode, for which this package currently is the only source.
comment:27 Changed 7 years ago by
You create a folder in the $SAGE_ROOT/build/pkgs
with files similar to the others. You will also need to put the tarball with the source code in $SAGE_ROOT/upstream
(and post it here or online so that it can be put on the online repo once merged) and then run sage sh sagefixpkgchecksums
. All of this folder should be under version control.
See also: http://doc.sagemath.org/html/en/developer/packaging.html
comment:28 Changed 7 years ago by
It seems that there is no way around creating a newstyle spkg, although I still find it odd to artificially create an upstream source in a situation where the spkg IS upstream. Sigh.
I have to
 get a github account (or reactivate an existing account?),
 learn how to convert the existing hg repository in the spkg to git,
 look up how to push my private repository to github,
 do what is said in comment:27.
Did I forget something?
Now comes the complicated part.
The spkg has three sources, namely
 CMeatAxe
 C programs written by David Green
 Cython, Python, Gap and Singular code written by me.
Ad 1.
Actually the starting point is a very old version of CMeatAxe. If I see that correctly, upstream currently isn't very active. However, compared with the version used in the spkg, upstream has changed most function names.
I apply big patches to CMeatAxe, as I implemented Strassen multiplication. It improves performance a lot (in the old sources). However, Strassen still is not implemented in the current upstream sources. Two years ago I attended a presentation of one CMeatAxe developer, and he said that they prefer to optimise the existing implementation rather than adding asymptotically fast implementations. It didn't convince him when I told him that in the old MeatAxe, Strassen beats school book multiplication for matrices as small as 200x200.
Ad 2.
David Green's programs have never been published independently. Hence, the spkg *is* upstream. The code uses the old CMeatAxe extensively.
Ad 3.
My code has never been published independently. Hence, the spkg *is* upstream. It contains a Cython wrapper for the old CMeatAxe.
Conclusion
I guess in an ideal world, I would create a new spkg for CMeatAxe, containing my patches and my Cython wrapper and the MeatAxe executable, but starting with current upstream. I would then create a repository for Green's and my code. p_group_cohomology would then depend on the new MeatAxe spkg.
If I recall correctly, I actually have a wrapper for a recent version of MeatAxe, perhaps even with my Strassen implementation. Hence, creating the new MeatAxe spkg might be a feasible first step.
However, rewriting Green's programs in terms of the new MeatAxe could be a pain in the neck.
So, as a short term solution, I'd like to keep the proposed version 2.1.5 of my spkg, that still uses the old CMeatAxe, with the prospect that version 3.0 should be based on the current MeatAxe upstream.
comment:29 followup: 32 Changed 7 years ago by
Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.
comment:30 followup: 33 Changed 7 years ago by
I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).
comment:31 Changed 7 years ago by
Description:  modified (diff) 

comment:32 Changed 7 years ago by
Replying to SimonKing:
Sigh. Talking about complications: The spkg is too large for attaching it on trac. But sage.math.washington isn't available.
how about you make a copy of your spkg on geom.math.washington.edu or combinat.math.washington.edu somewhere readable, e.g. to /tmp ?
At least then we can pick it up and put up somewhere...
comment:33 Changed 7 years ago by
Replying to tscrim:
I think you could just move the code into a tar file and the installation part of the spkg goes in the folder. The tarball just needs to be (temporarily) hosted somewhere until it gets put into the Sage upstream mirror I believe. I can host it on my github page or find a place to put it to lessen your burden. I have to go teach now, but I will take a detailed look tonight (including coming up with a better hosting plan).
Thank you! For now, I have put the oldstyle spkg at http://users.minet.unijena.de/cohomology/p_group_cohomology2.1.5.spkg
comment:34 Changed 7 years ago by
PS: The spkg uses a data base of precomputed cohomology rings that used to be at sage.math.washington. Hence, now it is not accessible.
If the spkg can't find a cohomology ring in the data base, then it will compute it from scratch. So, most selftests should work. However, some tests explicitly try to find data in the sage.math.washington data base. These will fail.
It seems that the data base (2030GB) can be relocated to Jena as well. But it will take at least till next week, and when it's done I need to change the spkg to using a new default url of the data base.
comment:35 Changed 7 years ago by
From the README file of the latest upstream sources (which is version 2.4.24 from October 2011(!)).
================================================================================ INSTALLATION (for Unix systems) ================================================================================ 1) Copy Makefile.conf.dist to Makefile.conf. Edit Makefile.conf and fill in the necessary parameters.
Is that how it is supposed to work with a modern build system?
comment:36 Changed 7 years ago by
There is #12103 for using MeatAxe implementation of matrix multiplication over small finite nonprime fields. But that was based on version 2.2.4, not 2.4.xx.
comment:37 followup: 39 Changed 7 years ago by
Do you have to convert the hg repository to a git repository? What we really need is (1) the directory build/pkgs/p_group_cohomology
with spkginstall
etc., and (2) a tarball containing the upstream source. Since you are the maintainer, it is up to you how to distribute that tarball, so I think if you want to distribute it with an hg repository, that's okay. If you just want to release a particular version with no repo at all included, that should be fine, too. (We don't always distribute repository information in the tarballs for other packages, for what that's worth.)
comment:38 Changed 7 years ago by
For now, by the way, I think the easiest path is the right one: use your current versions of everything, don't bother trying to use newer versions of MeatAxe, etc.
comment:39 Changed 7 years ago by
Replying to jhpalmieri:
Do you have to convert the hg repository to a git repository? What we really need is (1) the directory
build/pkgs/p_group_cohomology
withspkginstall
etc., and (2) a tarball containing the upstream source.
Is it that easy? That would be nice.
I was told somewhere that moving from hg (even with queues?) to git is not particularly difficult. I was reluctant to use git for Sage development and I still miss hg sometimes. But using git for Sage and at the same time hg for a Sage package seems awkward to me
comment:40 Changed 7 years ago by
I agree with John: try to keep things as intact as possible. It should be possible to simply detach the SPKG.txt
and sageinstall
scripts from spkg and put those (and sagecheck
and any patches) in build/pkgs/p_group_cohomology
(with a packageversion.txt
, checksums.ini
, and type
files; see any other spkg). Then run sage sh sagefixpkgchecksums
. You should put a clean tarball of the necessary source in the upstream
folder (this will be the upstream tarball that will go on the Sage mirrors), which should be everything else. Then, and only then, check that it installs (otherwise I found it deletes the tarball).
Just to be clear, I think it would be better to simply transition the spkg style here and now and consider doing upgrades later (with a version bump of your spkg).
Also have you considered bitbucket? They do hg repositories and you can keep your upstream there (and under hg control instead of git). Also they seem to be more generous than github for free (private) academic repositories.
comment:41 followup: 42 Changed 7 years ago by
So, what is the plan here? Keep the oldstyle package on this ticket or change to a newstyle package? In the latter case, the ticket should be set back to needs_work.
comment:42 Changed 7 years ago by
Status:  needs_review → needs_work 

Work issues:  → Create a newstyle package with least effort 
Replying to jdemeyer:
So, what is the plan here? Keep the oldstyle package on this ticket or change to a newstyle package?
If I understood correctly, the suggestion of John and Travis was to create a newstyle package with the least effort. I.e., take the current sources contained in the spkg and post them somewhere (aka "upstream"), and install newstylish from there.
Or perhaps a little better and still feasible: Use the *proper* existing upstream for MeatAxe (even though the package uses a very old version), and only put the rest of the sources to a new upstream.
In the latter case, the ticket should be set back to needs_work.
OK.
comment:43 Changed 7 years ago by
I have moved the documentation of the package. In case you are interested in testing an oldstyle version 2.1.5, see http://users.minet.unijena.de/cohomology/documentation/#installation
Next, I plan to create a leasteffort newstyle version 2.1.6. It will be basically identical with 2.1.5, except for the package style. And changing everything to the latest MeatAxe upstream will be considerable work taking some time, justifying a new ticket and a major new version 3.0.
comment:44 followup: 46 Changed 7 years ago by
Argh. There is more work to do. 2.1.5 installs, but doesn't work for nonprimepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.
comment:45 Changed 7 years ago by
Sigh. Apparently there is also a change in the pickling of Singular objects that I am responsible for and that I have now to cope with...
comment:46 followup: 47 Changed 7 years ago by
Replying to SimonKing:
Argh. There is more work to do. 2.1.5 installs, but doesn't work for nonprimepower groups. The error message mentions multiple values for a keyword argument. Strange, I haven't seen that before.
I guess it's because of the Cython upgrade. Earlier versions of Cython did not give an error for code like
f(foo="hello", foo="world")
but now that's an error just like in Python.
comment:47 followup: 48 Changed 7 years ago by
Replying to jdemeyer:
I guess it's because of the Cython upgrade.
Probably. That error (somewhere doing mutable = True, mutable = True
) is fixed. But the Singular pickling error is likely to be tricky.
Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.
comment:48 Changed 7 years ago by
Replying to SimonKing:
Until recently, pickling of an object in the Singular interface resulted in a pickle that couldn't be unpickled, and my spkg could cope with it. By a recent change that I authored, pickling an object in the Singular interface works more likely, BUT fails with an error if it is not defined in the current base ring. So, now I get an error, where previously I just had a partial pickle.
I'm afraid I can't help you there...
comment:49 Changed 7 years ago by
In the previous versions, I already had a class GapPickler to help with pickles of Gap interface objects (by default, their content is lost when pickling, although it can be reconstructed from the string representation in most cases). I guess I could do basically the same for other interfaces.
comment:50 Changed 7 years ago by
I replaced the spkg by a version that *almost* passes the test suite. The only remaining problem is that in the stored data the old pickling of Singular elements is used, which results in a deprecation warning.
So, my plan is: Recompute the cohomology for all groups of order 64, store them in a database, put the database into the package, and put it online.
Then, finally create a basic version of a newstyle package.
comment:51 followup: 52 Changed 7 years ago by
I created the current version 2.1.5 on my laptop and tested on the computer in my office. Most tests passed. Problems:
 It seems that since a while, web access during doctests is not possible. It used to work, and the failing tests do work in interactive sessions. No idea how that could be fixed; I guess it was caused by a change in the doctest framework.
 There have still been deprecation warnings when unpickling old pickled cohomology rings. Whether or not we will see the deprecation warning depends on the user: If (s)he has old pickles, then the tests will fail. If (s)he removes the old pickles before reinstalling the package, then the tests will work.
 There has been one error where a different homogeneous system of parameters was found. Mathematically there is no substantial difference, but it may be needed to change the expected output.
comment:52 Changed 7 years ago by
Replying to SimonKing:
 It seems that since a while, web access during doctests is not possible. It used to work, and the failing tests do work in interactive sessions. No idea how that could be fixed; I guess it was caused by a change in the doctest framework.
Strangely, these tests worked on my laptop. Is it possible that "screen" (which I use on the office computer) prevents internet access during doctests?
 There have still been deprecation warnings when unpickling old pickled cohomology rings. Whether or not we will see the deprecation warning depends on the user: If (s)he has old pickles, then the tests will fail. If (s)he removes the old pickles before reinstalling the package, then the tests will work.
Question to the reviewer: Should one add a remark in the documentation, telling that a certain deprecation warning is expected if old data is present?
 There has been one error where a different homogeneous system of parameters was found. Mathematically there is no substantial difference, but it may be needed to change the expected output.
I get the same on my laptop. Thus I changed it, and updated the package. Now I redo the tests on my office computer.
comment:54 Changed 7 years ago by
I guess I have to do some more fixes. It turns out that one can not install the spkg if before one has installed the meataxe package that I propose at #12103. Probable reason: Conflicting types in the meataxe version used here and used there.
Actually the cleanest solution would be to remove the old meataxe sources from the spkg, and rewrite the code of David Green by using the new meataxe sources. Hence, the meataxe package would become a dependency of the cohomology spkg.
But my plan was to do this (a MAJOR rewrite!) on the long run, not now. I will do some tests, to see how much work will be involved. If it merely is a cutandpaste modification (changing ALL function and type names used in meataxe) then it might be doable.
comment:55 followup: 56 Changed 7 years ago by
A more hack solution would be to make the meataxe that you include in this spkg have a different name, and so that it could be used along #12103. However, as you said, the cleanest solution would be to base this off of the meataxe spkg on #12103 as a dependency.
I would imagine that the majority of the work would just be some name/type replacements as long as you weren't doing anything too exotic...at least I doubt they made major changes to their API...
comment:56 Changed 7 years ago by
Replying to tscrim:
I would imagine that the majority of the work would just be some name/type replacements as long as you weren't doing anything too exotic...at least I doubt they made major changes to their API...
When translating my implementation of WinogradStrassen multiplication from meataxe2.2.4 to meataxe2.4.24, most of the work has indeed been cutandpaste.
However, we talk here about changing third party code (namely David Green's), and since I didn't write it I don't know if the transition will be smooth.
comment:57 Changed 7 years ago by
At least the data structure has remained basically the same (representing matrices as a contiguous block of memory organised in rows of the matrix, each byte contains up to 8 matrix coefficients, depending on the field size, and all arithmetic operations are done by table lookup).
comment:58 followups: 59 60 Changed 7 years ago by
Cc:  david.green@… added 

I am now confident that it is feasible to make the group cohomology software work with the new optional meataxe package from #12103. I managed to rewrite the standalone programs of David Green needed in the package, and the rest "should work" (TM) by searchandreplace operations (really, ALL names in meataxe have changed).
But I have questions concerning the new code structure.
The old spkg (p_group_cohomology2.1.5) consists of
 Original sources of meataxe2.2.4, together with my "fork" of meataxe,
 Cprograms written by David Green, in src/present/,
 GAPfunctions written by David Green, in src/pGroupCohomology,
 Singularfunctions written by me, in src/pGroupCohomology,
 Cython modules written by me, in src/pGroupCohomology,
 documentation, including a doc builder taken (and modified) from old Sage versions,
 a data base of cohomology rings for the groups of order 64.
For the new package, I suggest:
meataxe2.4.24
is used, as provided by the package from #12103. The Cprograms of David Green, converted to work with the new MeatAxe, will be a new optional package called
modular_resolutions1.0
(or do you prefermodres1.0
?) depending onmeataxe
, adding me as a second author, as the conversion to the new MeatAxe was very nontrivial, and I also added error handling.  The database will also be part of
modular_resolutions
.  The Cython modules will become optional extensions in the SageMath library, say,
sage.groups.modular_cohomology
, depending onmodular_resolutions
and ondatabase_gap
. In that way, the documentation would be taken care of.  Where shall the Singular and GAP functions go? Is there a natural place? There is
SAGE_LOCAL/share/singular/
for the Singular functions. But for GAP?
Remark: database_gap
is needed because of the SmallGroups library. Technically, the upcoming modular_resolutions
(or modres
) package would not depend on it; but it is needed in the cohomology computations, thus, perhaps it makes still sense to add database_gap
as a dependency of modular_resolutions
?
comment:59 followup: 61 Changed 7 years ago by
Replying to SimonKing:
or do you prefer
modres1.0
?
Use whatever name that upstream uses. If you're free to choose that, I prefer modular_resolutions
.
comment:60 followup: 62 Changed 7 years ago by
Replying to SimonKing:
 Where shall the Singular and GAP functions go? Is there a natural place? There is
SAGE_LOCAL/share/singular/
for the Singular functions. But for GAP?
About the GAP code: is it C code for GAP or GAP code? In the latter case, can you use src/ext/gap
?
comment:61 followup: 65 Changed 7 years ago by
Replying to jdemeyer:
Replying to SimonKing:
or do you prefer
modres1.0
?Use whatever name that upstream uses. If you're free to choose that, I prefer
modular_resolutions
.
Concerning "upstream": David Green is the original upstream. But as much as I know his code has never been published outside of the old spkg. And since the rebase on top of meataxe2.4.24 was major, I believe that I am to some extent upstream, too.
The original sources that David Green gave me are in a folder called "present". But I doubt that a user having a look at the list of optional packages would guess that "present" is about modular resolutions of pgroups. From my perspective, "modular_resolutions" is a bit lengthy. I will ask David, after all he is native speaker.
comment:62 Changed 7 years ago by
Replying to jdemeyer:
Replying to SimonKing:
 Where shall the Singular and GAP functions go? Is there a natural place? There is
SAGE_LOCAL/share/singular/
for the Singular functions. But for GAP?About the GAP code: is it C code for GAP or GAP code? In the latter case, can you use
src/ext/gap
?
Pure GAP code, in the files src/pGroupCohomology/GapMB
, .../GapMaxels
and .../GapSgs
.
Is src/ext/gap
somehow known to Gap? Or does one need to give the full path when loading code from there into Gap?
comment:63 followup: 64 Changed 7 years ago by
you might consider packaging your GAP code as a GAP package...
comment:64 followup: 66 Changed 7 years ago by
Replying to dimpase:
you might consider packaging your GAP code as a GAP package...
Hopefully that's a joke. I don't want to learn for yet another CAS how to contribute a package, in particular for code that I didn't author.
comment:65 Changed 7 years ago by
Replying to SimonKing:
Replying to jdemeyer:
Use whatever name that upstream uses. If you're free to choose that, I prefer
modular_resolutions
.
I got answer of David Green.
 He agrees that, by porting the code on top of the new meataxe, I became a coauthor.
 He coined his package "present", because the original purpose was to deal with presentations of modular group algebras of prime power groups.
 If I understand correctly, he wouldn't mind renaming it.
I believe that the main purpose of the code is not to compute a presentation of a modular group algebra, but to compute a minimal projective resolution of the modular group algebra. At least that's what I am using it for.
Since the old package is called p_group_cohomology and the cohomology part will be moved to the Sage library, the package's responsibility will be to compute resolutions. Perhaps (to remind the old name) p_resolution? I find modular_resolution too long, Jeroen finds modres too abbreviated (I guess).
comment:66 Changed 7 years ago by
Replying to SimonKing:
Replying to dimpase:
you might consider packaging your GAP code as a GAP package...
Hopefully that's a joke. I don't want to learn for yet another CAS how to contribute a package, in particular for code that I didn't author.
well, by package I just meant a way to package GAP files in a way understood by GAP.
Of course, you can also use GAP's Read()
command to load any GAP code from anywhere in the filesystem.
sage: gap.eval('Read("'+SAGE_LOCAL+'/foo/blah.g")') # GAP sage: libgap.Read('"'+SAGE_LOCAL+'/foo/blah.g"') # libGAP
HTH, Dima
comment:67 followup: 68 Changed 7 years ago by
by the way, would you mind renaming GAP source code files, by giving them suffix, '.g' or '.gap' ?
comment:68 Changed 7 years ago by
Replying to dimpase:
by the way, would you mind renaming GAP source code files, by giving them suffix, '.g' or '.gap' ?
No problem. And then put it into src/ext/gap, as I was told.
comment:69 followup: 71 Changed 7 years ago by
I think I need help, as I am virtually a beginner with Makefiles, compiling csources, and/or creating a library.
For creating .ofiles, I have the following rule in my Makefile, where
 VPATH is the path to the sources
CFLAGS=I$(MTXINC) I$(VPATH) Wall $(OO) g fPIC
 MTXINC is the SAGE_LOCAL/include
%.o: $(VPATH)/pgroup.h $(VPATH)/pincl.h $(VPATH)/pincl_decls.h $(VPATH)/$*.c @echo "Compiling $*.c" @$(CC) $(CFLAGS) c $(VPATH)/$*.c o $@
The above is a modification of MeatAxe's Makefile, which says
tmp/%.o: tmp/mk.dir src/%.c src/meataxe.h tmp/config.h @echo "Compiling $*.c" @$(CC) $(CFLAGS) c src/$*.c o $@
While the rule in MeatAxe's Makefile works, in my Makefile it is not executed. For example, when making pinc.o, the following is printed to stdout:
gcc I/home/king/Sage/git/sage/local/include I/home/king/Projekte/coho/p_group_cohomology3.0/src/present/src Wall g fPIC c o pincl.o /home/king/Projekte/coho/p_group_cohomology3.0/src/present/src/pincl.c
In particular, the @echo "compiling ..."
statement is not executed.
So, what is happening here? Is there an implicit make rule that takes precedence over the explicit rule??
And further, when I try to build an executable (say, makeInclusionMatrix
), the rule says
makeInclusionMatrix: libmodres.a mim.o $(CC) $(LFLAGS) o makeInclusionMatrix mim.o libmodres.a
with
LFLAGS=l$(MTXLIBNAME) L$(LIBDIR)
, MTXLIBNAME=mtx
 LIBDIR=$(SAGE_LOCAL)/lib
Building the executable, the output indeed contains
gcc lmtx L/home/king/Sage/git/sage/local/lib o makeInclusionMatrix mim.o libmodres.a
but aborts with an error, saying
/home/king/Projekte/coho/p_group_cohomology3.0/src/present/src/mim.c:51: undefined reference to `MatFree'
But MatFree
and other "undefined references" are defined in /home/king/Sage/git/sage/local/lib/libmtx.a
!
What is happening here?
comment:70 Changed 7 years ago by
When I instead do
gcc L/home/king/Sage/git/sage/local/lib o makeInclusionMatrix mim.o pgroup.o pincl.o lmtx
then it works. But why? And why did the other command not work?
The difference is that it says pgroup.o pincl.o
instead of libmodres.a
. And libmodres.a
was created with ar r libmodres.a pincl.o pgroup.o aufloesung.o
. Shouldn't then pgroup.o and pincl.o be available in libmodres.a anyway?
comment:71 followup: 72 Changed 7 years ago by
Replying to SimonKing:
I think I need help, as I am virtually a beginner with Makefiles, compiling csources, and/or creating a library.
If you're manually writing a Makefile
, you're already doing it wrong. The only way to reliably build a library is to use libtool
.
comment:72 followup: 73 Changed 7 years ago by
Replying to jdemeyer:
If you're manually writing a
Makefile
, you're already doing it wrong. The only way to reliably build a library is to uselibtool
.
OK, then MeatAxe is doing it wrong as well, but after all it is third party code and I won't touch it.
Can you point me to a good introduction to libtool? What I need to create is (a) a couple of executables and (b) a library against which I can link my cython code. So, the library is not all!
comment:73 followup: 74 Changed 7 years ago by
Replying to SimonKing:
OK, then MeatAxe is doing it wrong as well
Absolutely. Don't use the meataxe build system as an example.
Can you point me to a good introduction to libtool?
I don't know. For an example in Sage, you can look at the planarity
package.
comment:74 Changed 7 years ago by
Replying to jdemeyer:
Replying to SimonKing:
OK, then MeatAxe is doing it wrong as well
Absolutely. Don't use the meataxe build system as an example.
Can you point me to a good introduction to libtool?
I don't know. For an example in Sage, you can look at the
planarity
package.
as far as I can see, meataxe does not depend on any external libraries, right? Then it's really easy to autotoolize/libtoolize. An example of a Sage package I can suggest is csdp (well, I did this libtoolization for it :)) https://github.com/dimpase/csdp it's more complicated as it needs Lapack and Blas.
I'll email you a pdf of a readable book about autotools, so that you can get an idea what's going on. (there is no readable free online up to day text for these purposes, AFAIK).
comment:75 Changed 7 years ago by
Still I'd like to understand what went wrong in the scenario of comment:69 and comment:70. After all, before I can autotoolize it, I should probably first be able to build it.
comment:76 Changed 7 years ago by
As to what went wrong, I think gcc will not use static libraries if you just list them on the command line. You need lmodres
.
While building binaries is somewhat the same on all platforms, the process of building a shared library is very platform dependent. Hence Jeroen's very good advice about libtool.
comment:77 Changed 7 years ago by
Meanwhile I really see the point of using autotools. Perhaps the experts (i.e. you) have comments/suggestions on the following, but to me it seems to work.
src/Makefile.am
AM_CFLAGS = O3 # > Shared library lib_LIBRARIES = libmodres.a libmodres_a_SOURCES = pincl.c pgroup.c aufloesung.c # > Headers include_HEADERS = modular_resolution.h # > Executable (built from the shared library) bin_PROGRAMS = makeActionMatrices makeNontips groupInfo perm2Gap makeInclusionMatrix makeActionMatrices_SOURCES = mam.c makeActionMatrices_LDADD = $(lib_LIBRARIES) makeNontips_SOURCES = mnt.c makeNontips_LDADD = $(lib_LIBRARIES) groupInfo_SOURCES = gi.c groupInfo_LDADD = $(lib_LIBRARIES) perm2Gap_SOURCES = perm2Gap.c perm2Gap_LDADD = $(lib_LIBRARIES) makeInclusionMatrix_SOURCES = mim.c makeInclusionMatrix_LDADD = $(lib_LIBRARIES)
configure.ac
# * Autoconf * # Process this file with autoconf to produce a configure script. AC_PREREQ([2.69]) AC_INIT([modular_resolution], [1.0], [simon.king@unijena.de]) AM_INIT_AUTOMAKE LT_INIT AC_CONFIG_SRCDIR([src/fp_decls.h]) AC_CONFIG_HEADERS([config.h]) # Checks for programs. AC_PROG_CC AC_PROG_INSTALL # Checks for libraries. AC_SEARCH_LIBS([MtxError], [mtx]) # Checks for header files. AC_CHECK_HEADERS([stdlib.h string.h unistd.h meataxe.h]) # Checks for typedefs, structures, and compiler characteristics. AC_CHECK_HEADER_STDBOOL AC_C_INLINE AC_TYPE_SIZE_T # Checks for library functions. AC_FUNC_MALLOC AC_CONFIG_FILES([Makefile src/Makefile]) AC_OUTPUT
So far I only see a single change needed: My email address will very soon change.
comment:78 Changed 7 years ago by
Can you help me to solve the following problem, please?
I would like to add a little test script that relies on a file test.reg that can be found in the same directory as modular_resolution.h.
In configure.ac, I have AC_CONFIG_SRCDIR([src/modular_resolution.h])
When I go to a new directory and do
/path/to/sources/of/modular_resolution/configure prefix=`pwd`
then make check
works. The file test.reg is sought in /path/to/source/of/modular_resolution/src, where it can indeed be found.
However, when I do make distcheck
(which is a target provided by autotools), then first the distdir is created, and in distdir/_build the package is built and tested. Building it works fine. However, the test fails, because this time test.reg is sought in ../../src
. Apparently the test is executed in distdir/_build, not in distdir/_build/src. Thus, it should be sought in ../src
.
In other words, when doing "make check", an absolute path is used, whereas in "make distcheck" a wrong relative path is used.
How can that be fixed?
comment:79 followup: 80 Changed 7 years ago by
Use $(srcdir)
to refer to the source directory in Makefile.am
Another thing: avoid AM_CFLAGS = O3
. Just pass whatever flags you want for Sage to configure
.
comment:80 Changed 7 years ago by
Replying to jdemeyer:
Use
$(srcdir)
to refer to the source directory inMakefile.am
I did use it. Meanwhile I could fix the problem: the test script was in fact passing a wrong command line argument to some executable.
comment:81 followup: 82 Changed 7 years ago by
How to indicate that the packagetobe depends both on meataxe and I guess as well on some toolchain? I see the following in some dependencies
files:
$(INST)/$(SAGE_MP_LIBRARY) $(INST)/$(ZLIB)  All lines of this file are ignored except the first. It is copied by SAGE_ROOT/build/make/install into SAGE_ROOT/build/make/Makefile.
or
$(INST)/$(MARKUPSAFE) $(INST)/$(SETUPTOOLS) $(INST)/$(DOCUTILS)  All lines of this file are ignored except the first. It is copied by SAGE_ROOT/build/make/install into SAGE_ROOT/build/make/Makefile.
The developer manual doesn't seem to mention it.
comment:82 Changed 7 years ago by
Meanwhile I have a version of the Cpart of the old group cohomology package that installs fine and even passes a selftest (namely tests that several of the installed executables work together to build the basic data for the cohomology computation of the dihedral group of order 8).
However, so far I have no idea how to make it so that the package is only built if meataxe (from #12103) is installed.
And then of course I have to put all Singular and Gap files with the appropriate file name extensions into the appropriate folders, and put the Cython and Python code into the Sage library, as an optional extension.
If I understand correctly, it is needed to add "optional: modular_resolution" basically to each single line of test (several hundreds, I guess). Is there really no way to state that a whole file should only be tested optionally? I believe it is a fairly natural thing to test an optional extension optionally.
comment:83 Changed 7 years ago by
Branch:  → u/SimonKing/upgrade_of_group_cohomology_spkg 

comment:84 followup: 86 Changed 7 years ago by
Commit:  → ad6cea059d383bbb0a97c8430caca125d2ca78b7 

I have extracted the second standalone ingredient of the group cohomology package.
 First, install meataxe (see #12103).
 Second, get the branch from here.
 Third, get the sources for modular_resolution1.0. It is based on the original source code of David Green, which I refactored rather extensively: It now uses the optional meataxe2.4.24 package, rather than meataxe2.2.4 sources that used to be part of the old spkg. It propagates errors (at least to some extent). And it is autotoolized. I am upstream, and you find the tarball at http://users.minet.unijena.de/cohomology/ (To be precise: http://users.minet.unijena.de/cohomology/modular_resolution1.0.tar.bz2).
The next step will be to relocate all the Cython and Python source files from the old package into the SageMath library, say, sage.groups.modular_cohomology.
And the final step will be to fix the doctests somehow. I think there should be a way to declare that the tests of all files in sage.group.modular_cohomology are optional.
Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the tobecreated wrapper in the Sage library. Since database_gap is nonGPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages? Or better move the dependency to the wrapper?
Last 10 new commits:
c7d75fb  Implement and use StrassenWinograd matrix multiplication in MeatAxe

9e2a8c6  A very basic MeatAxe Cython wrapper

4bdd285  A full wrapper for MeatAxe matrices

d889e0b  Improve echelon computation in MeatAxe, and fix some compiler warnings

2e64257  Doctests and error handling for MeatAxe

7c38969  Fix computation of rowreduced echelon form

55a278d  Fix doctests when meataxe is installed

f733377  Use and propagate specific return values on error in matrixrelated MeatAxe functions.

efa1d75  Merge branch 't/12103/meataxe' into t/18514/upgrade_of_group_cohomology_spkg

ad6cea0  Add package modular_resolution

comment:85 Changed 7 years ago by
Description:  modified (diff) 

Please update the ticket description (the last comment you made is a good starting point).
comment:86 followup: 87 Changed 7 years ago by
Replying to SimonKing:
Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the tobecreated wrapper in the Sage library. Since database_gap is nonGPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages?
No, that's not possible within the dependency framework. Also, I personally don't think it's a good idea to add this kind of interactive install procedures.
Or better move the dependency to the wrapper?
The wrapper will become part of the Sage library, right? So what do you mean with "move the dependency to the wrapper"?
comment:87 followup: 88 Changed 7 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Question to the reviewer: I made the new package depend on database_gap. In fact, what really depends on database_gap is only the tobecreated wrapper in the Sage library. Since database_gap is nonGPL, I think it should only be installed (as a dependency) when the user agrees. Is that possible with the dependency framework of packages?
No, that's not possible within the dependency framework. Also, I personally don't think it's a good idea to add this kind of interactive install procedures.
Then we might be in trouble. If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a nonGPL package without asking the user.
Perhaps I should elaborate on the dependency on database_gap. The code does not link against database gap. However, it needs to be able to deal with elementary abelian groups. Sounds trivial, and it can certainly be solved abstractly, but for now the elementary abelian group is simply looked up in the small groups library.
Also, since the p_group_cohomology package is some kind of data base for cohomology rings, it is needed to assign a unique identifier to a groupwithafixedminimalgeneratingset. That is easier with the small groups library, but the package also allows to assign a name to a group (e.g., Co_3 for the third Conway group), and use the name as unique identifier for some item in the cohomology ring database (the fixed minimal generating set is stored along with the ring).
Or better move the dependency to the wrapper?
The wrapper will become part of the Sage library, right? So what do you mean with "move the dependency to the wrapper"?
Similarly to the MeatAxe wrapper, I plan to use OptionalExtension()
. Hence, the Cython code would only result in an extension module under the condition that both modular_resolution and database_gap are installed.
So, if the current framework does not allow that installation of modular_resolutions stops for asking the users permission to install database_gap first, then we can make it so that modular_resolution only makes sure that meataxe is installed (no problem, both are GPL compatible, and modular_resolution will build and test fine), and then the OptionalExtension()
will depend on both modular_resolution *and* database_gap.
comment:88 followup: 90 Changed 7 years ago by
Replying to SimonKing:
If I understood correctly, the GPL licence does not allow that a GPL package (i.e. modular_resolution) installs a nonGPL package without asking the user.
Where did you get this idea from?
comment:89 Changed 7 years ago by
Description:  modified (diff) 

comment:90 followup: 92 Changed 7 years ago by
comment:91 Changed 7 years ago by
Description:  modified (diff) 

comment:92 Changed 7 years ago by
Replying to SimonKing:
From discussions on sagedevel.
I recall that discussion and I asked exactly the same question. I don't think that was ever answered. I think it was even mentioned that we should only ask this question as a courtesy to the user, not that it was a requirement.
comment:93 followup: 94 Changed 7 years ago by
Replying to SimonKing:
Similarly to the MeatAxe wrapper, I plan to use
OptionalExtension()
. Hence, the Cython code would only result in an extension module under the condition that both modular_resolution and database_gap are installed.
Surely, the module will compile without database_gap
. So I think you should just add meataxe
as dependency for the Cython module.
The need of database_gap
will make it fail at runtime. That's no problem. It's even more userfriendly, since you can show a nice error message at runtime if database_gap
is not installed.
comment:94 followup: 95 Changed 7 years ago by
Replying to jdemeyer:
The need of
database_gap
will make it fail at runtime. That's no problem. It's even more userfriendly, since you can show a nice error message at runtime ifdatabase_gap
is not installed.
OK, sounds good, except for doctests: It means that all tests in the OptionalExtension
need to be marked as optional: modular_resolution database_gap
, which is rather awkward to do hundreds of times. Unless of course one can somehow declare that ALL tests of a file are optional.
comment:95 followup: 96 Changed 7 years ago by
Replying to SimonKing:
It means that all tests in the
OptionalExtension
need to be marked asoptional: modular_resolution database_gap
, which is rather awkward to do hundreds of times.
Well, it's not more awkward than just optional: modular_resolution
hundreds of times :)
Unless of course one can somehow declare that ALL tests of a file are optional.
I think this is intentionally not supported. A doctest should make sense by itself, not depend on external context like the file it appears in.
comment:96 Changed 7 years ago by
Replying to jdemeyer:
I think this is intentionally not supported. A doctest should make sense by itself, not depend on external context like the file it appears in.
Does a cohomology computation make sense if your CAS doesn't know about cohomology??
comment:97 Changed 7 years ago by
Dependencies:  #18494 → #12103 

comment:98 followup: 99 Changed 7 years ago by
It turns out that I need to expose more stuff in modular_resolution.h. My wrapper in the old spkg imports functions from various headers but I suppose it is nicer if modular_resolution1.0 only installs a single header, modular_resolution.h, and not in addition nDiag.h etc.
comment:99 followup: 100 Changed 7 years ago by
Replying to SimonKing:
It turns out that I need to expose more stuff in modular_resolution.h. My wrapper in the old spkg imports functions from various headers but I suppose it is nicer if modular_resolution1.0 only installs a single header, modular_resolution.h, and not in addition nDiag.h etc.
Or should I better create a folder SAGE_LOCAL/include/modular_resolution/
, where *all* headers of the package will be put?
comment:100 Changed 7 years ago by
comment:101 Changed 7 years ago by
Now I have a technical problem: I can not scp the new tarball to my account in Jena. rcp doesn't work (I suppose Jena is blocking it), scp doesn't work (my landlord is blocking ssh).
Do you have a better solution than to instrument sagemathcloud as a tool to scp the file?
comment:102 Changed 7 years ago by
It doesn't work via sagemathcloud either:
ssh: connect to host login.minet.unijena.de port 22: Network is unreachable
comment:103 Changed 7 years ago by
as a last resort, share it on sagemathcloud, and we'll put it somewhere.
comment:104 Changed 7 years ago by
I need to work on the package a bit more anyway: As it turns out, I didn't put into the library what I want to import later. So, I can post the new package tomorrow, when I am at university.
comment:105 followup: 106 Changed 7 years ago by
Odd. When I try to use the resulting library in Sage, I get the complaint
/usr/lib64/gcc/x86_64suselinux/4.8/../../../../x86_64suselinux/bin/ld: /home/king/Sage/git/sage/local/lib/libmodres.a(pgroup.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with fPIC /home/king/Sage/git/sage/local/lib/libmodres.a: error adding symbols: Bad value collect2: error: ld returned 1 exit status error: command 'gcc' failed with exit status 1
Is it not the case that autotools automatically takes care of fPIC?
comment:106 Changed 7 years ago by
Replying to SimonKing:
Is it not the case that autotools automatically takes care of fPIC?
Apparently not. OK, I simply have to add it to Makefile.am.
comment:107 followup: 110 Changed 7 years ago by
Autotools by itself doesn't care about pic (since it doesn't know anything about shared libraries, really). Libtool does the right thing, though.
Also, I'm confused: libmodres.a is a static library, but the ld error message refers to shared library creation.
comment:108 Changed 7 years ago by
Commit:  ad6cea059d383bbb0a97c8430caca125d2ca78b7 → 499dfbd2a4a25d3ab2f089d36e0d592ba6a0d99e 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
499dfbd  Add package modular_resolution

comment:109 Changed 7 years ago by
I have updated the tarball, consequently updated the checksum, and added a .pxd header for the package. The package contains a test ("Out of given basic data for the dihedral group of order 8, compute some additional data, and test that these data are readable and contain a certain result").
A little sanity test that I did:
sage: cython("""#!clib modres mtx ....: from sage.libs.modular_resolution cimport * ....: def test(): ....: cdef group_t *G = newGroupRecord() ....: freeGroupRecord(G) ....: """) sage: test()
comment:110 Changed 7 years ago by
Replying to vbraun:
Also, I'm confused: libmodres.a is a static library, but the ld error message refers to shared library creation.
Good point. Out of lack of experience I have no preference for either static or dynamic libraries (I heard that the former can be a little faster, but not by much). And I am really not sure if my Makefile.am results in a dynamic or a static library:
lib_LIBRARIES = libmodres.a libmodres_a_SOURCES = aufloesung.c aufnahme.c fileplus.c nBuchberger.c pgroup.c pincl.c slice.c urbild.c libmodres_a_CFLAGS = fPIC
I guess it is dynamic. But how to create a static library (if that's desired)?
comment:112 Changed 7 years ago by
If I should change to creating a static library then please tell me how, and if you say that I am creating the dynamic library in the wrong way then please tell me how to do better.
comment:113 followup: 114 Changed 7 years ago by
Always prefer shared libraries over static libraries
With libtool the Makefile.am should contain
lib_LTLIBRARIES = libmodres.la libmodres_la_SOURCES = aufloesung.c aufnahme.c fileplus.c nBuchberger.c pgroup.c pincl.c slice.c urbild.c
Are the upstream meataxe filenames already in German?
comment:114 Changed 7 years ago by
Replying to vbraun:
Always prefer shared libraries over static libraries
OK.
With libtool the Makefile.am should contain
lib_LTLIBRARIES = libmodres.la libmodres_la_SOURCES = aufloesung.c aufnahme.c fileplus.c nBuchberger.c pgroup.c pincl.c slice.c urbild.c
So, the difference is that it is lib_LTLIBRARIES, not lib_LIBRARIES, and that the file name extension is .la, not .a.
What about fPIC? Well, I will see if it is still needed.
Are the upstream meataxe filenames already in German?
Don't mix the two things ;)
 Meataxe is at #12103, the filenames are English, I only added one file window.c (thus, English as well), and I will *not* do upstream's job of autotoolisation.
 Here is modular_resolution, which was originally written by a native English speaker in Wuppertal, and he chose the filenames. By rebasing it on top of the new meataxe and by using autotools, I became part of upstream, but I didn't change the filenames.
After all, SageMath is an international project...
By the way, I am really glad that you guys convinced me of using autotools. It really is so much easier!
comment:115 followups: 118 119 Changed 7 years ago by
There is one more file whose new location has to be determined: GroupNames.sobj
. Thus, it is a Sage pickle, and it contains a dictionary assigning humanreadable names (such as DihedralGroup(8)) to some addresses from the small groups library (such as SmallGroup(8,3)).
So, questions:
 What is a natural location for a data file that is part of the SageMath distribution?
 Do you happen to know a Gap function that can provide humanreadable names (and other useful information) for a wider range of groups?
comment:116 Changed 7 years ago by
Commit:  499dfbd2a4a25d3ab2f089d36e0d592ba6a0d99e → cd644a301bd2a9c57b5c719bb47a4733c75e347b 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
cd644a3  Add package modular_resolution

comment:117 Changed 7 years ago by
I have posted an updated version of the upstream tarball. After changing from lib_LIBRARIES
to lib_LTLIBRARIES
, using fPIC was not needed any longer.
Now, I try to move the Cython wrappper.
comment:118 Changed 7 years ago by
Replying to SimonKing:
 What is a natural location for a data file that is part of the SageMath distribution?
Perhaps I should add that the old spkg used several default locations for data.
SAGE_SHARE/pGroupCohomology
for data that are obtained during installation of the package, which are shared among different users of one Sage installation, and which are not supposed to be modified by a user, unless (s)he knows what (s)he is doing and has write permission.DOT_SAGE/pGroupCohomology
for data that are created by a user, and which are only available to this user.
Is SAGE_SHARE/pGroupCohomology
still a good location for shared data? If that is the case, then I think GroupNames.sobj
should go there. But how? If I understand correctly, SAGE_SHARE is not under version control. Is it a possibility to add these data files in the modular_resolution tarball, and modifying its Makefile so that the data are installed in SAGE_SHARE/pGroupCohomology
?
comment:119 followup: 120 Changed 7 years ago by
Replying to SimonKing:
There is one more file whose new location has to be determined:
GroupNames.sobj
. Thus, it is a Sage pickle
Does it really have to be a pickle? Why not just a Python file?
comment:120 followup: 121 Changed 7 years ago by
Replying to jdemeyer:
Does it really have to be a pickle? Why not just a Python file?
Well, I could also put it directly into the code (the file is loaded in exactly one spot in the cohomology code). So, right, I could get rid of the pickle.
But can the shared database remain in SAGE_SHAR/pGroupCohomology? I think it is reasonable that the new modular_resolution package shall provide the cohomology ring data base, even though the actual code to access the data will only be in the Sage library.
comment:121 followup: 122 Changed 7 years ago by
Replying to SimonKing:
But can the shared database remain in
SAGE_SHARE/pGroupCohomology
?
Of course. Just as an example, the conway_polynomials
database is also in $SAGE_SHARE
.
comment:122 Changed 7 years ago by
Replying to jdemeyer:
Of course. Just as an example, the
conway_polynomials
database is also in$SAGE_SHARE
.
Hrmph. I need more help.
I created a data/ directory and put the untar'd cohomology data into that directory. Thus, data/ now has 268 subdirectories, namely 8gp3/ and 64gp1/ up to 64gp267/. The aim is to copy all these subdirectories into $(prefix)/share/pGroupCohomology (and to create $(prefix)/share/pGroupCohomology in the first place).
What I tried is to put the following into data/Makefile.am:
dbdir = $(prefix)/share/pGroupCohomology nobase_db_DATA = $(wildcard *gp*)
expecting that "make install" would create the directory and then recursively (because of nobase_) copy all folders *gp* into it.
However, autoreconf complains that wildcard *gp*: nonPOSIX variable name
, and when I do make install, then there is no share/pGroupCohomology
, and from the log I don't see any attempt to install any data.
What do I need to do instead?
comment:123 followup: 124 Changed 7 years ago by
Perhaps I should have a single tar file containing all the data, and then add an installdatahook that untars the data file?
comment:124 Changed 7 years ago by
Replying to SimonKing:
Perhaps I should have a single tar file containing all the data, and then add an installdatahook that untars the data file?
It works *almost*: I can do make install and make clean and make uninstall and so on, but nonetheless make distcheck fails. And I don't understand why.
I have this in the toplevel Makefile.am:
ACLOCAL_AMFLAGS = I m4 SUBDIRS = src dbdir = $(datadir)/pGroupCohomology dist_db_DATA = group_cohomology_database.tar installdatahook: cd $(dbdir) && tar xf $(dist_db_DATA) uninstallhook: rm r $(dbdir) cleanlocal: rm $(dbdir)/$(dist_db_DATA)
When I run make distcheck, then the command cd $(dbdir)
in installdatahook fails. The odd reason: datadir
is /home/king/Projekte/coho/bin/modular_resolution1.0/_inst/share
(I checked that), but during "make distcheck" the data are installed NOT in datadir
but in /tmp/amdc11743//home/king/Projekte/coho/bin/modular_resolution1.0/_inst/share/pGroupCohomology
. Consequently, when I want to cd into dbdir, the folder can't be found.
Do you have an idea why distcheck installs data not into datadir but into a temporary folder, whereas make install correctly installs into datadir?
comment:125 Changed 7 years ago by
Got it. make distcheck uses a staged install, using the DESTDIR variable. And I somewhere found the comment: "Support for DESTDIR is implemented by coding it directly into the install rules. If your Makefile.am uses a local install rule (e.g., installexeclocal) or an install hook, then you must write that code to respect DESTDIR."
comment:126 Changed 7 years ago by
Commit:  cd644a301bd2a9c57b5c719bb47a4733c75e347b → 9999966fb6d48fa4516e7d11220efab69ed457e2 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
9999966  Add package modular_resolution

comment:127 Changed 7 years ago by
Interesting commit number: 9999966...
Anyway. I have again updated the upstream tarball. Now, it contains the database of the modular cohomology rings of all groups of order 64 that is part of the old spkg. Recall that a much larger database is not part of the tarball but is available in internet.
And of course it will be needed to have code in the Sage library for actually reading the data. But I hope that's ok.
comment:128 followup: 129 Changed 7 years ago by
I can easily put the Gap code into $SAGE_ROOT/src/ext/gap/modular_cohomology/: It is under version control.
However, $SAGE_ROOT/local/share/singular is not under version control. Does that mean I should better use a different location for the Singular code?
comment:129 followup: 130 Changed 7 years ago by
Replying to SimonKing:
Singular code?
If it's Singular code, it should go into src/ext/singular
by analogy with src/ext/gap
and src/ext/pari
.
comment:130 Changed 7 years ago by
comment:131 followup: 132 Changed 7 years ago by
A question about copyright statements: When I put all the Cython and Python files into the Sage library, should I preserve the old copyright statements such as Copyright (C) 2009 Simon A. King <simon.king@unijena.de>
, or should I add new ones, i.e. Copyright (C) 2015 Simon A. King <simon.king@unikoeln.de>
?
comment:132 Changed 7 years ago by
Replying to SimonKing:
A question about copyright statements: When I put all the Cython and Python files into the Sage library, should I preserve the old copyright statements such as
Copyright (C) 2009 Simon A. King <simon.king@unijena.de>
, or should I add new ones, i.e.Copyright (C) 2015 Simon A. King <simon.king@unikoeln.de>
?
well, it's good to have an up to date contact info there, while keeping the copyright dates, such as Copyright (C) 2009, 2015 Simon A. King <simon.king@unikoeln.de>
comment:133 Changed 7 years ago by
Work issues:  Create a newstyle package with least effort → Debug the modification of the modular resolution code 

comment:134 Changed 7 years ago by
Information on "Debug the modification of the modular resolution code": The old meataxe had row and column numbers onebased, the new has it zerobased. It seems to me that in some locations of the modular resolution code, this needs to be coped with.
comment:135 Changed 6 years ago by
The work is almost done, but according to discussions on sagedevel I should not proceed according to the original ticket description. Hence, in this comment, I explain why, and then I will change the ticket description.
Replying to SimonKing:
The current "official" version of
p_group_cohomology
is 2.1.4. However, due to recent backward incompatible changes in Sage, the package would not install, respectively it would install if some header were present but wouldn't work. Note that such backward incompatible changes happened repeatedly.
And it has happened again: Meanwhile I offer an oldstyle spkg version 2.1.6, and both new versions do nothing but coping with backward incompatible changes.
The new package shall be modularised as follows.
 First, install meataxe (see #12103) and database_gap.
This is still the case  based on meataxe, there is an alternative implementation of matrices in the Sage src tree, and that will be used
Note that soon I will create a ticket to add a few methods to the meataxe wrapper that I intend to use in my cohomology programs.
 Third, get the sources for modular_resolution1.0. It is based on code of David Green, which I refactored rather extensively:
... The next step will be to relocate all the Cython and Python source files from the old package into the SageMath library, say, sage.groups.modular_cohomology.
I was told on sagedevel to not move the code to the Sage src tree, but make it an external package that depends on the meataxe spkg, and is formed by two parts: The first is a C library together with some executables provided by what above was called modular_resolution1.0, and a pipinstallable package formed by cython and python files. Reasons:
 My package adds functionality, but doesn't modify existing functionality.
 In order to use it, it isn't needed to cimport from my package.
 People wouldn't like to have more than 10,000 lines of code in the src tree that is useless without some optional package.
 There currently is no easy way to make all doctests in one file optional.
That said, I still believe that for me working in the src tree would be better; for the record:
 Currently I do the development in the src tree anyway. So, moving stuff from the src tree into an external package is a nontrivial additional step for me.
 Building the docs of python/cython code outside of the src tree is a nontrivial additional step for me.
 Being in the src tree would make it less likely that backward incompatible Sage changes breaking my package will happen unnoticed again.
 Having the documentation of my optional package in the Sage docs would make more people aware, so that they would be more likely to use my programs in research.
comment:136 Changed 6 years ago by
Branch:  u/SimonKing/upgrade_of_group_cohomology_spkg 

Commit:  9999966fb6d48fa4516e7d11220efab69ed457e2 
Description:  modified (diff) 
Work issues:  Debug the modification of the modular resolution code 
comment:137 Changed 6 years ago by
My oldstyle package does contain a setup.py for installing the cython and python code, and it is able to build a documentation (basically by a modified copy of an old version of Sage's docbuilder). However, very likely both is not how it should be done.
I have three requests to the reviewers:
 Please point me to a good example of a setup.py in a newstyle spkg for installing Cython code that cimports Sage types (such as Element and Parent).
 Please point me to a good example of a newstyle spkg that teaches me how to build a nice documentation based on docstrings.
 Please tell me what I need to do in order to have the cohomology spkg automatically rebuilt as soon as the cimported Sage types change.
comment:138 Changed 6 years ago by
Dependencies:  #12103 → #12103 #21437 

I need some modifications to the MeatAxe wrapper, and I'd appreciate it to be reviewed tat #21437.
comment:139 Changed 6 years ago by
Branch:  → u/SimonKing/upgrade_of_group_cohomology_spkg 

comment:140 Changed 6 years ago by
Commit:  → f1c0dbd51d0cab48b1c21f3faca9c7e2245764e4 

I guess it is better to provide code at some point. There are many doctest failures (many of them just because of different logging), none of them is marked optional, but as much as I see it would be possible to do nontrivial computations with the new code version.
INSTALLATION
 Put http://users.minet.unijena.de/cohomology/modular_resolution1.0.tar.gz into SAGE_ROOT/upstream/
 Pull the git branch
 Do
sage i meataxe
andsage i modular_resolution
 Do
make
HOWEVER...
The current branch does everything in Sage's src tree, which (as I was told) is evil.
But perhaps some of you can answer the questions that I have posted earlier: What do I need to do with the current code base in order to create a pipinstallable module and build its documentation?
Last 10 new commits:
93d59dc  Use Python's logging module for logging

a6144a3  Add resetfunction, fix some tests

09e76a6  New return format of verifyGroupIsAbelian; use some boilerplate

16c7912  Replace MatEchelonize by full echelon computation; use set_unsafe_int resp. set_unsafe more carefully

36bcb8e  fix various doctests

2cc1070  Fix MTX matrix slicing; fix doctest continuation

d900786  Add docs to the reference manual. Remove Ordinals(), fix one occurence of get_slice.

d384968  Use boilerplate instead of set_unsafe_int, since this is affected from meataxe being static library

8fbaad9  _reset() now involves the logger. Fix some tests

f1c0dbd  Fix usage of some matrix related functions

comment:141 Changed 6 years ago by
Commit:  f1c0dbd51d0cab48b1c21f3faca9c7e2245764e4 → 710f6f5848226bffd0895ed30f93c46b421c8460 

comment:142 followups: 143 144 201 Changed 6 years ago by
There are good news:
I have just finished unit_test_64
, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.
The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i55200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.
Out of curiosity, I will repeat the test on the same machine with the old package version, and see how long it takes.
comment:143 Changed 6 years ago by
Replying to SimonKing:
Out of curiosity, I will repeat the test on the same machine with the old package version, and see how long it takes.
It takes roughly the same time. Actually the old package was two minutes faster.
comment:144 followup: 146 Changed 6 years ago by
Replying to SimonKing:
I have just finished
unit_test_64
, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i55200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.
Since the old version was a little faster, I did some profiling. The profiler told me that much time is spent (both with the old and the new version) on do_system
. With the help of sagedevel, I could track it down to several calls to system()
, which all turned out to be useless.
After removing all the system calls, the performance has drastically improved! On the same machine, unit_test_64
now only needed 9:26 minutes wall time resp. 8:37 min CPU time, which is only about one third resp. one half of the old timing, and I guess a new world record for the computation of the modular cohomology rings of all groups of order 64.
I currently do not have the bandwidth to post the new version of modular_resolution1.0.tar.gz, and will do so on Monday.
comment:145 Changed 6 years ago by
Commit:  710f6f5848226bffd0895ed30f93c46b421c8460 → 0eed64d245f15296a3beda5535cabd3c417ed25c 

Branch pushed to git repo; I updated commit sha1. New commits:
966ec0e  Fix documentation of unit_test_64

0aa2980  Update modular_resolution1.0, removing all system calls

3e130ae  Always try to save the basic ring setup on disc

d11e678  Fix attribute reconstruction, to make unit_test_64 work with 'nosave'

e5f88a4  Improve handling of global options

d9ffcf4  Various small speedups

e17e4df  Document and test _rowlist_()

291f4dc  Some minor tweaks for performance

0eed64d  Merge branch 't/21437/fixmtxmatrices' into t/18514/upgrade_of_group_cohomology_spkg

comment:146 Changed 6 years ago by
Replying to SimonKing:
Replying to SimonKing:
I have just finished
unit_test_64
, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i55200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.
Since the old version was a little faster, I did some profiling. The profiler told me that much time is spent (both with the old and the new version) on
do_system
. With the help of sagedevel, I could track it down to several calls tosystem()
, which all turned out to be useless.After removing all the system calls, the performance has drastically improved! On the same machine,
unit_test_64
now only needed 9:26 minutes wall time resp. 8:37 min CPU time, which is only about one third resp. one half of the old timing, and I guess a new world record for the computation of the modular cohomology rings of all groups of order 64.I currently do not have the bandwidth to post the new version of modular_resolution1.0.tar.gz, and will do so on Monday.
Done.
And in addition I obtained some further minor improvements, basically by using a C profiler. The computation of the modular cohomology rings of all groups of order 64 on my laptop now takes about 6:50 min of wall time resp. 6:10 min of CPU time (plus the time spent by Singular and GAP, which counts for the wall time but not for the CPU time).
I started to do time series to optimise the default choice of some options, but the standard deviation is considerable. So, I guess I'll stick with the C profiler...
Let me come back to my question: How can I move this away from the src tree and make it an independent package while being able to build a nice documentation and to run doc tests? Note that all this did work in the old spkg, but certainly can be done a lot better.
comment:147 Changed 6 years ago by
Commit:  0eed64d245f15296a3beda5535cabd3c417ed25c → 1c43593103884c96021568553cf37d2b3884bf3a 

Branch pushed to git repo; I updated commit sha1. New commits:
1c43593  Sanitise handling of global options

comment:148 Changed 6 years ago by
Commit:  1c43593103884c96021568553cf37d2b3884bf3a → 54e1a0071d54fe3bc6189e2faa3af5d956b67f2a 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
aa1985a  Fix tests in resolution.pyx

5554cef  Use SageMath's framework for _richcmp_ of elements

fbbdff9  Use boilerplate to avoid transformation of matrix rows into lists

77bdc99  Cache whether the group is abelian

3ce31d4  Remove more instances of _rowlist_

d04b020  update of modular_resolution

8d2c563  Change modular_resolution to a more efficient monomial ordering

e8759a7  Documentation of new functions. Fix doctest in cochain.pyx

3413fad  Fixing almost all tests in cohomology.pyx. Change Simon King's address

54e1a00  Remove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx

comment:149 Changed 6 years ago by
Description:  modified (diff) 

Status:  needs_work → needs_info 
If MeatAxe was a standard spkg (which might actually be something to consider) and the SmallGroups library was GPL compatible (something which the authors refuse), I could promote the modular group cohomology computation as a new standard package: With the changes that I just pushed, all tests pass (on my machine at least), and moreover the computations have still become faster.
I have changed the status to "needs info", stating my information request in the ticket description.
comment:150 Changed 5 years ago by
Replying to SimonKing:
 Please point me to an example of a new style spkg that builds a C library and Cython modules (not in Sage's src tree) that link against the library.
This is most easily treated as two separate things (the C library + the Cython wrapper). There are plenty examples for each of those two. Simple examples are https://github.com/graphalgorithms/edgeadditionplanaritysuite for the C library and https://github.com/defeo/cypari2 for the Cython wrapper.
 Please point me to an example of a new style spkg that builds documentation based on doc strings in the Cython code.
https://github.com/sagemath/cysignals (result: http://cysignals.readthedocs.io/en/latest/)
See also http://opendreamkit.org/2017/06/09/CythonSphinx/
These are all standard Sage packages (planarity
, cypari
and cysignals
)
comment:151 Changed 5 years ago by
Dependencies:  #12103 #21437 → #12103 #21437 #23352 

comment:152 Changed 5 years ago by
Commit:  54e1a0071d54fe3bc6189e2faa3af5d956b67f2a → 63355a85360616cff0d4befb3d5e0c78b4d129e5 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
08b1724  Use SageMath's framework for _richcmp_ of elements

b3156e4  Use boilerplate to avoid transformation of matrix rows into lists

883dae6  Cache whether the group is abelian

c80d8a5  Remove more instances of _rowlist_

6100c34  update of modular_resolution

c7a1ae9  Change modular_resolution to a more efficient monomial ordering

33bcb59  Documentation of new functions. Fix doctest in cochain.pyx

eb3b6e2  Fixing almost all tests in cohomology.pyx. Change Simon King's address

8f7001a  Remove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx

63355a8  Get the cohomology code to build

comment:153 Changed 5 years ago by
Commit:  63355a85360616cff0d4befb3d5e0c78b4d129e5 → ae029297d600471d77f3de795c9ad8d726988a3b 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
d0102a4  update of modular_resolution

919d7f7  Change modular_resolution to a more efficient monomial ordering

15464b4  Documentation of new functions. Fix doctest in cochain.pyx

72837b7  Fixing almost all tests in cohomology.pyx. Change Simon King's address

67b3ccf  Remove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx

5360988  Get the cohomology code to build

aa07e43  Modified handling of MtxLibDir

32a411d  Use a fixed copy of MtxLibDir

55ddf00  More clearly document from_scratch option

ae02929  Use meataxe_init() instead of manually setting the parameters

comment:154 followup: 155 Changed 5 years ago by
Dependencies:  #12103 #21437 #23352 → #21437 

Now I guess the meataxe problems are basically dealt with in #23352, #23400, #23399 and #21437. Therefore I have rebased the current cohomology code.
The current layout is:
 MeatAxe is a separate optional package
 The C code of David Green to compute minimal resolutions is a separate package, which also installs the data base of cohomology rings of all groups of order 64 and which also installs certain GAP and Singular code interpreted modules.
 The Python and Cython code is in the Sage source tree. The Cython code is optional extension module, which needs the package from 2. as a build time dependency. Moreover, at run time, it depends on the small groups library.
What layout should there be, instead? I guess MeatAxe has to stay separate. But shall everything else be included in one optional (newstyle) spkg? So, that spkg would provide 2. and 3.?
comment:155 followup: 156 Changed 5 years ago by
Replying to SimonKing:
Now I guess the meataxe problems are basically dealt with in #23352, #23400, #23399 and #21437. Therefore I have rebased the current cohomology code.
The current layout is:
 MeatAxe is a separate optional package
 The C code of David Green to compute minimal resolutions is a separate package, which also installs the data base of cohomology rings of all groups of order 64 and which also installs certain GAP and Singular code interpreted modules.
 The Python and Cython code is in the Sage source tree. The Cython code is optional extension module, which needs the package from 2. as a build time dependency. Moreover, at run time, it depends on the small groups library.
Apparently, the Cython code can not stay in the source tree, as it doesn't qualify as optional extension module.
Possible layouts:
 Two packages
 meataxe (as it is now)
 everything else put into one upstream tarball, giving rise to one newstyle spkg
 Three packages
 meataxe (as it is now)
 C code in one upstream tarball, giving rise to a newstyle spkg "modular_resolution" that provides a C library and some executables but would probably be of no interest on its own?
 Cython and Python code in another upstream tarball, giving rise to another newstyle spkg "p_group_cohomology" that depends on "modular_resolution".
Would it be OK to have an spkg "modular_resolution" if it basically has no other purpose than being a dependency for "p_group_cohomology"?
And I still wonder what to do with the GAP code and the Singular code. On the one hand, such code clearly belongs to src/ext/gap respectively to src/ext/singular. On the other hand, an spkg is not supposed to inject code into src/ext.
Is it ok to put the relevant files into src/ext even when the optional packages are not installed? I would think so; after all, the files I am talking about are not particularly large.
comment:156 followup: 157 Changed 5 years ago by
Replying to SimonKing:
Apparently, the Cython code can not stay in the source tree, as it doesn't qualify as optional extension module.
Can you elaborate a bit more on this? I don't quite understand why not from a quick glance.
Possible layouts:
 Two packages
 meataxe (as it is now)
 everything else put into one upstream tarball, giving rise to one newstyle spkg
 Three packages
 meataxe (as it is now)
 C code in one upstream tarball, giving rise to a newstyle spkg "modular_resolution" that provides a C library and some executables but would probably be of no interest on its own?
 Cython and Python code in another upstream tarball, giving rise to another newstyle spkg "p_group_cohomology" that depends on "modular_resolution".
Would it be OK to have an spkg "modular_resolution" if it basically has no other purpose than being a dependency for "p_group_cohomology"?
Yes, this is perfectly fine to have this kind of granularity. Although if it could not have any independent interest/development, then it probably would be better to fold it into p_group_cohomology
from a more macroscale maintenance viewpoint. I guess it makes the p_group_cohomology less homogeneous as a project? I also had the impression that it would be able to be a standalone library.
And I still wonder what to do with the GAP code and the Singular code. On the one hand, such code clearly belongs to src/ext/gap respectively to src/ext/singular. On the other hand, an spkg is not supposed to inject code into src/ext.
I feel like those are no different than having Cython files whose compilation depends on an optional spkg or Python files that are only useful if an optional spkg is installed. So, by analogy, they should not be injected by the spkg during its installation, but instead part of the main Sage git control.
Is it ok to put the relevant files into src/ext even when the optional packages are not installed? I would think so; after all, the files I am talking about are not particularly large.
IMO, I think it is fine to put it in the source tree if it is meant to be something for Sage as suggested above. I'm not sure if I have a complete understanding of everything at this point to say anything definitive about what you should do in this situation. Also, someone might come along and say you should do it all in a pip package, but then I feel it is more likely to break on future upgrades of Sage.
comment:157 followup: 158 Changed 5 years ago by
Replying to tscrim:
Replying to SimonKing:
Apparently, the Cython code can not stay in the source tree, as it doesn't qualify as optional extension module.
Can you elaborate a bit more on this? I don't quite understand why not from a quick glance.
I have asked about it on sagedevel, and was told that a cython OptionalExtension should only be used if an existing functionality (i.e., a functionality that is provided by vanilla Sage) can be implemented in a different (better) way if the user has installed some optional package. Example: You have matrices over GF(p^n)
with p>2
and n>1
in Vanilla Sage, but if you install meataxe then a much faster implementation is available. Or you could also think of different graph backends.
However, the situation here is different: This is about a functionality that is not available in vanilla Sage and can only be provided if the user installs some optional package. I was told on sagedevel that I shouldn't use OptionalExtension in that situation.
Would it be OK to have an spkg "modular_resolution" if it basically has no other purpose than being a dependency for "p_group_cohomology"?
Yes, this is perfectly fine to have this kind of granularity. Although if it could not have any independent interest/development, then it probably would be better to fold it into
p_group_cohomology
from a more macroscale maintenance viewpoint. I guess it makes the p_group_cohomology less homogeneous as a project? I also had the impression that it would be able to be a standalone library.
David Green's code to compute resolutions of modular group algebras of finite pgroups has originally been a bunch of executables operating on data stored on disk. I made a standalone library out of it.
So, it could be used on its own, but I guess nobody has ever done.
Currently, I am developer for both the C / Gap code that I inherited from David Green, and for the Cython / Python / Singular code that I wrote myself. I do not foresee active development for the C code in the near future. So, from a development point of view, it would be easier to make a separate package out of the C code  it is not supposed to change in future versions. On top of that, the active development will take place in the Cython / Python / Singular part of the code, which thus should be another separate package.
And I still wonder what to do with the GAP code and the Singular code. On the one hand, such code clearly belongs to src/ext/gap respectively to src/ext/singular. On the other hand, an spkg is not supposed to inject code into src/ext.
I feel like those are no different than having Cython files whose compilation depends on an optional spkg or Python files that are only useful if an optional spkg is installed. So, by analogy, they should not be injected by the spkg during its installation, but instead part of the main Sage git control.
It is more like "some code snippets which can be used by some optional package that the user may install". So, I could understand that we wouldn't like to have these code snippets in the src tree. However, they are supposed to be used in Sage's GAP and Singular, and thus it makes sense to put them into src/sage/libs. That's the dilemma...
Is it ok to put the relevant files into src/ext even when the optional packages are not installed? I would think so; after all, the files I am talking about are not particularly large.
IMO, I think it is fine to put it in the source tree if it is meant to be something for Sage as suggested above. I'm not sure if I have a complete understanding of everything at this point to say anything definitive about what you should do in this situation. Also, someone might come along and say you should do it all in a pip package, but then I feel it is more likely to break on future upgrades of Sage.
I just thought: What does database_gap do? It is installing databases into Sage's version of GAP. What is the purpose of the GAP / Singular files that are used in the cohomology package? Well, they are to be installed into Sage's version of GAP respectively Singular. So, I should try to understand how database_gap manages to install stuff into Sage's GAP without inserting stuff into Sage's src tree.
comment:158 followup: 159 Changed 5 years ago by
Replying to SimonKing:
It is more like "some code snippets which can be used by some optional package that the user may install". So, I could understand that we wouldn't like to have these code snippets in the src tree. However, they are supposed to be used in Sage's GAP and Singular, and thus it makes sense to put them into src/sage/libs. That's the dilemma...
An important question is how exactly are these GAP/Singular files going to be used? For example, database_gap
and gap_packages
contain GAP packages which are installed in exactly the same way as GAP packages would be installed outside of Sage. They are not used by Sage directly, only by the GAPinSage. Would that work for you?
comment:159 Changed 5 years ago by
Replying to jdemeyer:
An important question is how exactly are these GAP/Singular files going to be used? For example,
database_gap
andgap_packages
contain GAP packages which are installed in exactly the same way as GAP packages would be installed outside of Sage. They are not used by Sage directly, only by the GAPinSage. Would that work for you?
The GAP functions currently are in files and are loaded into GAP like this:
if not gap('IsBoundGlobal("exportMTXLIB")'): gap.eval('Read("%s/src/ext/gap/modular_cohomology/GapMaxels.g");'%(SAGE_ROOT)) gap.eval('Read("%s/src/ext/gap/modular_cohomology/GapMB.g");'%(SAGE_ROOT)) gap.eval('Read("%s/src/ext/gap/modular_cohomology/GapSgs.g");'%(SAGE_ROOT)) gap.eval('InstallValue(exportMTXLIB,"MTXLIB=%s; export MTXLIB; ")'%(os.path.join(DOT_SAGE,"meataxe")))
The Singular functions are in interpreted libraries that are loaded like this:
singular.load('{}/dickson.lib'.format(os.path.join(SAGE_ROOT,'src','ext','singular'))) singular.load('{}/filterregular.lib'.format(os.path.join(SAGE_ROOT,'src','ext','singular')))
It would be OK from my point of view to copy filterregular.lib
and dickson.lib
into the shared folder containing all Singular libraries. Then, one could simply do
singular.LIB('dickson.lib') singular.LIB('filterregular.lib')
I certainly could make the modular cohomology package insert stuff into SAGE_LOCAL/share/singular/LIB/
. Actually I'd prefer this.
And it would also be OK to have some kind of GAP package, so that one could say
gap.LoadPackage('WhateverThisPackageIsCalled')
However, I simply don't know how to create a Gap package, thus, I'd appreciate a pointer.
comment:160 followup: 161 Changed 5 years ago by
have a look at https://github.com/gappackages/example
(if your GAP code does not call external programs, this is an overkill
comment:161 followup: 162 Changed 5 years ago by
Replying to dimpase:
have a look at https://github.com/gappackages/example
Thank you for the pointer!
(if your GAP code does not call external programs, this is an overkill
Actually the GAP code is the only part of the group cohomology computation that calls external programs  currently, for example, like that:
Exec(Concatenation("mkdir p ", dir)); ... buffer := Concatenation(exportMTXLIB, "makeInclusionMatrix ", Gstem, " ", Hstem, " ", Istem); Print(buffer, "\n"); Exec(buffer);
I somehow have the feeling that external programs (here: mkdir and !makeInclusionMatrix) shouldn't be called in that way. So, if gap_packages/example tells me how to do better, I'd be glad!
comment:162 followup: 163 Changed 5 years ago by
Replying to SimonKing:
(if your GAP code does not call external programs, this is an overkill
Actually the GAP code is the only part of the group cohomology computation that calls external programs
Maybe I misunderstood what you meant be "call external programs". In contrast to gappackages/example, my gap package would simply consist of a bunch of gapreadable files, but it would not provide the c sources for the external programs it is calling.
Anyway, is Exec(...) still the way to call programs from within GAP?
comment:163 followup: 164 Changed 5 years ago by
Replying to SimonKing:
Replying to SimonKing:
(if your GAP code does not call external programs, this is an overkill
Actually the GAP code is the only part of the group cohomology computation that calls external programs
Maybe I misunderstood what you meant be "call external programs". In contrast to gappackages/example, my gap package would simply consist of a bunch of gapreadable files, but it would not provide the c sources for the external programs it is calling.
is there a reason for not packaging them in? Are they called from anywhere else?
Anyway, is Exec(...) still the way to call programs from within GAP?
Yes. But I suppose calling mkdir
the way you do is a bit rough.
GAP has its own facilities for that, see DirectoryTemporary.
comment:164 followup: 165 Changed 5 years ago by
Replying to dimpase:
Replying to SimonKing:
Maybe I misunderstood what you meant be "call external programs". In contrast to gappackages/example, my gap package would simply consist of a bunch of gapreadable files, but it would not provide the c sources for the external programs it is calling.
is there a reason for not packaging them in? Are they called from anywhere else?
The executables are part of a C code base which also provides a library. My Cython code is linked against that library. So, no, the executables aren't called anywhere else, but, yes, the package which the executables belong to is used somewhere else.
Anyway, is Exec(...) still the way to call programs from within GAP?
Yes. But I suppose calling
mkdir
the way you do is a bit rough. GAP has its own facilities for that, see DirectoryTemporary.
The directory to be created is supposed to be permanent.
comment:165 followup: 166 Changed 5 years ago by
Replying to SimonKing:
Replying to dimpase:
Replying to SimonKing:
Maybe I misunderstood what you meant be "call external programs". In contrast to gappackages/example, my gap package would simply consist of a bunch of gapreadable files, but it would not provide the c sources for the external programs it is calling.
is there a reason for not packaging them in? Are they called from anywhere else?
The executables are part of a C code base which also provides a library. My Cython code is linked against that library. So, no, the executables aren't called anywhere else, but, yes, the package which the executables belong to is used somewhere else.
OK, makes sense then. This means it's simpler, in the sense that one won't really need much configure/makefile stuff, if at all.
Anyway, is Exec(...) still the way to call programs from within GAP?
Yes. But I suppose calling
mkdir
the way you do is a bit rough. GAP has its own facilities for that, see DirectoryTemporary.The directory to be created is supposed to be permanent.
This sounds like trouble. E.g. think about a multiuser situation... Does it mean to be some soft of caching? There is a package like this in GAP, AtlasRep. There data is fetched from the net, rather than computed, IIRC.
comment:166 followup: 167 Changed 5 years ago by
Replying to dimpase:
The directory to be created is supposed to be permanent.
This sounds like trouble. E.g. think about a multiuser situation...
I thought of a multiuser situation.
Does it mean to be some sort of caching? There is a package like this in GAP, AtlasRep. There data is fetched from the net, rather than computed, IIRC.
It's kind of similar.
 There is a repository in the web.
 There is a local repository that all users of a Sage installation can read; I call this the public database. Of course, only a user with write privileges can add data to the public database.
 Each user has his or her own local repository (I call it "private database"). The data in the private database is soft links to the public database (when the data is available there), otherwise is fetched from the web (when the data is available there), otherwise (or if explicitly requested) is computed from scratch.
In particular, in a multiuser setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.
comment:167 followup: 168 Changed 5 years ago by
Replying to SimonKing:
Replying to dimpase:
The directory to be created is supposed to be permanent.
This sounds like trouble. E.g. think about a multiuser situation...
I thought of a multiuser situation.
Does it mean to be some sort of caching? There is a package like this in GAP, AtlasRep. There data is fetched from the net, rather than computed, IIRC.
It's kind of similar.
 There is a repository in the web.
 There is a local repository that all users of a Sage installation can read; I call this the public database. Of course, only a user with write privileges can add data to the public database.
 Each user has his or her own local repository (I call it "private database"). The data in the private database is soft links to the public database (when the data is available there), otherwise is fetched from the web (when the data is available there), otherwise (or if explicitly requested) is computed from scratch.
In particular, in a multiuser setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.
Then you perhaps can borrow their design. Surely Thomas Breuer, the main developer of AtlasRep, would be glad to help.
comment:168 followup: 169 Changed 5 years ago by
Replying to dimpase:
In particular, in a multiuser setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.
Then you perhaps can borrow their design. Surely Thomas Breuer, the main developer of AtlasRep, would be glad to help.
On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?
comment:169 followup: 170 Changed 5 years ago by
Replying to SimonKing:
Replying to dimpase:
In particular, in a multiuser setting, the users share the data from the public database. However, each user may individually compute further information on a cohomology ring, that is stored in his/her private database and won't interfere with the shared data.
Then you perhaps can borrow their design. Surely Thomas Breuer, the main developer of AtlasRep, would be glad to help.
On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?
I haven't looked at its source for many years; it's reasonably welldocumented, have a look.
comment:170 followup: 172 Changed 5 years ago by
Replying to dimpase:
On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?
I haven't looked at its source for many years; it's reasonably welldocumented, have a look.
I just had a look, and apparently it is just a bunch of gap readable files. It seems that it contains no compiled code, in particular no linking against third party libraries.
comment:171 Changed 5 years ago by
Commit:  ae029297d600471d77f3de795c9ad8d726988a3b → 12e1bc50e4ef6ceedaaf30e66e1fd03d5d1cadf6 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
d4f8496  Documentation of new functions. Fix doctest in cochain.pyx

282ae43  Fixing almost all tests in cohomology.pyx. Change Simon King's address

99f2f3b  Remove 'timing' option. Remove commented out old code. Fix tests in modular_cohomology.pyx

10ea380  Remove 'shebang lines' from modular_resolution's spkg* scripts

4bf8add  Conversion of matrix data into Matrix_t* moved from matrix_gfpn_dense to libs/meataxe

c032ac7  Add libraries that the cohomology code is to be linked with

13482a1  Remove everything but CohomologyRing from __init__.py

a73ef8f  Fix import locations and types in cochain.*

83ec3ef  (modular_)cohomology.pyx: Fix import locations, cope with changes in matrix_gfpn_dense

12e1bc5  resolution.pyx: Fix import locations, cope with changes in matrix_gfpn_dense, fix dict keys in LIFTContainer

comment:172 followup: 174 Changed 5 years ago by
Replying to SimonKing:
Replying to dimpase:
On the other hand, the current question is not about the design of data storage/caching, but about the code layout. Is the AtlasRep package similar in that regard as well?
I haven't looked at its source for many years; it's reasonably welldocumented, have a look.
I just had a look, and apparently it is just a bunch of gap readable files. It seems that it contains no compiled code, in particular no linking against third party libraries.
Why do you need the latter? Don't you just call executables? They call wget
.
comment:173 Changed 5 years ago by
Description:  modified (diff) 

I got the code running! Probably some tests need fixing (some log strings may have changed, I am confident that the mathematical content didn't). Also, I should apply what I recently learnt about sig_on/sig_off vs. sig_check.
To be on the safe side, I tried to post the latest modular_resolution1.0.tar.gz, but apparently there is a problem on the webserver in Jena, and it is too large to post here. Anyway, IIRC, the file at http://users.minet.unijena.de/cohomology/modular_resolution1.0.tar.gz is still the correct version (not 100% sure though).
It certainly isn't ready for a formal review. The main part of the code still consists of cython OptionalExtension, which should eventually be moved into a pip installable module. Also, the location of the GAP and Singular code files should be fixed.
In any case, it should now provide the possibility to do some testing. Installation:
 checkout the branch
sage i database_gap
sage i meataxe
 copy modular_resolution1.0.tar.gz into SAGE_ROOT/upstream
sage i modular_resolution
 make
IIRC, Sage won't build the documentation of OptionalExtension. For the documentation of the old version of the package, see http://users.minet.unijena.de/cohomology/documentation/, but note that in the current shape of the code, from pGroupCohomology import ...
needs to be replaced by from sage.groups.modular_cohomology import ...
.
comment:174 followups: 176 177 Changed 5 years ago by
Replying to dimpase:
Why do you need the latter? Don't you just call executables? They call
wget
.
The meataxe package provides a library "mtx" and some executables.
The C code in modular_resolution1.0 (which is based on David Green's code but for which I became upstream) needs to link against mtx. It provides a library "modres" and some executables.
The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!) need to be linked both against mtx and against modres.
I am not calling the meataxe executables directly. However, IIRC, the executables provided by modular_resolution1.0 do call some meataxe executables.
At several stages of the cohomology computation, I need to call the modular_resolution executables; it results in many data files stored on disk in special folders, all data together determine an initial segment of the minimal projective resolution of the group algebra, and the data is then used by Cython code to compute the ring structure of the cohomology ring.
The calls to modular_resolution executables partially occur in GAP functions, and GAP functions are also responsible to create the special folders. However, the path of the special folders and also the names of the data files are determined by the Cython code. So, it should (when needed) be possible to move all the calls from GAP code to Cython code.
Concerning wget: It is the responsibility of my Python code to check whether the data of a cohomology ring are found locally. If data is to be fetched from web, which is done with Python's urllib2 module. I guess urllib2 uses wget, but I am not using wget directly.
I hope that clarified the structure of the code base.
comment:175 Changed 5 years ago by
PS: Perhaps I should tell why the data of a single cohomology ring are distributed in dozens of data files.
When developing the very first version of the group cohomology spkg, I tried to turn the modular_resolution executables into library functions that simply create the resolution data in memory, rather then storing them on disk. But I soon ran out of memory. So, I kept the executables and use the data files.
The cohomology computation is in increasing degree, and I originally tried to keep the cohomology data of lower degrees in memory when computing higher degrees. But I soon ran out of memory again. So, I started to cache data on disk.
Originally, I tried to pickle the cohomology data into one single file (plus all the files constituting the resolution data). However, in some examples, the process of pickling a cohomology ring took hours  simply the data was too large. Hence, I started to dump cohomology data in several smaller files, basically one in each degree, and then some more defining the ring maps induced by inclusion of certain special subgroups.
I hope that explains why I care about how data are stored. I am sure that the computation of the mod2 cohomology of the third Conway group or of all groups of order 128 would have been hardly possible when using a simple "one group, one file" approach to store data. Instead, it is "one group, one folder".
comment:176 followup: 178 Changed 5 years ago by
Replying to SimonKing:
The meataxe package provides a library "mtx" and some executables.
The C code in modular_resolution1.0 (which is based on David Green's code but for which I became upstream) needs to link against mtx. It provides a library "modres" and some executables.
The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!) need to be linked both against mtx and against modres.
Is meataxe still a static library? I remember that we had some discussion about that. If you link from different libraries or Python modules to meataxe, every instance will get an independent copy of meataxe. This is generally considered to be bad practice. For example, I remember that meataxe uses some kind of data directory. That needs to be set for every individual copy of the meataxe static library. With a dynamic library on the other hand, a program uses only one copy of meataxe. Setting the data directory once should then be sufficient.
comment:177 followup: 179 Changed 5 years ago by
Replying to SimonKing:
The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!)
If you plan to move it to a separate package, why take the detour via Sage? Maybe you could do it just for testing purposes, that might make sense. However, it seems like you are making things difficult for yourself if you first want your code to be merged in Sage and then taken out again as a separate package.
comment:178 Changed 5 years ago by
Replying to jdemeyer:
Is meataxe still a static library? I remember that we had some discussion about that. If you link from different libraries or Python modules to meataxe, every instance will get an independent copy of meataxe. This is generally considered to be bad practice. For example, I remember that meataxe uses some kind of data directory. That needs to be set for every individual copy of the meataxe static library. With a dynamic library on the other hand, a program uses only one copy of meataxe. Setting the data directory once should then be sufficient.
I totally agree with that, but recall that I am not upstream for MeatAxe (I only contributed patches to the current MeatAxe spkg). Hence, unless someone tells me how to easily change MeatAxe's Makefile, it will remain static. However, if there is an easy way to change things, I'll be happy to include yet another patch to the MeatAxe spkg.
comment:179 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
The Cython OptionalExtensions that you'll currently find in sage/groups/modular_cohomology (but that will be moved into a pip installable module!)
If you plan to move it to a separate package, why take the detour via Sage? Maybe you could do it just for testing purposes, that might make sense.
Exactly, that was my reason.
However, it seems like you are making things difficult for yourself if you first want your code to be merged in Sage and then taken out again as a separate package.
That's why I did not change this ticket to "needs review" in the last couple of years.
comment:180 followup: 181 Changed 5 years ago by
If database_gap
is a dependency of meataxe
(it seems so from the commands above) then it should be in build/pkgs/meataxe/dependencies
.
Same commentary for build/pkgs/modular_resolution/dependencies
 it should mention meataxe
.
comment:181 followup: 182 Changed 5 years ago by
Replying to dimpase:
If
database_gap
is a dependency ofmeataxe
(it seems so from the commands above) then it should be inbuild/pkgs/meataxe/dependencies
.
It isn't.
database_gap is a runtime dependency for the cohomology package (in the sense that it provides the SmallGroups library), but it has nothing to do with MeatAxe or with modular_resolution.
comment:182 followups: 183 185 Changed 5 years ago by
Replying to SimonKing:
Replying to dimpase:
If
database_gap
is a dependency ofmeataxe
(it seems so from the commands above) then it should be inbuild/pkgs/meataxe/dependencies
.It isn't.
database_gap is a runtime dependency for the cohomology package (in the sense that it provides the SmallGroups library), but it has nothing to do with MeatAxe or with modular_resolution.
runtime dependency is a dependency too.
OK, I see, you are saying that installing modular_decomposition
should trigger installation of meataxe and database_gap (the format of dependencies
there is weird, I am not sure these $(INST)/
are the right thing  most if not all packages don't have it`)
comment:183 Changed 5 years ago by
Replying to dimpase:
database_gap is a runtime dependency for the cohomology package (in the sense that it provides the SmallGroups library), but it has nothing to do with MeatAxe or with modular_resolution.
runtime dependency is a dependency too.
I can only repeat myself. database_gap has NOTHING TO DO WHATSOEVER with MeatAxe.
So, if anything, we are talking here about a runtime dependency for what currently is (but soon will not be) OptionalExtension in sage/groups/modular_cohomology/. IIRC, I was told that the dependencies listed in OptionalExtension only concern build time dependencies (which is mtx and modres), but not runtime dependencies.
comment:184 followup: 187 Changed 5 years ago by
files in src/sage/groups/modular_cohomology/
are not python3ready (all these print blah
instead of print(blah)
), this causes havoc while attempting to run tests, it seems. E.g.
print CohomologyRing(Integer(64),Integer(12)) # indirect doctest ^ SyntaxError: invalid syntax
comment:185 Changed 5 years ago by
Replying to dimpase:
OK, I see, you are saying that installing
modular_decomposition
[ I guess you meanmodular_resolution
] should trigger installation of meataxe and database_gap (the format ofdependencies
there is weird, I am not sure these$(INST)/
are the right thing  most if not all packages don't have it`)
No, an automatic installation (without explicit consent of the user) of database_gap is not allowed (licence issue) and not proposed.
Also, there are various ways to install the SmallGroups library. So, I think the correct way to deal with the problem is:
 meataxe clearly has no dependency whatsoever.
 modular_resolution will still have the dependency
meataxe
, but no other dependency. I just checked: There is no use of SmallGroup in the whole code.  take the current experimental OptionalExtension from sage/groups/modular_cohomology and put them into a pip installable upstream tarball
modular_group_cohomology
. build/pkgs/modular_group_cohomology/dependencies
is, as far as I understand, for build time dependencies, thus, it will only listmeataxe
andmodular_resolution
, but notdatabase_gap
.build/pkgs/spkginstall
will check that the SmallGroups library is in fact installed (it doesn't matter if it comes from database_gap or another source). If it isn't it will fail and advise the user to install it one way or another.
comment:186 followup: 188 Changed 5 years ago by
there are also some import errors, like
Running doctests with ID 2017121009593795694ca7. Git branch: cohomolo Using optional=4ti2,cbc,ccache,cmake,database_gap,dot2tex,gambit,gap_packages,latte_int,libhomfly,libogg,lidia,lrslib,meataxe,modular_resolution,mpir,normaliz,pynormaliz,python2,qhull,sage,scons Doctesting 1 file using 4 threads. sage t warnlong 68.7 barcode.py ********************************************************************** File "barcode.py", line 99, in sage.groups.modular_cohomology.barcode Failed example: B158 = H158.bar_code('UpperCentralSeries') Exception raised: Traceback (most recent call last): File "/home/dima/Sage/sagedev/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 515, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/dima/Sage/sagedev/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 885, in compile_and_execute exec(compiled, globs) File "<doctest sage.groups.modular_cohomology.barcode[18]>", line 1, in <module> B158 = H158.bar_code('UpperCentralSeries') File "sage/groups/modular_cohomology/cohomology.pyx", line 1673, in sage.groups.modular_cohomology.cohomology.permanent_result.__call__ (build/cythonized/sage/groups/modular_cohomology/cohomology.c:28178) val = self._f(inst,*args,**kwds) File "sage/groups/modular_cohomology/cohomology.pyx", line 11034, in sage.groups.modular_cohomology.cohomology.COHO.bar_code (build/cythonized/sage/groups/modular_cohomology/cohomology.c:156393) OUT[i,j] = C[j].hom(GH[i,j],C[i]).poincare_of_image() File "sage/groups/modular_cohomology/cochain.pyx", line 7054, in sage.groups.modular_cohomology.cochain.ChMap.poincare_of_image (build/cythonized/sage/groups/modular_cohomology/cochain.c:76428) from sage.groups.modular_cohomology import singular ImportError: cannot import name singular
comment:187 Changed 5 years ago by
Replying to dimpase:
files in
src/sage/groups/modular_cohomology/
are not python3ready (all theseprint blah
instead ofprint(blah)
), this causes havoc while attempting to run tests, it seems. E.g.print CohomologyRing(Integer(64),Integer(12)) # indirect doctest ^ SyntaxError: invalid syntax
OMG, I thought the old syntax is still fine in tests. Sure, I need to change all print statements in the tests, then.
comment:188 Changed 5 years ago by
Replying to dimpase:
there are also some import errors, like
ImportError: cannot import name singular
Thanks for finding this. I did do some computations in interactive sessions, but didn't run the test suite yet.
The point is that all the modules involved in the computation are supposed to use the same singular instance. And apparently I have changed something.
While we are at it: Is it OK to just import singular from sage.interfaces.singular? Or would you think it makes sense to provide a hook so that a user can plug in his/her own singular interface to the computation? I guess the former is easier and the latter doesn't add an advantage.
comment:190 Changed 5 years ago by
Dependencies:  #21437 → #24359 

comment:191 Changed 5 years ago by
After dealing with #24359, I'm back at the cohomology package.
There now remains to deal with two things:
 C code providing a library
libmodres
and some binaries; it is autotoolized and links against libmtx.  Python and Cython code linked against libmtx and libmodres and is NOT standalone as it makes use of Sage types and Sage infrastructure.
How would you recommend to proceed?
 Have two separate packages
modular_resolution
(installing libmodres) andp_group_cohomology
(pipinstalling the Python/Cython? modules) that the user has to install both?  Have a single package that both installs libmodres and the Python/Cython? code?
 Something else?
I somehow tend towards 2., since libmodres and the executables hardly are of any use without the Cython and Python code. But I can also see the advantage of a strict modularisation, as in 1.
comment:192 followup: 202 Changed 5 years ago by
Also, I need some pointers on how to create a pip installable module.
 Somewhere I found the information that both setup.py and Manifest.in are required for pip installation. However, in some pip installable packages in SAGE_ROOT/upstream I do not find Manifest.in. So, what's the matter with it? If setup.py is enough, is there any advantage of
pip install
overpython setup.py install
?
 In the old spkg, I was using distutils in setup.py. But somewhere I found the information that
pip uninstall
won't work with distutils. But I reckon that in future Sage will support uninstalling spkgs. So, should I use setuptools instead? Would that be able to build Cython code as well?
comment:193 followups: 195 203 Changed 5 years ago by
The third topic for which I need pointers: Documentation.
When I was young, Sage's documentation used a script builder.py to build its documentation. In the first versions of the cohomology spkg, I copied and modified it. Now, I see that builder.py has gone from Sage.
The question is how I could most easily build a reference manual for my spkg from all the docstrings in Cython and Python.
 Somewhere I found a blog post of Jeroen about "sphinx and Cython", advising to compile the Cython code with
binding=True
. Is that currently done in Sage? Wouldn't that involve a mild slowdown?  My impression is that using Sphinx with Cython is currently done in conf.py by replacing inspect.getargspec() by some custom version of getargspec() that works on Cython code. Would that already be enough? I could easily provide such a function, by creating a modified copy sage.misc.sageinspect or perhaps even by monkeypatching it.
 Would it be worthwhile to exploit Sage's doc build infrastructure? Say, by overriding
SAGE_DOC
andSAGE_DOC_SRC
insage_setup.docbuild.__init__
? Does that have a chance to work?
comment:194 Changed 5 years ago by
Dependencies:  #24359 → #24359 #24468 

comment:195 Changed 5 years ago by
For the record:
Replying to SimonKing:
 My impression is that using Sphinx with Cython is currently done in conf.py by replacing inspect.getargspec() by some custom version of getargspec() that works on Cython code. Would that already be enough? I could easily provide such a function, by creating a modified copy sage.misc.sageinspect or perhaps even by monkeypatching it.
This works. At least when using compiler_directives={'embedsignature': True}
and then adding some helper function to doc/sources/conf.py
that removes the signature from the docstring before passing it to sphinx.
comment:196 Changed 5 years ago by
Commit:  12e1bc50e4ef6ceedaaf30e66e1fd03d5d1cadf6 → 5e011619bc4814aeb02f785e327fb7ecff8eda1f 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
8d26842  Upgrade meataxe spkg to SharedMeatAxe 1.0 (dynamic library)

13828d1  Various fixes to spkginstall

1bc5815  Use compatible implementation for MatrixSpace in mtx_unpickle

09ef3e0  Merge branch 'u/jdemeyer/fix_comparison_of_matrices_that_live_in_equivalent_matrix_spaces' of git://trac.sagemath.org/sage into coho_rebase_base

2ac7e04  More stuff in the meataxe interface, and a meataxe helper function

fbc779e  Expose matrix_gfpn_dense.FieldConverter to the interface

17787a1  Fix reading MeatAxe matrix from a file

5e01161  Add p_group_cohomology3.0 spkg

comment:197 Changed 5 years ago by
Description:  modified (diff) 

Status:  needs_info → needs_review 
comment:198 Changed 5 years ago by
Description:  modified (diff) 

comment:199 Changed 5 years ago by
As usual, documentation is built, if SAGE_SPKG_INSTALL_DOCS=yes
.
sage i c p_group_cohomology
installs and tests the package. The tests are formed by a very short standalone test for David Greens programs and by a thorough test suite for the python/cython part of the package.
comment:200 Changed 5 years ago by
Commit:  5e011619bc4814aeb02f785e327fb7ecff8eda1f → 5f6033d8b17a7915e55733a9161b58a0d00c0048 

Branch pushed to git repo; I updated commit sha1. New commits:
5f6033d  Remove old cohomology doc before installing the new doc

comment:201 followup: 206 Changed 5 years ago by
Replying to SimonKing:
There are good news:
I have just finished
unit_test_64
, which does a computation of the cohomology rings of all groups of order 64, compares the results with the data in the data base, and provides timings.The results obtained with the new package version coincide with the old data. Moreover, the computation on a laptop with Intel(R) Core(TM) i55200U CPU @ 2.20GHz and 8GB of RAM only took 27:32 minutes of wall time resp. 15:56 minutes of CPU time, which is pretty fast.
There are even better news.
On the same machine(!) it now only takes 7:18.12 min walltime and 6:32.00 CPUtime. So, it is a very clear improvement compared with previous package version. For reference: The very first computation of these cohomology rings by the Magma programs of Jon Carlson took several months.
I don't know where the improvement is coming from. Can the change from a static meataxe library to a dynamic meataxe library explain the improvement?
comment:202 followup: 204 Changed 5 years ago by
Replying to SimonKing:
 Somewhere I found the information that both setup.py and Manifest.in are required for pip installation. However, in some pip installable packages in SAGE_ROOT/upstream I do not find Manifest.in. So, what's the matter with it?
MANIFEST.in
has nothing to do with pip
. It is needed to create a source package from the source tree. With "source tree" I mean what git checkout
gives you. That is not the same as a source package (a tarball created by make sdist
in Python or make dist
with autotools). The difference may not be so important for Python, so simple packages can get away with ignoring this difference. Typically, a source tarball would contain certain autogenerated files that the source tree does not.
 In the old spkg, I was using distutils in setup.py. But somewhere I found the information that
pip uninstall
won't work with distutils.
That is partially correct. To be more precise: a project installed using distutils (i.e. with ./setup.py install
) cannot be uninstalled with pip. However, a project installed with pip (using pip install ...
) can be uninstalled with pip, even if the setup.py
script uses distutils.
I know, it's confusing :) You should remember that pip is just a tool on top of either distutils or setuptools, it doesn't replace them. And many features of pip come from pip itself, not from the underlying distutils or setuptools.
So, should I use setuptools instead? Would that be able to build Cython code as well?
Personally, I would say: keep using distutils if that works for you. I'm sure that other people will disagree and say that you really should use setuptools.
comment:203 followup: 205 Changed 5 years ago by
Replying to SimonKing:
The question is how I could most easily build a reference manual for my spkg from all the docstrings in Cython and Python.
 Somewhere I found a blog post of Jeroen about "sphinx and Cython", advising to compile the Cython code with
binding=True
.
Yes, that should work "out of the box".
Is that currently done in Sage?
No. See #22747 for that.
Wouldn't that involve a mild slowdown?
Yes, it does involve a significant (in the statistics sense) slowdown, but only at the Python > Cython boundary. If your package is all Cython, then there should be no slowdown within your package. So that leaves the boundary between user code and your package. Do you expect many calls to your package in tight Python loops? If not, then I would go for the binding=True
solution.
comment:204 Changed 5 years ago by
Thank you for your explanations!
Replying to jdemeyer:
Personally, I would say: keep using distutils if that works for you. I'm sure that other people will disagree and say that you really should use setuptools.
The new package's setup.py does all imports from setuptools, whereas the old package's setup.py did the imports from distutils. So, that was a trivial change.
comment:205 followup: 207 Changed 5 years ago by
Replying to jdemeyer:
Wouldn't that involve a mild slowdown?
Yes, it does involve a significant (in the statistics sense) slowdown, but only at the Python > Cython boundary. If your package is all Cython, then there should be no slowdown within your package. So that leaves the boundary between user code and your package. Do you expect many calls to your package in tight Python loops? If not, then I would go for the
binding=True
solution.
I do care for speed, but I think all tight loops occur in Cython, not Python.
In any case, the only real problem for building the docs is to determine the argspec of methods and functions. The new package solves it with compiler_directives={'embedsignature': True}
and some helper functions that you can find in pyxsources/doc/sources/conf.py
comment:206 followup: 208 Changed 5 years ago by
Replying to SimonKing:
Can the change from a static meataxe library to a dynamic meataxe library explain the improvement?
All other things being equal, a dynamic library has a little bit of overhead to deal with the dynamic linking, so it should rather be slower instead of faster.
So there must be another explanation. Are you using different compiler flags maybe? Are there certain computations in meataxe which are cached and which might be reused?
comment:207 Changed 5 years ago by
Replying to SimonKing:
I do care for speed, but I think all tight loops occur in Cython, not Python.
Then I think that binding=True
is the way to go because it uses standard tools without hacks like a custom getargspec.
In any case, you could use binding=True
and do the timings again to check.
comment:208 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Can the change from a static meataxe library to a dynamic meataxe library explain the improvement?
All other things being equal, a dynamic library has a little bit of overhead to deal with the dynamic linking, so it should rather be slower instead of faster.
Yes, that's why I am surprised.
So there must be another explanation. Are you using different compiler flags maybe? Are there certain computations in meataxe which are cached and which might be reused?
comment:142 was with the thencurrent meataxe spkg, i.e., static, with patches, using a custom Makefile. If I remember correctly, it did use some O
flag, but I don't remember which one (O2
?). The difference with respect to the current meataxe is (1) it is now dynamic library, and (2) autotools is used. The current compiler flags for meataxe are chosen by autotools.
comment:210 followup: 211 Changed 5 years ago by
The old Sage meataxe spkg used O
(which means O1
) as optimization flag. So the change of flag might explain the increase in performance that you saw.
comment:211 followup: 212 Changed 5 years ago by
Replying to jdemeyer:
The old Sage meataxe spkg used
O
(which meansO1
) as optimization flag. So the change of flag might explain the increase in performance that you saw.
OK, if O
was hardcoded in the Makefile shipped with upstream MeatAxe 2.4.24, then the flags chosen by autotools in SharedMeatAxe 1.0 could have improved things.
By the way, I didn't try the binding=True
compiler flag yet. Do you think that not having it and instead using a helper function for building the docs is a problem?
comment:212 followups: 213 214 Changed 5 years ago by
Replying to SimonKing:
Do you think that not having it and instead using a helper function for building the docs is a problem?
If it works, then it's not a problem. I do think that it's more fragile.
comment:213 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Do you think that not having it and instead using a helper function for building the docs is a problem?
If it works, then it's not a problem. I do think that it's more fragile.
As you can see here, building the docs does work, i.e., the signatures of the methods of cdef classes are correctly given, and also the signature of methods that are wrapped with @tempory_result or @permanent_result.
In any case, I will soon(ish) test what happens with binding:True
. If the performance loss is acceptable, then p_group_cohomology3.1 will use it.
comment:214 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Do you think that not having it and instead using a helper function for building the docs is a problem?
If it works, then it's not a problem. I do think that it's more fragile.
Remark: The process_*
functions in src/doc/source/conf.py are only mild modifications of the corresponding functions in Sage's documentation's conf.py.
Anyway, I now try to do without it.
comment:215 Changed 5 years ago by
Is it advised to replace "old" string formatting such as "%s bar"%foo
with "new" string formatting such as "{} bar".format(foo)
? If it is: Is there a tool that converts the first syntax into the second? I wouldn't like to change it manually, as there are hundreds of occurrences in the code.
comment:216 followup: 219 Changed 5 years ago by
Jeroen, I just tested binding=True
.
First problem: The signature of methods of extension classes are not correctly determined. For example, I get
sage: from inspect import getargspec sage: from pGroupCohomology.cochain import COCH sage: getargspec(COCH.setname) ArgSpec(args=['self', 's', 'is_polyrep'], varargs=None, keywords=None, defaults=None)
but the method is
def setname(self, s, is_polyrep=False):
and therefore the html documentation would not be built correctly.
Second problem: Performance.
sage: from pGroupCohomology.factory import unit_test_64 sage: L,t = unit_test_64() ... #267: Walltime 9:24.40 min CPUtime 8:31.86 min Singular 0:48.71 min Gaptime 0:19.50 min
Here is what I get with the current version of the package (i.e., with embedsignature=True
and more elaborate helper functions):
sage: from sage.misc.sageinspect import sage_getargspec sage: from pGroupCohomology.cochain import COCH sage: sage_getargspec(COCH.setname) ArgSpec(args=['self', 's', 'is_polyrep'], varargs=None, keywords=None, defaults=(False,))
Here, inspect.getargspec won't work at all, but sage_getargspec gives a correct answer, which can also be seen in the published package documentation.
And concerning performance:
sage: from pGroupCohomology.factory import unit_test_64 sage: L,t = unit_test_64() ... #267: Walltime 7:50.20 min CPUtime 6:59.89 min Singular 0:46.04 min Gaptime 0:20.58 min
Conclusion
The loss in performance is quite noticeable, and moreover, binding=True
doesn't help with the documentation. Therefore, I won't use it.
comment:217 followup: 218 Changed 5 years ago by
In recent versions of Python and Cython, there is the even newer syntax f"{foo} bar"
. Unless you specifically want to support older versions, I would recommend that.
comment:218 followup: 220 Changed 5 years ago by
Replying to jdemeyer:
In recent versions of Python and Cython, there is the even newer syntax
f"{foo} bar"
. Unless you specifically want to support older versions, I would recommend that.
Is that in Sage already? If it is, I would certainly use that syntax in future. But the question is: Can I upgrade the syntax that I used in the past, without manually editing hundreds of lines of code?
comment:219 followups: 221 223 Changed 5 years ago by
Replying to SimonKing:
and therefore the html documentation would not be built correctly.
Did you correctly apply all steps from http://opendreamkit.org/2017/06/09/CythonSphinx, in particular the patching of inspect.isfunction
?
comment:220 followup: 222 Changed 5 years ago by
Replying to SimonKing:
Replying to jdemeyer:
In recent versions of Python and Cython, there is the even newer syntax
f"{foo} bar"
. Unless you specifically want to support older versions, I would recommend that.Is that in Sage already?
Yes.
But the question is: Can I upgrade the syntax that I used in the past, without manually editing hundreds of lines of code?
No, but there is also no real reason to do that. The old syntax still works and it's not even deprecated.
comment:221 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
and therefore the html documentation would not be built correctly.
Did you correctly apply all steps from http://opendreamkit.org/2017/06/09/CythonSphinx, in particular the patching of
inspect.isfunction
?
No. I only took your comment:203 that it "would work out of the box"...
comment:222 followup: 224 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Replying to jdemeyer:
In recent versions of Python and Cython, there is the even newer syntax
f"{foo} bar"
. Unless you specifically want to support older versions, I would recommend that.Is that in Sage already?
Yes.
No, it isn't!
sage: version() 'SageMath version 8.2.beta2, Release Date: 20180101' sage: s = "foo" sage: f"{s} bar" File "<ipythoninput17b4ba52efd1d0>", line 1 f"{s} bar" ^ SyntaxError: invalid syntax
comment:223 followup: 225 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
and therefore the html documentation would not be built correctly.
Did you correctly apply all steps from http://opendreamkit.org/2017/06/09/CythonSphinx, in particular the patching of
inspect.isfunction
?
Now I am puzzled. The monkeypatch you suggested is:
def isfunction(obj): return hasattr(type(obj), "__code__") import inspect inspect.isfunction = isfunction
However, when you inspect `inspect.isfunction, you'll find
Signature: inspect.isfunction(obj) Source: def isfunction(obj): """ ... """ # We use type(obj) instead of just obj to avoid __getattr__(). # Some types, like methods, will return the __code__ of the # underlying function in __getattr__() but we don't want to # detect those as functions. return hasattr(type(obj), "__code__")
So, the patch seems pointless.
comment:224 Changed 5 years ago by
Replying to SimonKing:
No, it isn't!
Sorry, I meant that the CythoninSage supports it. The Python 2 language does not support it.
comment:225 followup: 226 Changed 5 years ago by
Replying to SimonKing:
So, the patch seems pointless.
That is because the monkeypatch is already applied in Sage.
But I just realized that you are right: even with binding=True
, default values are not displayed in the docs. So you found an actual bug!
comment:226 Changed 5 years ago by
comment:227 followup: 228 Changed 5 years ago by
The binding=True option led to 20% performance loss, that's why I wouldn't use it even if the bug with default options was fixed.
But here's something for p_group_cohomology3.1: Replace the pexpect interfaces to gap and singular by calls to libgap and libsingular. After all, meanwhile all Singular data structures relevant for cohomology computation (namely: commutative and supercommutative algebras with degree weights and arbitrary matrix ordering) are wrapped in Sage. But the transition would be a lot of work, that's why I wouldn't do it in version 3.0.
comment:228 followup: 402 Changed 5 years ago by
Replying to SimonKing:
The binding=True option led to 20% performance loss, that's why I wouldn't use it even if the bug with default options was fixed.
Yes, I understood that. And this is not easy to fix, it would require changes to Python itself.
comment:229 Changed 5 years ago by
For reference: In order to achieve global dominance (i.e., to make the future package version p_group_cohomology3.1
a standard package of Sage), the following would be needed in preparation:
 Make SharedMeatAxe standard. I think it should be standard anyway, as it offers matrix arithmetic over small nonprime fields of odd characteristics that is vastly superior compared with the currently used
Matrix_generic_dense
.  Remove the dependency on the SmallGroups library. In fact, the current cohomology package is able to create the cohomology ring of a group as long as it can be expressed as a permutation group in GAP; one doesn't necessarily need to know the SmallGroups address of it.
Hence, I could rewrite the relevant parts of the code so that
IdGroup()
(a function to determine the SmallGroups address) is used when available for efficiency, and to work around it (e.g., create an elementary abelian group without referring toSmallGroup(p^n, NumberSmallGroups(p^n)1)
) otherwise.
In that way, the cohomology package would only depend on standard packages, and thus the Cython parts of it could be moved into the Sage src tree.
However, it would be good if p_group_cohomology3.0
(which this ticket is about) could be reviewed first.
comment:230 followup: 232 Changed 5 years ago by
Status:  needs_review → needs_work 

Work issues:  → Cope with backwards incompatible changes  again. 
It is a pity that the spkg hasn't been reviewed yet  because once again, a backward incompatibly change in SageMath broke the spkg!
Reason this time: _add_
of ModuleElement has changed from cdef (which it still is for Element!) to cpdef.
I must say that those things have happened far too often, and I find it quite annoying. That's one of the reasons why I will try to modify the spkg so that it will not depend on the SmallGroups library, with the aim to make group cohomology a standard package with cython code in the SageMath source tree. In that way, the patchbots would immediately notice if some ticket makes it necessary to change my code.
comment:231 Changed 5 years ago by
GAP 4.9 (a beta released 1 Feb) has SmallGroups licensed under Artistic License 2.0, which is GPLcompatible. So we should just make database_gap standard, and you should not try to work around this nonproblem...
comment:232 followup: 242 Changed 5 years ago by
Replying to SimonKing:
Reason this time:
_add_
of ModuleElement has changed from cdef (which it still is for Element!) to cpdef.
It was always meant to be used as cpdef _add_
(that is what is documented in element.pyx
). So in this case, I would argue that your old code was wrong and should have been fixed anyway.
comment:233 Changed 5 years ago by
Dependencies:  #24359 #24468 

comment:234 followup: 236 Changed 5 years ago by
This ticket makes some seemingly unrelated changes to src/sage/libs/meataxe
and src/sage/matrix
. Is that intentional?
comment:235 followup: 238 Changed 5 years ago by
Replying to SimonKing:
 The spkginstall script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.
If it's a dependency, it should be declared as dependency in build/pkgs/p_group_cohomology/dependencies
.
comment:236 followup: 239 Changed 5 years ago by
Replying to jdemeyer:
This ticket makes some changes to
src/sage/libs/meataxe
. Is that intentional?
The Cython code in the package links against libmtx, and I think it is cleanest to have the relevant pxdheader file in a single central place (src/sage/libs
), rather than making a copy of it. About the changes: I need a bit more stuff from meataxe than what sage.matrix.matrix_gfpn_dense needs, therefore the changes.
comment:237 followup: 241 Changed 5 years ago by
Is there a public website or repository for this package? If so, it should be put in SPKG.txt
. And I would remove the version number from the first line of that file, you'll just forget to update it anyway.
comment:238 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
 The spkginstall script first tests whether the SmallGroups library is installed in GAP. If it isn't, a warning is printed; however, the package still gets installed, as it only is a runtime dependency.
If it's a dependency, it should be declared as dependency in
build/pkgs/p_group_cohomology/dependencies
.
No. I think that was meanwhile discussed enough often.
comment:239 followup: 240 Changed 5 years ago by
Don't be afraid of Unicode: you can put names like Sikirić and Poincaré in SPKG.txt
comment:240 Changed 5 years ago by
Replying to jdemeyer:
Don't be afraid of Unicode: you can put names like Sikirić
Ouch, and thank you for spotting that I was misspelling his name as "Sikitic" (with t instead of r)!
comment:241 Changed 5 years ago by
Replying to jdemeyer:
Is there a public website or repository for this package? If so, it should be put in
SPKG.txt
.
I thought the pointer to the package's public documentation would be enough. But of course I can also add information where to download (it is basically the same address).
And I would remove the version number from the first line of that file, you'll just forget to update it anyway.
Good point...
comment:242 followup: 243 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Reason this time:
_add_
of ModuleElement has changed from cdef (which it still is for Element!) to cpdef.It was always meant to be used as
cpdef _add_
(that is what is documented inelement.pyx
). So in this case, I would argue that your old code was wrong and should have been fixed anyway.
In any case, I should cope with that API change. Question: Should it give rise to a new package version 3.0.1, or can it remain 3.0? From my point of view, publication of the package is only finished as soon as it became an optional package in SageMath, hence, I consider the current tar ball not as a "published version".
comment:243 Changed 5 years ago by
Replying to SimonKing:
Question: Should it give rise to a new package version 3.0.1, or can it remain 3.0? From my point of view, publication of the package is only finished as soon as it became an optional package in SageMath, hence, I consider the current tar ball not as a "published version".
I personally don't care much...
comment:244 Changed 5 years ago by
Commit:  5f6033d8b17a7915e55733a9161b58a0d00c0048 → ccae2cc49ae941069b96067f9bce45355975da25 

comment:245 Changed 5 years ago by
Commit:  ccae2cc49ae941069b96067f9bce45355975da25 → 4d696e0a19c1edb4832cc5804be9c92eb21dcaf0 

Branch pushed to git repo; I updated commit sha1. New commits:
4d696e0  p_group_cohomology3.0: Fix checksum

comment:246 followup: 247 Changed 5 years ago by
Status:  needs_work → needs_review 

I have modified cochain.pyx so that it builds with the current pxd header in sage.structure.element. The SPKG.txt is modified as requested. The checksum is updated. The tarball is updated as well (still at the location stated in the ticket description), still with version number 3.0. And I just checked that the package still installs.
Concerning the dependencies, I think it was consented at some point that the dependencies
files list build time dependencies, so that all packages can be built in a correct order. spkginstall does test whether the SmallGroups library is available and prints a warning if it isn't, but this won't prevent the package from being installable.
Needs review, then...
comment:247 followup: 261 Changed 5 years ago by
Replying to SimonKing:
Concerning the dependencies, I think it was consented at some point that the
dependencies
files list build time dependencies, so that all packages can be built in a correct order.
Well, that advice is meant for dependencies on standard packages. Standard packages are built anyway, so at runtime you can count on the fact that they will be present. This is not true for optional packages, so runtime dependencies should be put in dependencies
too (as orderonly dependency).
comment:248 followup: 253 Changed 5 years ago by
[p_group_cohomology3.0] Installing p_group_cohomology3.0 [p_group_cohomology3.0] [p_group_cohomology3.0] Error compiling Cython file: [p_group_cohomology3.0]  [p_group_cohomology3.0] ... [p_group_cohomology3.0] if (len(L)!=R.Data.projrank[n]): [p_group_cohomology3.0] raise IndexError("Last parameter must be of length %d"%(R.Data.projrank[n])) [p_group_cohomology3.0] self.Resl = R [p_group_cohomology3.0] self.Deg = n [p_group_cohomology3.0] self.Name = str(Nick) [p_group_cohomology3.0] self.Data = makeMTX(rawMatrix(R.G_Alg.Data.p, [L])) [p_group_cohomology3.0] ^ [p_group_cohomology3.0]  [p_group_cohomology3.0] [p_group_cohomology3.0] pGroupCohomology/cochain.pyx:491:32: undeclared name not builtin: rawMatrix [p_group_cohomology3.0] [p_group_cohomology3.0] Error compiling Cython file: [p_group_cohomology3.0]  [p_group_cohomology3.0] ... [p_group_cohomology3.0] if (len(L)!=R.Data.projrank[n]): [p_group_cohomology3.0] raise IndexError("Last parameter must be of length %d"%(R.Data.projrank[n])) [p_group_cohomology3.0] self.Resl = R [p_group_cohomology3.0] self.Deg = n [p_group_cohomology3.0] self.Name = str(Nick) [p_group_cohomology3.0] self.Data = makeMTX(rawMatrix(R.G_Alg.Data.p, [L])) [p_group_cohomology3.0] ^ [p_group_cohomology3.0]  [p_group_cohomology3.0] [p_group_cohomology3.0] pGroupCohomology/cochain.pyx:491:41: Cannot convert Python object to 'Matrix_t *' [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "setup.py", line 96, in <module> [p_group_cohomology3.0] ext_modules=cythonize(ext_mods, compiler_directives={'embedsignature': True}), [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/Cython/Build/Dependencies.py", line 1039, in cythonize [p_group_cohomology3.0] cythonize_one(*args) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/Cython/Build/Dependencies.py", line 1161, in cythonize_one [p_group_cohomology3.0] raise CompileError(None, pyx_file) [p_group_cohomology3.0] Cython.Compiler.Errors.CompileError: pGroupCohomology/cochain.pyx [p_group_cohomology3.0] Error: could not determine package name [p_group_cohomology3.0] ******************************************************************************** [p_group_cohomology3.0] Error installing p_group_cohomology3.0 [p_group_cohomology3.0] ********************************************************************************
I guess you are missing a dependency on the Sage library. But to be completely honest, I don't really know how to declare that dependency. This is the first package like this.
comment:249 Changed 5 years ago by
Status:  needs_review → needs_work 

After first building the Sage library, it still doesn't work:
[p_group_cohomology3.0] building 'pGroupCohomology.resolution' extension [p_group_cohomology3.0] creating build/temp.linuxppc64le2.7 [p_group_cohomology3.0] creating build/temp.linuxppc64le2.7/pGroupCohomology [p_group_cohomology3.0] gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall Wnounused fPIC I/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/cpython I/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/libs/ntl I/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/cysignals I/home/jdemeyer/sagetest/local/include I/home/jdemeyer/sagetest/local/include/python2.7 I/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/numpy/core/include I/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages I/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/ext I/home/jdemeyer/sagetest/local/include/python2.7 c pGroupCohomology/resolution.c o build/temp.linuxppc64le2.7/pGroupCohomology/resolution.o [p_group_cohomology3.0] pGroupCohomology/resolution.c:564:10: fatal error: modular_resolution/pgroup.h: No such file or directory [p_group_cohomology3.0] #include "modular_resolution/pgroup.h" [p_group_cohomology3.0] ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [p_group_cohomology3.0] compilation terminated. [p_group_cohomology3.0] error: command 'gcc' failed with exit status 1 [p_group_cohomology3.0] Running setup.py install for pGroupCohomology: finished with status 'error'
comment:250 followup: 259 Changed 5 years ago by
And the smallgroups warning is broken too:
./spkginstall: line 33: [: too many arguments
comment:252 followup: 255 Changed 5 years ago by
A solution for #24106 would be to use plain $MAKE install
instead of sdh_make_install
.
comment:253 followup: 254 Changed 5 years ago by
Replying to jdemeyer:
[p_group_cohomology3.0] Installing p_group_cohomology3.0 [p_group_cohomology3.0] [p_group_cohomology3.0] Error compiling Cython file: [p_group_cohomology3.0]  [p_group_cohomology3.0] ... [p_group_cohomology3.0] if (len(L)!=R.Data.projrank[n]): [p_group_cohomology3.0] raise IndexError("Last parameter must be of length %d"%(R.Data.projrank[n])) [p_group_cohomology3.0] self.Resl = R [p_group_cohomology3.0] self.Deg = n [p_group_cohomology3.0] self.Name = str(Nick) [p_group_cohomology3.0] self.Data = makeMTX(rawMatrix(R.G_Alg.Data.p, [L])) [p_group_cohomology3.0] ^ [p_group_cohomology3.0]  [p_group_cohomology3.0] [p_group_cohomology3.0] pGroupCohomology/cochain.pyx:491:32: undeclared name not builtin: rawMatrix
It seems that you didn't use the branch from this ticket. That's exactly one of the functions that is added to sage.libs.meataxe, because logically it belongs there.
comment:254 followup: 256 Changed 5 years ago by
Replying to SimonKing:
It seems that you didn't use the branch from this ticket.
You didn't understand my comment. I did use this branch, it's just that the Sage library was not rebuilt. But since I don't see an easy fix, let's ignore this issue for now.
comment:255 followup: 258 Changed 5 years ago by
Replying to jdemeyer:
A solution for #24106 would be to use plain
$MAKE install
instead ofsdh_make_install
.
Now seriously??? I just got used to using yet another Sage build tool. So then, why not use plain pip install
, too?
Also, what seems to be the problem? modular_resolution is an autotoolized package, and I'd rather say that if sdh_make_install
doesn't work on it then the fix should be in Sage, not in the package.
But since I don't intend to wait for yet another ticket (namely one for fixing sdh_make_install
), I'll switch back to using $MAKE
and $MAKE install
.
comment:256 followups: 257 260 Changed 5 years ago by
Replying to jdemeyer:
You didn't understand my comment. I did use this branch, it's just that the Sage library was not rebuilt. But since I don't see an easy fix, let's ignore this issue for now.
I don't think that there is an issue here. As soon as the branch from this ticket is merged, it will of course be built before installing the group cohomology package. So, what happened to you will not happen to the users.
comment:257 Changed 5 years ago by
Replying to SimonKing:
I don't think that there is an issue here. As soon as the branch from this ticket is merged, it will of course be built before installing the group cohomology package. So, what happened to you will not happen to the users.
+1
comment:258 Changed 5 years ago by
Replying to SimonKing:
Also, what seems to be the problem?
The problem is that Sage is trying to separate the build and install steps of packages. This is in general a good thing to do. But in your package, the build of p_group_cohomology
relies on the installation of modular_resolution
.
I'll switch back to using
$MAKE
and$MAKE install
.
Only $MAKE install
should be affected.
comment:259 Changed 5 years ago by
Replying to jdemeyer:
And the smallgroups warning is broken too:
./spkginstall: line 33: [: too many arguments
Line 33? That's puzzling:
$ wc l build/pkgs/p_group_cohomology/spkginstall 27 build/pkgs/p_group_cohomology/spkginstall
Anyway, I guess the problem arises in these lines:
if [ `echo "SmallGroup(13,1); quit;"  gap b T  grep "13"` = "" ]; then echo "==================================================================" echo " Warning: It seems that GAP's SmallGroups library is missing. " echo " It is a runtime dependency for the p_group_cohomology package. " echo " One way to install it is as part of the database_gap package. " echo "==================================================================" fi
I was once told that this is how to test whether SmallGroups is installed. But now I see that executing the test (in a Sage shell and without SmallGroups installed) gives this:
(sagesh) king@kingC70B:sage$ echo "SmallGroup(13,1); quit;"  gap b T  grep "13" Error, the Small Groups library is required but not installed
Not ideal.
Two potential ways out:
 Put
database_gap
into the dependencies, which it seems is what you want me to do anyway. This would ensure that SmallGroups is installed. But technically my package does not depend on the fulldatabasegap
but on SmallGroups only, which theoretically could be installed in different ways. By consequence, I would also need to adapt some parts of the documentation and I guess of SPKG.txt.  Find a different way of testing whether SmallGroups is installed or not.
comment:260 followup: 262 Changed 5 years ago by
Replying to SimonKing:
Replying to jdemeyer:
You didn't understand my comment. I did use this branch, it's just that the Sage library was not rebuilt. But since I don't see an easy fix, let's ignore this issue for now.
I don't think that there is an issue here. As soon as the branch from this ticket is merged, it will of course be built before installing the group cohomology package. So, what happened to you will not happen to the users.
OTOH: Is it possible and does it make sense to put sagelib
into the dependencies of the package? After all, it does link against stuff from the sage library.
comment:261 followup: 263 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Concerning the dependencies, I think it was consented at some point that the
dependencies
files list build time dependencies, so that all packages can be built in a correct order.Well, that advice is meant for dependencies on standard packages. Standard packages are built anyway, so at runtime you can count on the fact that they will be present. This is not true for optional packages, so runtime dependencies should be put in
dependencies
too (as orderonly dependency).
PS: See your comment:93
comment:262 Changed 5 years ago by
Replying to SimonKing:
OTOH: Is it possible to put
sagelib
into the dependencies of the package?
I don't think so.
comment:263 Changed 5 years ago by
Replying to SimonKing:
PS: See your comment:93
That comment is from the time when you had a very different packaging in mind, so it's not relevant now. That was talking about the Cython module, not the package.
comment:264 Changed 5 years ago by
Commit:  4d696e0a19c1edb4832cc5804be9c92eb21dcaf0 → a4fa76230828ae8b3e771f968f0665a2048c06eb 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
a4fa762  p_group_cohomology3.0: Fix checksum, dependencies and SPKG.txt

comment:265 Changed 5 years ago by
Commit:  a4fa76230828ae8b3e771f968f0665a2048c06eb → 98e6c213914bfe601a54e66640c5425da03bc161 

Branch pushed to git repo; I updated commit sha1. New commits:
98e6c21  p_group_cohomology3.0: Remove useless test in spkginstall

comment:266 followup: 269 Changed 5 years ago by
Status:  needs_work → needs_review 

Work issues:  Cope with backwards incompatible changes  again. 
I think it is now again ready for review. Summary of what I did today:
 Cope with the changes in
src/sage/structure/element.pxd
 Add
database_gap
as a dependency. I am not totally happy about it, but as it seems the dependency will soon be GPL compatible.  Remove the test for SmallGroup from spkginstall, as it is now tested by the dependency list.
 Change accents (Sikirić, Poincaré) both in SPKG.txt and in the index.rst file of the documentation (in the docstrings, I did use accents already).
 Fix the information on where it find the local documentation (it used to tell that it can be found in
SAGE_LOCAL/share/p_group_cohomology/html/
, but it isSAGE_LOCAL/share/p_group_cohomology/
.  Fix the checksum and update both the documentation and the tarball on the project's web pages.
comment:267 Changed 5 years ago by
Milestone:  sage6.8 → sage8.2 

Status:  needs_review → needs_work 
[p_group_cohomology3.0] Invalid checksum for cached file /home/jdemeyer/sagetest/upstream/p_group_cohomology3.0.tar.gz, deleting
comment:268 followups: 270 273 Changed 5 years ago by
The Python package is packaged badly. It contains a leftover build
directory. Really, the proper way to package this is to use python setup.py sdist
for the Python part and make dist
for the autotools part.
Also, given that the package is rather large, you could consider using .xz
compression. Note that xz
is a standard package in Sage, just add  xz
to the dependencies.
comment:269 followup: 271 Changed 5 years ago by
Replying to SimonKing:
 Add
database_gap
as a dependency.
Again a detail, but it would be better to have this as orderonly dependency ( database_gap
)
comment:270 Changed 5 years ago by
Replying to jdemeyer:
The Python package is packaged badly. It contains a leftover
build
directory. Really, the proper way to package this is to usepython setup.py sdist
for the Python part andmake dist
for the autotools part.
OK, good point. For the autotools part, I did already. But for the Python part I didn't.
Also, given that the package is rather large, you could consider using
.xz
compression. Note thatxz
is a standard package in Sage, just add xz
to the dependencies.
OK, as soon as I read the xz
manual, I'll try it.
comment:271 followup: 272 Changed 5 years ago by
comment:272 Changed 5 years ago by
Replying to SimonKing:
Replying to jdemeyer:
Replying to SimonKing:
 Add
database_gap
as a dependency.Again a detail, but it would be better to have this as orderonly dependency (
 database_gap
)What does that mean? Are you saying that the
dependencies
file should contain this:meataxe  database_gap?
Well really
meataxe  database_gap xz
comment:273 followup: 274 Changed 5 years ago by
Replying to jdemeyer:
The Python package is packaged badly. It contains a leftover
build
directory. Really, the proper way to package this is to usepython setup.py sdist
for the Python part andmake dist
for the autotools part.
Now I recall why I didn't use python setup.py sdist
 because actually I tried it before.
python setup.py sdist
creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.
comment:274 followups: 275 276 Changed 5 years ago by
Replying to SimonKing:
python setup.py sdist
creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.
Well, you need to add those file to MANIFEST.in
. See for example https://github.com/defeo/cypari2/blob/master/MANIFEST.in
comment:275 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
python setup.py sdist
creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.Well, you need to add those file to
MANIFEST.in
. See for example https://github.com/defeo/cypari2/blob/master/MANIFEST.in
... and exclude the .c files, right?
comment:276 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
python setup.py sdist
creates a gzipped tar file that contains the .c files, but neither the .pyx nor the .pxd files. So, it is not a source distribution in the sense of a cython source distribution.Well, you need to add those file to
MANIFEST.in
. See for example https://github.com/defeo/cypari2/blob/master/MANIFEST.in
I put Manifest.in in the same folder as setup.py, and it contains this:
include COPYING globalinclude pGroupCohomology/*.py pGroupCohomology/*.pyx pGroupCohomology/*.pxd pGroupCohomology/*.pxi globalexclude pGroupCohomology/*.c prune dist
Before that, I tried to be less specific and only wrote *.py
instead of pGroupCohomology/*.py
. However, the effect is that all .pxd
files and all .c
files are INCLUDED and all .py
and .pyx
files are missing.
So, what did I do wrong?
comment:277 Changed 5 years ago by
PS:
include pGroupCohomology/*.py pGroupCohomology/*.pyx pGroupCohomology/*.pxd pGroupCohomology/*.pxi exclude pGroupCohomology/*.c prune dist
didn't work either.
comment:278 followup: 279 Changed 5 years ago by
I think that globalinclude
is meant to specify files, not paths.
See https://docs.python.org/2/distutils/sourcedist.html#commands
comment:279 Changed 5 years ago by
Replying to jdemeyer:
See https://docs.python.org/2/distutils/sourcedist.html#commands
AHAAA, now I see what went wrong. I called it Manifest.in, not MANIFEST.in.
comment:280 Changed 5 years ago by
Commit:  98e6c213914bfe601a54e66640c5425da03bc161 → f06de90c639233c6219cf13b60f302ee5c864e3a 

Branch pushed to git repo; I updated commit sha1. New commits:
f06de90  p_group_cohomology3.0: Use xz, not gzip

comment:281 Changed 5 years ago by
Description:  modified (diff) 

Status:  needs_work → needs_review 
Note that the upstream tarball name has changed from ...tar.gz to ...tar.xz. It was a good suggestion to use xz, as the new tarball's size is only about 54% of the old.
I hope the packaging is now cleaner, thanks to using MANIFEST.in. I took the liberty to use SPKG.txt as README file in the Python part of the package (python setup.py sdist
expects some kind of README). Hope that's OK.
I checked that the package installs fine and correctly builds the docs. In particular, the checksum provided in the latest commit should be fine.
comment:282 followup: 284 Changed 5 years ago by
Status:  needs_review → needs_work 

There is still something wrong with the installation:
sage t pyxsources/pGroupCohomology/auxiliaries.py ********************************************************************** File "pyxsources/pGroupCohomology/auxiliaries.py", line 56, in pGroupCohomology.auxiliaries.safe_save Failed example: from pGroupCohomology.auxiliaries import safe_save Exception raised: Traceback (most recent call last): File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 534, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 937, in compile_and_execute exec(compiled, globs) File "<doctest pGroupCohomology.auxiliaries.safe_save[0]>", line 1, in <module> from pGroupCohomology.auxiliaries import safe_save File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/pGroupCohomology/__init__.py", line 2481, in <module> from pGroupCohomology.factory import CohomologyRing File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/pGroupCohomology/factory.py", line 43, in <module> from pGroupCohomology.auxiliaries import coho_options, coho_logger, safe_save, _gap_init, gap, singular File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/pGroupCohomology/auxiliaries.py", line 488, in <module> _gap_init() File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/pGroupCohomology/auxiliaries.py", line 469, in __call__ G.eval('Read("{}");'.format(os.path.join(SAGE_ROOT,'local','share','sage','ext','gap','modular_cohomology','GapMaxels.g'))) File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/interfaces/gap.py", line 581, in eval result = Expect.eval(self, input_line, **kwds) File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/interfaces/expect.py", line 1297, in eval for L in code.split('\n') if L != '']) File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/interfaces/gap.py", line 777, in _eval_line raise RuntimeError(message) RuntimeError: Gap produced error output Error, file "/home/jdemeyer/sagetest/local/share/sage/ext/gap/modular_cohomology/GapMaxels\ .g" must exist and be readable executing Read("/home/jdemeyer/sagetest/local/share/sage/ext/gap/modular_cohomology/GapMaxels.g"); **********************************************************************
The file is installed in $SAGE_SHARE/sage/ext/gap/GapMaxels.g
instead of $SAGE_SHARE/sage/ext/gap/modular_cohomology/GapMaxels.g
comment:283 followup: 285 Changed 5 years ago by
I noticed several serious compiler warnings during the build of the csources, for example:
slice.c: In function 'loadBlock': slice.c:325:8: warning: implicit declaration of function 'readhdrplus' [Wimplicitfunctiondeclaration] fp = readhdrplus(storedProductFile(ngs, ngs>dimLoaded), NULL, &totrows, ^~~~~~~~~~~ slice.c:325:6: warning: assignment makes pointer from integer without a cast [Wintconversion] fp = readhdrplus(storedProductFile(ngs, ngs>dimLoaded), NULL, &totrows, ^
comment:284 Changed 5 years ago by
Replying to jdemeyer:
The file is installed in
$SAGE_SHARE/sage/ext/gap/GapMaxels.g
instead of$SAGE_SHARE/sage/ext/gap/modular_cohomology/GapMaxels.g
Interestingly, I have the file installed in both locations. That's probably an artefact of a previous installation. I'll delete them and see where they will be installed.
comment:285 followup: 288 Changed 5 years ago by
Replying to jdemeyer:
I noticed several serious compiler warnings during the build of the csources, for example:
slice.c: In function 'loadBlock': slice.c:325:8: warning: implicit declaration of function 'readhdrplus' [Wimplicitfunctiondeclaration] fp = readhdrplus(storedProductFile(ngs, ngs>dimLoaded), NULL, &totrows, ^~~~~~~~~~~ slice.c:325:6: warning: assignment makes pointer from integer without a cast [Wintconversion] fp = readhdrplus(storedProductFile(ngs, ngs>dimLoaded), NULL, &totrows, ^
Thanks for spotting it. readhdrplus is declared in fp_decls.h, but the header is not included in slice.c.
I too noticed some warnings. But which ones are serious and which aren't? What is the criterion?
comment:286 Changed 5 years ago by
Commit:  f06de90c639233c6219cf13b60f302ee5c864e3a → b6b55f18b852bd2024c7d55974858ef20fb7af82 

Branch pushed to git repo; I updated commit sha1. New commits:
b6b55f1  p_group_cohomology3.0: Upstream tarball with fixed compiler warnings

comment:287 Changed 5 years ago by
Status:  needs_work → needs_review 

The current version of modular_resolution (as subpackage of p_group_cohomology) should now install without warning. The only warnings left in the installation of the package seem harmless to me: Some cython variables may be used uninitialised (but in fact I checked at some point that they will definitely be used initialised; their initialisation is in some "ifthenelse" construct), some probably harmless warning from python setup.py ...
:
reading manifest template 'MANIFEST.in' warning: no files found matching '*.pxi' under directory 'pGroupCohomology' warning: no previouslyincluded files matching '*.o' found under directory 'pGroupCohomology' warning: no previouslyincluded files matching '*.so' found under directory 'pGroupCohomology' warning: no previouslyincluded files matching '*' found under directory 'doc/build'
and a warning concerning docs:
copying static files... WARNING: html_static_path entry u'/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.0/src/pyxsources/doc/source/_static' does not exist
I would expect that graft doc/source
in MANIFEST.in would also copy the empty folder doc/source/_static
, but apparently it doesn't. Since the documenation builds fine, I'd say one can ignore that warning.
comment:288 followup: 289 Changed 5 years ago by
Replying to SimonKing:
But which ones are serious and which aren't? What is the criterion?
I can't say that there is an absolute criterion, but the Wimplicitfunctiondeclaration
is actually an error in both C++ and C99. So it's barely standardcompliant code.
comment:289 followup: 290 Changed 5 years ago by
Replying to jdemeyer:
So it's barely standardcompliant code.
Or rather: It was. The warnings about implicit function declaration are gone now.
comment:290 Changed 5 years ago by
Replying to SimonKing:
Or rather: It was. The warnings about implicit function declaration are gone now.
Yes, I understood that. I was just replying to that comment.
comment:291 followup: 292 Changed 5 years ago by
On a ppc64le machine, I'm getting some doctest failures:
[p_group_cohomology3.0] sage t pyxsources/pGroupCohomology/__init__.py [p_group_cohomology3.0] ********************************************************************** [p_group_cohomology3.0] File "pyxsources/pGroupCohomology/__init__.py", line 674, in pGroupCohomology [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] H.8.massey_power() [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 534, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 937, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology[100]>", line 1, in <module> [p_group_cohomology3.0] H.gen(8).massey_power() [p_group_cohomology3.0] File "pGroupCohomology/cochain.pyx", line 2041, in pGroupCohomology.cochain.COCH.massey_power [p_group_cohomology3.0] File "pGroupCohomology/cochain.pyx", line 4522, in pGroupCohomology.cochain.YCOCH.find_cobounding_yoneda_cochains [p_group_cohomology3.0] File "sage/structure/element.pyx", line 3673, in sage.structure.element.Matrix.__mul__ (build/cythonized/sage/structure/element.c:23460) [p_group_cohomology3.0] return coercion_model.bin_op(left, right, mul) [p_group_cohomology3.0] File "sage/structure/coerce.pyx", line 1113, in sage.structure.coerce.CoercionModel_cache_maps.bin_op (build/cythonized/sage/structure/coerce.c:9671) [p_group_cohomology3.0] action = self.get_action(xp, yp, op, x, y) [p_group_cohomology3.0] File "sage/structure/coerce.pyx", line 1656, in sage.structure.coerce.CoercionModel_cache_maps.get_action (build/cythonized/sage/structure/coerce.c:16889) [p_group_cohomology3.0] action = self.discover_action(R, S, op, r, s) [p_group_cohomology3.0] File "sage/structure/coerce.pyx", line 1806, in sage.structure.coerce.CoercionModel_cache_maps.discover_action (build/cythonized/sage/structure/coerce.c:18411) [p_group_cohomology3.0] action = (<Parent>R).get_action(S, op, True, r, s) [p_group_cohomology3.0] File "sage/structure/parent.pyx", line 2495, in sage.structure.parent.Parent.get_action (build/cythonized/sage/structure/parent.c:21664) [p_group_cohomology3.0] action = self.discover_action(S, op, self_on_left, self_el, S_el) [p_group_cohomology3.0] File "sage/structure/parent.pyx", line 2606, in sage.structure.parent.Parent.discover_action (build/cythonized/sage/structure/parent.c:23017) [p_group_cohomology3.0] if parent_is_integers(S) and not self.has_coerce_map_from(S): [p_group_cohomology3.0] File "sage/structure/parent.pyx", line 2024, in sage.structure.parent.Parent.has_coerce_map_from (build/cythonized/sage/structure/parent.c:17925) [p_group_cohomology3.0] return self._internal_coerce_map_from(S) is not None [p_group_cohomology3.0] File "sage/structure/parent.pyx", line 2166, in sage.structure.parent.Parent._internal_coerce_map_from (build/cythonized/sage/structure/parent.c:18944) [p_group_cohomology3.0] mor = self.discover_coerce_map_from(S) [p_group_cohomology3.0] File "sage/structure/parent.pyx", line 2303, in sage.structure.parent.Parent.discover_coerce_map_from (build/cythonized/sage/structure/parent.c:19396) [p_group_cohomology3.0] user_provided_mor = self._coerce_map_from_(S) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 341, in sage.structure.parent_old.Parent._coerce_map_from_ (build/cythonized/sage/structure/parent_old.c:7549) [p_group_cohomology3.0] return self.coerce_map_from_c(S) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 159, in sage.structure.parent_old.Parent.coerce_map_from_c (build/cythonized/sage/structure/parent_old.c:4044) [p_group_cohomology3.0] mor = self.coerce_map_from_c(sage_type) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 143, in sage.structure.parent_old.Parent.coerce_map_from_c (build/cythonized/sage/structure/parent_old.c:3768) [p_group_cohomology3.0] mor = self.coerce_map_from_c_impl(S) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 191, in sage.structure.parent_old.Parent.coerce_map_from_c_impl (build/cythonized/sage/structure/parent_old.c:4710) [p_group_cohomology3.0] if self.has_coerce_map_from_c(S): [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 277, in sage.structure.parent_old.Parent.has_coerce_map_from_c (build/cythonized/sage/structure/parent_old.c:6312) [p_group_cohomology3.0] x = self.has_coerce_map_from_c_impl(S) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 286, in sage.structure.parent_old.Parent.has_coerce_map_from_c_impl (build/cythonized/sage/structure/parent_old.c:6489) [p_group_cohomology3.0] self._coerce_c((<parent.Parent>S).an_element()) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 219, in sage.structure.parent_old.Parent._coerce_c (build/cythonized/sage/structure/parent_old.c:5382) [p_group_cohomology3.0] return self._coerce_impl(x) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/matrix/matrix_space.py", line 1102, in _coerce_impl [p_group_cohomology3.0] return self._coerce_try(x, self.base_ring()) [p_group_cohomology3.0] File "sage/structure/parent_old.pyx", line 257, in sage.structure.parent_old.Parent._coerce_try (build/cythonized/sage/structure/parent_old.c:5902) [p_group_cohomology3.0] return self(y) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/matrix/matrix_space.py", line 875, in __call__ [p_group_cohomology3.0] return self.matrix(entries, coerce, copy) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/matrix/matrix_space.py", line 1888, in matrix [p_group_cohomology3.0] return MC(self, x, copy=copy, coerce=coerce) [p_group_cohomology3.0] File "sage/matrix/matrix_gfpn_dense.pyx", line 425, in sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense.__init__ (build/cythonized/sage/matrix/matrix_gfpn_dense.c:5278) [p_group_cohomology3.0] raise ValueError("Cannot initialise nonsquare matrix from {}".format(data)) [p_group_cohomology3.0] ValueError: Cannot initialise nonsquare matrix from 1 [p_group_cohomology3.0] ********************************************************************** [p_group_cohomology3.0] File "pyxsources/pGroupCohomology/__init__.py", line 676, in pGroupCohomology [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] H.element_as_polynomial(_) [p_group_cohomology3.0] Expected: [p_group_cohomology3.0] b_2_0^2*a_1_1*a_3_5+b_2_0^2*a_1_0*a_3_4+b_2_0*c_6_8: 8Cocycle in H^*(E27; GF(3)) [p_group_cohomology3.0] Got: [p_group_cohomology3.0] a_3_4: 3Cocycle in H^*(E27; GF(3)) [p_group_cohomology3.0] **********************************************************************
and various similar failures.
comment:292 followup: 295 Changed 5 years ago by
Replying to jdemeyer:
On a ppc64le machine, I'm getting some doctest failures:
... [p_group_cohomology3.0] ValueError: Cannot initialise nonsquare matrix from 1
That's very odd. Why should there be a nonsquare matrix involved? Can you please try on that machine whether you get a different result than I in the following code:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_user_db(tmp_dir()) sage: H = CohomologyRing(27,3) sage: H.make() sage: print H Cohomology ring of Extraspecial 3group of order 27 and exponent 3 with coefficients in GF(3) Computation complete Minimal list of generators: [b_2_0: 2Cocycle in H^*(E27; GF(3)), b_2_1: 2Cocycle in H^*(E27; GF(3)), b_2_2: 2Cocycle in H^*(E27; GF(3)), b_2_3: 2Cocycle in H^*(E27; GF(3)), c_6_8: 6Cocycle in H^*(E27; GF(3)), a_1_0: 1Cocycle in H^*(E27; GF(3)), a_1_1: 1Cocycle in H^*(E27; GF(3)), a_3_4: 3Cocycle in H^*(E27; GF(3)), a_3_5: 3Cocycle in H^*(E27; GF(3))] Minimal list of algebraic relations: [a_1_0*a_1_1, b_2_1*a_1_0b_2_0*a_1_1, b_2_2*a_1_1+b_2_1*a_1_1b_2_0*a_1_1, b_2_2*a_1_0b_2_1*a_1_1+b_2_0*a_1_1, b_2_3*a_1_0b_2_0*a_1_1, b_2_1^2+b_2_0*b_2_2+b_2_0*b_2_1, b_2_2^2+b_2_1*b_2_3+b_2_1*b_2_2b_2_1^2, b_2_1*b_2_2b_2_1^2+b_2_0*b_2_3, b_2_2^2+b_2_1*b_2_2+a_1_1*a_3_4, b_2_1*b_2_2b_2_1^2+b_2_0*b_2_1+a_1_0*a_3_4, b_2_2*b_2_3+b_2_2^2+a_1_1*a_3_5, b_2_2^2b_2_1^2+b_2_0*b_2_1+a_1_0*a_3_5, b_2_3*a_3_4b_2_1*a_3_4, b_2_2*a_3_4+b_2_1*a_3_5+b_2_1*a_3_4+b_2_0*b_2_1*a_1_1b_2_0^2*a_1_1, b_2_1*a_3_4+b_2_0*a_3_5b_2_0*a_3_4b_2_0*b_2_1*a_1_1+b_2_0^2*a_1_1, b_2_2*a_3_5+b_2_0*b_2_1*a_1_1b_2_0^2*a_1_1, a_3_4*a_3_5+b_2_0*a_1_1*a_3_5+b_2_0*a_1_0*a_3_5b_2_0*a_1_0*a_3_4] sage: print H.8 3Cocycle in H^*(E27; GF(3)), represented by [0 0 0 0 1 0] rdeg = 0 ydeg = 1
and various similar failures.
All in .massey_power
, or in very different kinds of tests? What is particular about ppc64le? If I recall correctly, there have been some issues on that machine with meataxe matrices (that we have solved, though).
comment:293 Changed 5 years ago by
Status:  needs_review → needs_work 

Work issues:  → Investigate failing tests 
Ouch, sorry. I just realise that I get the same error.
Actually I did successfully run the test suite in a previous version of the package, and I didn't expect that the changes I did since the last test could possibly affect the computations: The changes have been in the build system and in avoiding compiler warnings, IIRC.
comment:294 Changed 5 years ago by
Sorry, but in the next few days I will very probably not find the time to fix it.
comment:295 followup: 296 Changed 5 years ago by
Replying to SimonKing:
If I recall correctly, there have been some issues on that machine with meataxe matrices (that we have solved, though).
Yes, the char
!= signed char
issue.
comment:296 followup: 297 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
If I recall correctly, there have been some issues on that machine with meataxe matrices (that we have solved, though).
Yes, the
char
!=signed char
issue.
Yes, that was it. Under what circumstances would it be a problem? Only if one treats a char like a byte, i.e., if one does arithmetic computations with it, right? I briefly hat a look at the code of the modular_resolution subpackage, and it seems that char is really only used for letters, which should be fine, isn't it?
In any case, I see the same error on my computer, too. So, there must be a recent fix that broke things...
comment:297 Changed 5 years ago by
Replying to SimonKing:
Yes, that was it. Under what circumstances would it be a problem?
Typically, when using char
for arithmetic. For strings, it probably does not matter.
comment:298 Changed 5 years ago by
I didn't find the time to work on the current bug (see comment:292), but here is something else.
Note to self: Use #22611 for the docstrings of instances (here: the @permanent_result and @temporary_result decorators)
comment:299 Changed 5 years ago by
Meanwhile I got the impression that the reason for ValueError: Cannot initialise nonsquare matrix from 1
stems from a change in Sage and not from a change in my code. Namely, I went back to a version in which I know that all doctests used to pass, changed some cdef _mul_
into cpdef _mul_
and tested again. But I still got the same ValueError.
So, apparently I need to initialise the matrix differently.
comment:300 Changed 5 years ago by
As it turns out, the whole optional meataxe backend somehow got broken. There have been some matrix related changes in Sage recently  and apparently they haven't been tested with optional packages installed. I'll open a ticket for that issue.
comment:301 Changed 5 years ago by
To be more specific: Multiplication of a nonsquare matrix with the integer 1 is not calling _lmul_ (which it is supposed to!) but tries to convert the integer 1 into the parent of the matrix (which is idiotic since it isn't an algebra), and of course fails as the integer 1 cannot be represented by a nonsquare matrix.
comment:303 Changed 5 years ago by
Dependencies:  #25076 → #25079 

comment:304 Changed 5 years ago by
Commit:  b6b55f18b852bd2024c7d55974858ef20fb7af82 → 138814b1c5fac5c150379e778bd9876d376860c6 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
f0e97af  Modify the test that ensures that #24742 remains fixed

31f085e  Enable _mul_long for matrices

cae999a  More stuff in the meataxe interface, and a meataxe helper function

71a546b  Fix reading MeatAxe matrix from a file

b9986d5  Add p_group_cohomology3.0 spkg

55b1364  p_group_cohomology3.0: Cope with some API change, improve SPKG.txt

e63bc23  p_group_cohomology3.0: Fix checksum, dependencies and SPKG.txt

58858a3  p_group_cohomology3.0: Remove useless test in spkginstall

14d28e2  p_group_cohomology3.0: Use xz, not gzip

138814b  p_group_cohomology3.0: Upstream tarball with fixed compiler warnings

comment:305 Changed 5 years ago by
Status:  needs_work → needs_review 

Work issues:  Investigate failing tests 
With the current upstream tarball (posted at the location stated in the ticket description) and the current branch (and the dependency), the package builds and passes its test suite. Finally it can be reverted to "needs review"!
comment:306 followups: 308 313 Changed 5 years ago by
On OS X, two issues: first, I couldn't get it to install at all because
/bin/bash: .././installsh: Permission denied
Once I changed permissions on src/csources/installsh
to make it executable, the package installed.
Second, I get lots of doctest failures:
********************************************************************** File "pyxsources/pGroupCohomology/auxiliaries.py", line 56, in pGroupCohomology.auxiliaries.safe_save Failed example: from pGroupCohomology.auxiliaries import safe_save Exception raised: Traceback (most recent call last): File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 551, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 961, in compile_and_execute exec(compiled, globs) File "<doctest pGroupCohomology.auxiliaries.safe_save[0]>", line 1, in <module> from pGroupCohomology.auxiliaries import safe_save File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/__init__.py", line 2481, in <module> from pGroupCohomology.factory import CohomologyRing File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.py", line 43, in <module> from pGroupCohomology.auxiliaries import coho_options, coho_logger, safe_save, _gap_init, gap, singular File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/auxiliaries.py", line 45, in <module> import sage.libs.meataxe ImportError: No module named meataxe
I just ran sage b
and now I'm trying again...
comment:307 followup: 309 Changed 5 years ago by
To clarify, I only ran ./sage i c p_group_cohomology
, at which point Sage installed database_gap
and meataxe
and then p_group_cohomology
, running the test suite for each. Running ./sage b
and then repeating the ./sage i c ...
command seems to have worked: doctests are passing.
A question: why not run long tests, since I see that a few tests are marked # long time
? I haven't tried that; how much time would it add to the tests?
comment:308 followup: 311 Changed 5 years ago by
Replying to jhpalmieri:
On OS X, two issues: first, I couldn't get it to install at all because
/bin/bash: .././installsh: Permission denied
I am a bit puzzled, as on my machine I have
$ ls l installsh rwrr 1 king king 13997 Jan 7 2016 installsh
and thus it isn't executable either, but the package does install.
But if it is needed on some machines to change the permission, then I should do it.
Second, I get lots of doctest failures:
********************************************************************** File "pyxsources/pGroupCohomology/auxiliaries.py", line 56, in pGroupCohomology.auxiliaries.safe_save Failed example: from pGroupCohomology.auxiliaries import safe_save Exception raised: Traceback (most recent call last): File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 551, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 961, in compile_and_execute exec(compiled, globs) File "<doctest pGroupCohomology.auxiliaries.safe_save[0]>", line 1, in <module> from pGroupCohomology.auxiliaries import safe_save File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/__init__.py", line 2481, in <module> from pGroupCohomology.factory import CohomologyRing File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.py", line 43, in <module> from pGroupCohomology.auxiliaries import coho_options, coho_logger, safe_save, _gap_init, gap, singular File "/Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/auxiliaries.py", line 45, in <module> import sage.libs.meataxe ImportError: No module named meataxeI just ran
sage b
and now I'm trying again...
Hm. Now I am even more puzzled.
meataxe
is declared as a dependency of the package. Thus, installation of the cohomology package should fail if meataxe isn't installed  and installation of meataxe does involve sage b
, as there are optional cython modules in the Sage library.
Maybe I should ask on sagedevel what exactly is the meaning of a dependency of a package, and why install.sh works on some machines and on others not.
comment:309 followup: 312 Changed 5 years ago by
Replying to jhpalmieri:
To clarify, I only ran
./sage i c p_group_cohomology
, at which point Sage installeddatabase_gap
andmeataxe
Hm. I believe the dependencies shouldn't be automatically installed. I expected it to fail with an error message asking the user to install both dependencies.
and then
p_group_cohomology
, running the test suite for each. Running./sage b
and then repeating the./sage i c ...
command seems to have worked: doctests are passing.
Yes, sage b
is essential for installing meataxe.
A question: why not run long tests, since I see that a few tests are marked
# long time
? I haven't tried that; how much time would it add to the tests?
I think at some point I have considered that. I will see what it gives. The "short" tests already have a cumulative wall time of 2082.1 seconds on my machine.
comment:310 Changed 5 years ago by
Actually the time for sage t long
happened to be shorter than the time for sage t
(so, perhaps my laptop was more busy when I installed and tested the package in its current version and had less load when I retested with long tests).
That indicates that the long tests should be part of the test suite. I'll change it accordingly, but will wait for input of sagedevel concerning the other issues.
comment:311 followup: 315 Changed 5 years ago by
Replying to SimonKing:
installation of meataxe does involve
sage b
, as there are optional cython modules in the Sage library.
No, it does not and this is why it fails.
This is not easy to solve because make
does not allow to specify optional dependencies (*). Ideally, we would like to say that sagelib
depends on meataxe
if meataxe
has been installed but I don't know how to do that.
(*) interestingly, it does allow the exact opposite: you can specify that A depends on B only if B has not been installed yet.
comment:312 followup: 316 Changed 5 years ago by
Replying to SimonKing:
I believe the dependencies shouldn't be automatically installed.
Really? Why not? That's exactly the whole point of specifying dependencies.
comment:313 followups: 317 318 Changed 5 years ago by
If the testsuite depends on running Sage, then $(SAGERUNTIME)
must be specified as dependency: just append $(SAGERUNTIME)
to build/pkgs/p_group_cohomology/dependencies
.
comment:314 Changed 5 years ago by
Status:  needs_review → needs_work 

comment:315 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
installation of meataxe does involve
sage b
, as there are optional cython modules in the Sage library.No, it does not and this is why it fails.
Maybe I should have been clearer: There are optional Cython modules in the Sage library depending on meataxe, and the cohomology package uses both meataxe as a clibrary libmtx and uses the optional Cython modules in the Sage library.
But while the dependency on libmtx is a build time dependency, IIUC the dependency on the optional Cython modules is a runtime dependency (that's because the .pxd headers are present anyway).
This is not easy to solve because
make
does not allow to specify optional dependencies (*). Ideally, we would like to say thatsagelib
depends onmeataxe
ifmeataxe
has been installed but I don't know how to do that.
It was suggested on sagedevel that meataxe
's spkginstall
should end with sage b
. So, if it is OK that an optional package triggers building the optional modules in the sage library, that'll be the cleanest solution, I think.
comment:316 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
I believe the dependencies shouldn't be automatically installed.
Really? Why not? That's exactly the whole point of specifying dependencies.
If I recall correctly, it is totally fine for a GPL package to have a runtime dependency that isn't GPL (but it is not OK if it is a build time dependency). Is it OK if installation of a GPL package automatically triggers the installation of a nonGPL package that will be used at runtime?
And that's the situation here. database_gap is still nonGPL, although IIRC its authors finally agreed to relicense it. And p_group_cohomology uses database_gap at run time (i.e., also its test suite depends on it).
comment:317 Changed 5 years ago by
Replying to jdemeyer:
If the testsuite depends on running Sage, then
$(SAGERUNTIME)
must be specified as dependency: just append$(SAGERUNTIME)
tobuild/pkgs/p_group_cohomology/dependencies
.
Cool thing!
Would it only trigger sage b
if the test suite is run, or also when just building the package without running tests?
comment:318 followup: 324 Changed 5 years ago by
Replying to jdemeyer:
If the testsuite depends on running Sage, then
$(SAGERUNTIME)
must be specified as dependency: just append$(SAGERUNTIME)
tobuild/pkgs/p_group_cohomology/dependencies
.
I couldn't find a documentation of $(SAGERUNTIME)
.
Where exactly should $(SAGERUNTIME)
be placed? I.e., should build/pkgs/p_group_cohomology/dependencies
be
meataxe  database_gap xz $(SAGERUNTIME)
or rather
meataxe $(SAGERUNTIME)  database_gap xz
?
Note that p_group_cohomology links against the Sage library. So, I'd guess it should be the latter version, correct?
comment:319 followup: 320 Changed 5 years ago by
Doctest failures:
Test ran for 290.75 s ********************************************************************** File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 134, in pGroupCohomology.modular_cohomology._IdGroup Failed example: sorted(D.items()) Expected: [(2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)] Got: [('prime', 3), (2520, {0: H^*(8gp3_2520_0; GF(3))})] ********************************************************************** File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 155, in pGroupCohomology.modular_cohomology._IdGroup Failed example: sorted(D.items()) Expected: [(360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)] Got: [('prime', 3), (360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))})] **********************************************************************
and
File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 159, in pGroupCohomology.modular_cohomology._IdGroup Failed example: sorted(D.items()) Expected: [(360, {118: H^*(SmallGroup(360,118); GF(3))}), (2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)] Got: [('prime', 3), (360, {118: H^*(SmallGroup(360,118); GF(3))}), (2520, {0: H^*(8gp3_2520_0; GF(3))})] **********************************************************************
and
********************************************************************** File "pyxsources/pGroupCohomology/resolution.pyx", line 1017, in pGroupCohomology.resolution.RESL Failed example: 0.5 < Tu.stats[3]*D.get(Tu.stats[4],1)/(Ta.stats[3]*D.get(Ta.stats[4],1)) < 2 Expected: True Got: False **********************************************************************
comment:320 followups: 321 323 Changed 5 years ago by
Replying to jhpalmieri:
Doctest failures:
Test ran for 290.75 s ********************************************************************** File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 134, in pGroupCohomology.modular_cohomology._IdGroup Failed example: sorted(D.items()) Expected: [(2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)] Got: [('prime', 3), (2520, {0: H^*(8gp3_2520_0; GF(3))})] ********************************************************************** File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 155, in pGroupCohomology.modular_cohomology._IdGroup Failed example: sorted(D.items()) Expected: [(360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)] Got: [('prime', 3), (360, {}), (2520, {0: H^*(8gp3_2520_0; GF(3))})] **********************************************************************and
File "pyxsources/pGroupCohomology/modular_cohomology.pyx", line 159, in pGroupCohomology.modular_cohomology._IdGroup Failed example: sorted(D.items()) Expected: [(360, {118: H^*(SmallGroup(360,118); GF(3))}), (2520, {0: H^*(8gp3_2520_0; GF(3))}), ('prime', 3)] Got: [('prime', 3), (360, {118: H^*(SmallGroup(360,118); GF(3))}), (2520, {0: H^*(8gp3_2520_0; GF(3))})] **********************************************************************
Interesting. Apparently these are the same dictionaries, but sorting works differently (i.e., is machine dependent).
So, question: How to test that a returned dictionary is what one has expected, if sorting the items doesn't work? Perhaps I should just remove the key prime
, assuming that sorting integers still works.
********************************************************************** File "pyxsources/pGroupCohomology/resolution.pyx", line 1017, in pGroupCohomology.resolution.RESL Failed example: 0.5 < Tu.stats[3]*D.get(Tu.stats[4],1)/(Ta.stats[3]*D.get(Ta.stats[4],1)) < 2 Expected: True Got: False **********************************************************************
Interesting. That test is supposed to compare the efficiency of different algorithms, and when I created the tests I took care that the quotients of timings are well within the range [0.5, 2]
.
Is it possible for you to manually recreate the test, to see in what range the quotient of timings is located? But perhaps I should just mark this test "random".
comment:321 followup: 322 Changed 5 years ago by
Replying to SimonKing:
Interesting. Apparently these are the same dictionaries, but sorting works differently (i.e., is machine dependent).
So, question: How to test that a returned dictionary is what one has expected, if sorting the items doesn't work? Perhaps I should just remove the key
prime
, assuming that sorting integers still works.
You can test sorted(D.items()) == sorted([actual list])
or test whether blah in D.items()
is True. Sorting integers should work, so removing prime
would also work.
comment:322 Changed 5 years ago by
Replying to jhpalmieri:
You can test
sorted(D.items()) == sorted([actual list])
Right. Or I could actually manually create a copy of the expected dictionary and compare with the actual output. Good idea.
comment:323 followup: 325 Changed 5 years ago by
Replying to SimonKing:
Interesting. That test is supposed to compare the efficiency of different algorithms, and when I created the tests I took care that the quotients of timings are well within the range
[0.5, 2]
.Is it possible for you to manually recreate the test, to see in what range the quotient of timings is located? But perhaps I should just mark this test "random".
I spent a few minutes trying to recreate it, but I got tired of tracking down all of the definitions to do it by hand. When I run the test (by doing ./sage t forcelib /path/to/resolution.pyx
), it fails every time. I added a statement printing the quantity in the inequality, and it comes out around between 10 and 11 each time. Maybe marking it # random
is the right thing to do.
comment:324 followup: 326 Changed 5 years ago by
Replying to SimonKing:
It was suggested on sagedevel that
meataxe
'sspkginstall
should end withsage b
.
A very bad idea. That will badly break parallel builds: you might end up with two instances of make
both building the Sage library.
Replying to SimonKing:
Would it only trigger
sage b
if the test suite is run, or also when just building the package without running tests?
It will trigger sage b
unconditionally.
Note that it is not guaranteed that it will run sage b
after building meataxe
(this is the part that cannot be fixed). In a serial build make
uses the order as specified in the dependencies, so that is why it is better to put $(SAGERUNTIME)
after meataxe
. But in a parallel build, make
can build meataxe
and sagelib
in any order.
Replying to SimonKing:
Where exactly should
$(SAGERUNTIME)
be placed? I.e., shouldbuild/pkgs/p_group_cohomology/dependencies
bemeataxe  database_gap xz $(SAGERUNTIME)
This is the right one. The other alternative would rebuild p_group_cohomology
every time.
comment:325 followup: 327 Changed 5 years ago by
Replying to jhpalmieri:
I spent a few minutes trying to recreate it, but I got tired of tracking down all of the definitions to do it by hand. When I run the test (by doing
./sage t forcelib /path/to/resolution.pyx
), it fails every time. I added a statement printing the quantity in the inequality, and it comes out around between 10 and 11 each time. Maybe marking it# random
is the right thing to do.
Thank you! That's very interesting, and I'll have a look what these numbers mean.
comment:326 followup: 330 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Where exactly should
$(SAGERUNTIME)
be placed? I.e., shouldbuild/pkgs/p_group_cohomology/dependencies
bemeataxe  database_gap xz $(SAGERUNTIME)This is the right one. The other alternative would rebuild
p_group_cohomology
every time.
What do you mean by "every time"? Do you mean that a change in the Sage library followed by make
would trigger rebuilding p_group_cohomology
? In that case I'd actually prefer the alternative, as I've seen too often that a change in sagelib leaves the cohomology spkg broken, making it necessary to rebuild it.
Ideally, it should of course only be rebuilt if critical headers change, say, sage/structure/element.pxd. But I guess that's not possible, unless all pyx files are put into the sage library.
comment:327 Changed 5 years ago by
Replying to SimonKing:
Replying to jhpalmieri:
I spent a few minutes trying to recreate it, but I got tired of tracking down all of the definitions to do it by hand. When I run the test (by doing
./sage t forcelib /path/to/resolution.pyx
), it fails every time. I added a statement printing the quantity in the inequality, and it comes out around between 10 and 11 each time. Maybe marking it# random
is the right thing to do.Thank you! That's very interesting, and I'll have a look what these numbers mean.
I see now. What is compared are two ways to lift a chain map. One method ("autolift") is used by default in small degrees, as it is fast but tends to consume much memory in higher degrees; thus, the other method ("Urbild Gröbnerbasis") is used in higher degrees.
It is stated in the comment before the text that autolift used to be 250 times faster than UrbildGB, but after some recent optimisations they became comparable. They still are comparable, on my machine (since the test passes), but on your machine autolift is about 10 times faster than UrbildGB.
I will change the test so that I give the output on my machine, mark the test as "random", and include in the comment that the ratio is highly machine dependent.
comment:328 Changed 5 years ago by
Here is the test:
sage: from pGroupCohomology.cochain import COCH sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_user_db(tmp_dir()) sage: H = CohomologyRing(8,3, from_scratch=True) sage: R = H.resolution() sage: C = COCH(H,2,'C',[1,0,1]) sage: c0 = R.CochainToChainmap(2, C.MTX()) sage: c1 = R.liftChainMap(c0) sage: R.makeAutolift(2) sage: timeit.eval('cX = R.liftChainMap(c1)') 625 loops, best of 3: 230 µs per loop sage: timeit.eval('cX = R.ugb_liftChainMap(c1[0]+1,c1[1]+1,c1[2])') 625 loops, best of 3: 297 µs per loop
What do you think: Is it better to have a test that checks that the ratio of execution times is at least a certain number (I am sure I'll find an example where the timings consistently differ by a factor of at least 5)? Or is it better to have a random test and just give the raw figures?
comment:329 Changed 5 years ago by
I would prefer a random test and just the raw figures. That will be more robust.
comment:330 followup: 332 Changed 5 years ago by
Replying to SimonKing:
Do you mean that a change in the Sage library followed by
make
would trigger rebuildingp_group_cohomology
?
Yes, even without a change in the Sage library.
comment:331 followup: 343 Changed 5 years ago by
Concerning autolift: I am currently playing with the default degree up to which the autolift method is used. Let n
be a number, and then I do
sage: from pGroupCohomology.factory import unit_test_64 sage: from pGroupCohomology.auxiliaries import coho_options sage: coho_options['autolift'] = n sage: unit_test_64()
Here are the figures I'm getting (the first item should be []
, and the second item is a tuple "wall time, cpu time of Sage, cpu time of Singular, cpu time of GAP":
 n=0
([], [396.6325900554657, 352.84, 38.83, 18.056])
 n=1, which is the current default
([], [391.3751630783081, 350.676, 37.89, 17.484])
 n=2, which used to be the default in an older version of the spkg
([], [386.2917420864105, 345.348, 37.52, 17.628])
 n=3
([], [386.8177170753479, 346.348, 37.6, 18.056])
 n=4
([], [385.62233090400696, 346.06399999999996, 37.18, 17.368])
 n=5
([], [400.68896293640137, 356.97999999999996, 39.72, 18.344])
Of course each test was in a separate Sage session.
I think the numbers indicate that I should return to the old default, n=2. But, John, since the two lifting methods differ a lot more on your machine than on mine, could you perhaps do these tests as well and tell me the results?
comment:332 followup: 333 Changed 5 years ago by
@John, OK, I will use the robust method to expose the timings in tests.
Replying to jdemeyer:
Replying to SimonKing:
Do you mean that a change in the Sage library followed by
make
would trigger rebuildingp_group_cohomology
?Yes, even without a change in the Sage library.
Ouch, that's not what I want. So, I'll put $(SAGERUNTIME)
to the very end of the dependencies.
And who knows, perhaps in the future it will be possible to put everything into the sage library. My plan for version 3.1 is:
 Make
meataxe
standard (rationale: It is A LOT faster than the generic implementation of matrices over nonprime fields of odd characteristic)  Make
p_group_cohomology
work without the SmallGroups library. There are only few places where I really use the SmallGroups identifiers.
If I understand correctly, these used to be the two main obstructions against putting p_group_cohomology into the sage library.
comment:333 followup: 335 Changed 5 years ago by
Replying to SimonKing:
If I understand correctly, these used to be the two main obstructions against putting p_group_cohomology into the sage library.
Neither of those is actually an obstruction for moving the code to the Sage library. We already have code in sagelib depending on meataxe and other code depending on smallgroups.
comment:334 followup: 336 Changed 5 years ago by
I do not think it is worthwhile to spend any nontrivial amount of time on "work without the SmallGroups library". The new GAP release, 4.9, with the changed SmallGroups library license will be released very soon. While it might take some time to get 4.9 into Sage (!libGAP, you know...), we can trivially (I think) backport the "new" SmallGroups, if needed.
comment:335 followup: 337 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
If I understand correctly, these used to be the two main obstructions against putting p_group_cohomology into the sage library.
Neither of those is actually an obstruction for moving the code to the Sage library. We already have code in sagelib depending on meataxe and other code depending on smallgroups.
If you look at the development of this ticket, at some point I did put all cython and python modules into the sage library. I was told at some point that OptionalExtension
should only be used when the optional extension provides a different implementation of a functionality that is also provided by the default sage library. In contrast, one shouldn't use OptionalExtension
for functionality that would simply not be available without an optional spkg.
My dream for p_group_cohomology3.1
is to make it a standard package, and in that case all the python and cython modules should of course be in the sage library.
comment:336 followup: 339 Changed 5 years ago by
Replying to dimpase:
I do not think it is worthwhile to spend any nontrivial amount of time on "work without the SmallGroups library". The new GAP release, 4.9, with the changed SmallGroups library license will be released very soon. While it might take some time to get 4.9 into Sage (!libGAP, you know...), we can trivially (I think) backport the "new" SmallGroups, if needed.
Making database_gap a standard spkg?
comment:337 followup: 338 Changed 5 years ago by
Replying to SimonKing:
I was told at some point that
OptionalExtension
should only be used when the optional extension provides a different implementation of a functionality that is also provided by the default sage library. In contrast, one shouldn't useOptionalExtension
for functionality that would simply not be available without an optional spkg.
I don't know where you got this idea. There is nothing wrong with truly providing optional functionality (not available otherwise) this way.
comment:338 followup: 340 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
I was told at some point that
OptionalExtension
should only be used when the optional extension provides a different implementation of a functionality that is also provided by the default sage library. In contrast, one shouldn't useOptionalExtension
for functionality that would simply not be available without an optional spkg.I don't know where you got this idea. There is nothing wrong with truly providing optional functionality (not available otherwise) this way.
OK, in that case I could indeed return to the code layout that I had in some intermediate stage of development (i.e.: Have an optional module_resolution1.0
spkg and have everything else in the Sage library).
Potential issues:
 If I recall correctly, building documentation of an optional extension module is a problem. Can you confirm? What I'd really like to have is that the documentation of the optional extension in the Sage library is built even if the optional spkg is not installed.
 The cohomology computations need some helper files that rightfully belong to SAGE_SHARE (namely: code for Singular and GAP), which IIRC are not under version control. So, how to install them? Should
modular_resolution1.0
be install them?  Making the cohomology code optional extensions would mean to mark hundreds of doc tests as "optional: modular_resolution", which is highly awkward.
Would anything of the above be a showstopper? Do you see further problems with that code layout?
comment:340 followup: 342 Changed 5 years ago by
Replying to SimonKing:
 If I recall correctly, building documentation of an optional extension module is a problem. Can you confirm?
Indeed, that does not work.
 The cohomology computations need some helper files that rightfully belong to SAGE_SHARE (namely: code for Singular and GAP), which IIRC are not under version control. So, how to install them?
Just install them. Any package can install stuff in SAGE_SHARE
.
Should
modular_resolution1.0
be install them?
If they are logically considered part of that package, then yes.
 Making the cohomology code optional extensions would mean to mark hundreds of doc tests as "optional: modular_resolution", which is highly awkward.
Awkward but acceptable.
comment:341 Changed 5 years ago by
Note that I'm not suggesting that optional modules in the Sage library is the best solution. I'm just saying what the possibilities are.
comment:342 Changed 5 years ago by
Jeroen, I am also not suggesting that either of the possibilities is clearly best.
Replying to jdemeyer:
Replying to SimonKing:
 If I recall correctly, building documentation of an optional extension module is a problem. Can you confirm?
Indeed, that does not work.
I find documentation extremely important. How else could a user learn how to use to cohomology code? So, for now I'd prefer to keep the current layout (i.e., not using optional Cython extensions in the Sage library).
comment:343 followup: 344 Changed 5 years ago by
Replying to SimonKing:
Concerning autolift: I am currently playing with the default degree up to which the autolift method is used. Let
n
be a number, and then I dosage: from pGroupCohomology.factory import unit_test_64 sage: from pGroupCohomology.auxiliaries import coho_options sage: coho_options['autolift'] = n sage: unit_test_64()
I think the numbers indicate that I should return to the old default, n=2. But, John, since the two lifting methods differ a lot more on your machine than on mine, could you perhaps do these tests as well and tell me the results?
On two different machines. First, a slowish laptop:
 n=0:
([], [922.0159947872162, 690.2828870000001, 58.54, 22.397])
 n=1:
([], [889.0557789802551, 667.6387410000001, 57.38, 21.683])
 n=2:
([], [890.1577479839325, 666.102922, 58.01, 22.42])
 n=3:
([], [881.2708470821381, 658.588641, 57.66, 22.383])
 n=4:
([], [887.1860530376434, 667.076394, 57.81, 21.924])
 n=5:
([], [877.5219340324402, 655.3260490000001, 57.6, 22.078])
Next, a faster desktop:
 n=0:
([], [401.840313911438, 262.332941, 24.72, 10.603])
 n=1:
([], [378.8941731452942, 261.727588, 24.61, 10.553])
 n=2:
([], [391.03160214424133, 263.670184, 24.92, 10.487])
 n=3:
([], [392.65900707244873, 260.529421, 24.41, 10.477])
 n=4:
([], [383.2508010864258, 262.07234, 24.52, 10.461])
 n=5:
([], [386.03561902046204, 261.946665, 24.81, 10.523])
comment:344 followup: 345 Changed 5 years ago by
Replying to jhpalmieri:
On two different machines.
Thank you!
I don't see a very clear winner, but there is some tendency towards n=2 or n=3. I am now running some tests with groups of order 128. Since I am doing some changes in the package anyway, I can as well change the default behaviour of lifting, although it matters only slightly.
BTW, should I change the permission of install.sh?
comment:345 followup: 346 Changed 5 years ago by
Replying to SimonKing:
BTW, should I change the permission of install.sh?
It doesn't build for me otherwise, so either change it in the tar file or I suppose you could change spkginstall
:

build/pkgs/p_group_cohomology/spkginstall
diff git a/build/pkgs/p_group_cohomology/spkginstall b/build/pkgs/p_group_cohomology/spkginstall index d7ff1ee75b..82f940cecf 100644
a b cd src 2 2 3 3 # building modular_resolution1.0 4 4 cd csources 5 chmod a+x installsh 5 6 sdh_configure 6 7 sdh_make 7 8 # sdh_make install got broken by trac ticket #24106
(That's what I did, because then I didn't have to prepare a new tar file and change the checksums. It seems cleaner to change it in the actual tar file, though.)
comment:346 followup: 347 Changed 5 years ago by
Replying to jhpalmieri:
(That's what I did, because then I didn't have to prepare a new tar file and change the checksums. It seems cleaner to change it in the actual tar file, though.)
On the other hand, changing it in the tar file makes it more difficult to maintain the spkg  as changing the permission could be easily forgotten.
comment:347 followup: 348 Changed 5 years ago by
Replying to SimonKing:
On the other hand, changing it in the tar file makes it more difficult to maintain the spkg  as changing the permission could be easily forgotten.
Actually, I'm guessing that you did something wrong in creating this tar file, or that the automake
on your system is subtly broken. Normally, installsh
is created executable.
comment:348 followup: 349 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing: Actually, I'm guessing that you did something wrong in creating this tar file, or that the
automake
on your system is subtly broken. Normally,installsh
is created executable.
When I do make dist
for the autotoolized modular_resolution1.0
subpackage, then I get a .gz
and a .bz2
tarball. For these, I get
$ tar xpf modular_resolution1.0.tar.gz $ ls l modular_resolution1.0/installsh rwrr 1 king king 13997 Jan 7 2016 modular_resolution1.0/installsh $ rm r modular_resolution1.0 $ tar xpf modular_resolution1.0.tar.bz2 $ ls l modular_resolution1.0/installsh rwrr 1 king king 13997 Jan 7 2016 modular_resolution1.0/installsh
ltmain.sh
isn't executable either. And I think tar xpf
should preserve permissions (even tar xf
should, right?).
Moreover, when I do make distdir
instead, then even in the uncompressed source tree the permissions are wrong. So, I suppose the error is in my autotools.
I tried some duckduckgo search with autotools install.sh wrong permissions
, which pointed me to
https://github.com/healpy/healpy/issues/154. There, the first comments seem to indicate that inspite of install.sh not being executable, installation works on linux but not on OS X. However, I am not sure if that helps here. BTW, a search where I replace install.sh
with installsh
revealed even less.
Is there really no way around manually editing the permissions? Perhaps letting spkginstall
do the job is best, after all.
comment:349 followup: 350 Changed 5 years ago by
Replying to SimonKing:
When I do
make dist
I'm guessing that the problem is not with make dist
, that just packages things up. What matters is the permission of installsh
in the sources, before you even run make dist
. That installsh
is normally created by automake
.
comment:350 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
When I do
make dist
I'm guessing that the problem is not with
make dist
, that just packages things up. What matters is the permission ofinstallsh
in the sources, before you even runmake dist
. Thatinstallsh
is normally created byautomake
.
The permissions are wrong even in the sources.
So, I guess the maintenance problem can be solved by changing the permissions in the sources. For all *sh files, right?
comment:351 followup: 352 Changed 5 years ago by
The following works for me:
cd csources autoreconf fiv
comment:352 Changed 5 years ago by
Replying to jdemeyer:
The following works for me:
cd csources autoreconf fiv
Great, it does work! Thank you. By the way, it also slightly changed the INSTALL file.
comment:353 Changed 5 years ago by
It's generally a good idea to run autoreconf fiv
before packaging a release as it ensures that all autotoolsgenerated files are uptodate. This includes also config.guess
which has regularly been an issue in the past (see #19714 for example).
comment:354 Changed 5 years ago by
Commit:  138814b1c5fac5c150379e778bd9876d376860c6 → 441ee4250588968bd122c8ff54989e91c75344f8 

Branch pushed to git repo; I updated commit sha1. New commits:
29a5fda  p_group_cohomology3.0: Let test suite include long tests

c2541fb  p_group_cohomology3.0: Add SAGERUNTIME to the dependencies

1103493  Use classify_elements more efficiently

e41b514  Merge branch 't/25079/use__mul_long_matrix_int' into p_group_cohomology3.0

441ee42  p_group_cohomology3.0: Change upstream tarball (permissions, doctest changes)

comment:355 Changed 5 years ago by
I have updated the upstream tarball and added changes to Sage, as discussed above.
However, there is a surprising problem: sage f c p_group_cohomology
now hangs in the tests, after compiling the spkg. top
shows that Sage works for some time, but then it simply shows no activity. I tried three times with no success.
What could be wrong??
comment:356 followup: 357 Changed 5 years ago by
Strangely, the tests eventually are executed. But certainly Sage shouldn't be sleeping during tests.
comment:357 Changed 5 years ago by
Replying to SimonKing:
But certainly Sage shouldn't be sleeping during tests.
It would help to run the tests with verbose
so you can see what is going on.
comment:358 Changed 5 years ago by
Also, when I interrupt the tests with Ctrlc, I don't get a bash. So, somehow Sage hangs even when it is interrupted. I will now retry with verbose tests.
comment:359 followup: 363 Changed 5 years ago by
It hangs for a long time at lines such as
H1 = CohomologyRing(8,3)
Perhaps the program is trying to access the internet. I am now in my office, where I often have problems with internet connection. At home, internet works fine, perhaps that is why the tests did work there.
However, I thought that I had disallowed internet access during tests. In any case, if an internet connection cannot be established, the program should compute from scratch.
Hm. On second thought: CohomologyRing(8,3)
is part of the database that comes with the package. Thus, it shouldn't even attempt to use internet! Is perhaps my hard disc drive broken?
comment:360 Changed 5 years ago by
I just tested in a new Sage session: Yes, H1 = CohomologyRing(8,3)
does what I expected. It just takes the ring out of the data base and is done in a fraction of a second.
comment:361 Changed 5 years ago by
During the tests, basically each cohomology ring definition hangs for several minutes before it suddenly works.
Can some of you please run the tests, too?
comment:362 Changed 5 years ago by
Dependencies:  #25079 → #25079, #25106 

[p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/src/bin/sageruntests", line 127, in <module> [p_group_cohomology3.0] err = DC.run() [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 1172, in run [p_group_cohomology3.0] self.run_doctests() [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/control.py", line 895, in run_doctests [p_group_cohomology3.0] self.dispatcher = DocTestDispatcher(self) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 1485, in __init__ [p_group_cohomology3.0] init_sage() [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 116, in init_sage [p_group_cohomology3.0] import matplotlib.font_manager [p_group_cohomology3.0] ImportError: No module named matplotlib.font_manager
I created #25106 for this.
comment:363 followups: 364 365 Changed 5 years ago by
Replying to SimonKing:
Thus, it shouldn't even attempt to use internet!
Perhaps this part of the package/installation is broken?
comment:364 followups: 369 477 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Thus, it shouldn't even attempt to use internet!
Perhaps this part of the package/installation is broken?
Some tests expose logging. I guess in these tests it would be visible whether or not the package attempts to access internet.
ANd has matplotlib.font_manager (previous comment) really something to do with this ticket?? If not, please remove the additional dependency.
comment:365 followup: 367 Changed 5 years ago by
comment:367 followup: 372 Changed 5 years ago by
Replying to SimonKing:
I don't see what change could cause this. There haven't been any changes to the code in the last few commits.
Maybe the bug exists for a long time. Why do you specifically say "in the last few commits"?
comment:368 followup: 378 Changed 5 years ago by
Your package is trying to read database files that do not exist. For example pGroupCohomology/4gp2/H4gp2.sobj
but there is no 4gp2
directory.
comment:369 followup: 382 Changed 5 years ago by
Replying to SimonKing:
ANd has matplotlib.font_manager (previous comment) really something to do with this ticket??
No, but it does cause doctest failures. So it must be a dependency.
comment:370 followup: 373 Changed 5 years ago by
Is it intentional that the tarball pGroupCohomology/8gp3.tar.gz
is installed? That looks like a bug to me.
comment:371 followup: 376 Changed 5 years ago by
There is also this doctest failure:
[p_group_cohomology3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 4513, in pGroupCohomology.cohomology.COHO.__getattr__ [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] import sagenb.misc.support as s [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 551, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 961, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology.cohomology.COHO.__getattr__[9]>", line 1, in <module> [p_group_cohomology3.0] import sagenb.misc.support as s [p_group_cohomology3.0] ImportError: No module named sagenb.misc.support
Why do you need sagenb
?
comment:372 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
I don't see what change could cause this. There haven't been any changes to the code in the last few commits.
Maybe the bug exists for a long time. Why do you specifically say "in the last few commits"?
Because there have been changes in that aspect in the past. Feels like a year ago or so. But certainly that has been a long time before the version that passed tests easily (which was 4 days ago in commit:305).
comment:373 followup: 375 Changed 5 years ago by
Replying to jdemeyer:
Is it intentional that the tarball
pGroupCohomology/8gp3.tar.gz
is installed? That looks like a bug to me.
The bug is probably that 8gp3.tar.gz
is in group_cohomology_database.tar
.
comment:374 Changed 5 years ago by
Do you still have the previous version which didn't have the "internet connection" problem?
comment:375 followup: 401 Changed 5 years ago by
When I fixed the internet problem (I DID fix it!), I made sure that all tests pass, regardless whether internet connection is available or not. Even the tests that were explicitly about accessing the database in the web has been modified so that only local file access is needed.
Replying to jhpalmieri:
Replying to jdemeyer:
Is it intentional that the tarball
pGroupCohomology/8gp3.tar.gz
is installed? That looks like a bug to me.The bug is probably that
8gp3.tar.gz
is ingroup_cohomology_database.tar
.
If I recall correctly, 8gp3.tar.gz
was specifically added in order to be able to simulate a web database by local file access in doctests. So, rather not a bug.
comment:376 followup: 379 Changed 5 years ago by
Replying to jdemeyer:
There is also this doctest failure:
[p_group_cohomology3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 4513, in pGroupCohomology.cohomology.COHO.__getattr__ [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] import sagenb.misc.support as s [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 551, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/jdemeyer/sagetest/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 961, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology.cohomology.COHO.__getattr__[9]>", line 1, in <module> [p_group_cohomology3.0] import sagenb.misc.support as s [p_group_cohomology3.0] ImportError: No module named sagenb.misc.supportWhy do you need
sagenb
?
See #24998 for a fix for this (a replacement for the completions
method from sagenb.misc.support
).
comment:377 Changed 5 years ago by
Dependencies:  #25079, #25106 → #25079, #25106, #24998 

comment:378 followup: 381 Changed 5 years ago by
Dependencies:  #25079, #25106, #24998 → #25079, #25106 

Replying to jdemeyer:
Your package is trying to read database files that do not exist. For example
pGroupCohomology/4gp2/H4gp2.sobj
but there is no4gp2
directory.
Unless requested otherwise: The program first looks into the current private database (in DOT_SAGE). If the file doesn't exist then the program looks into the current public database (in SAGE_SHARE) and creates links from the private database to relevant files of the public database (and if the user does computations with the ring, then new files are added only to the private database, not to the public database, since the user may not have write permission there). And if the program doesn't find it in the public database either, it tries to access a repository in the internet. It would download a .tar.gzfile, unpack it in the private database, and load the relevant data. If that fails (and it is supposed to fail very quickly, since Sage is supposed to block internet access during tests  it certainly did in the past!) then the ring is computed from scratch, with files residing in the private database.
Hence, the missing file should simply be created. It shouldn't give rise to an error.
comment:379 followup: 380 Changed 5 years ago by
Replying to jhpalmieri:
Why do you need
sagenb
?
To simulate tab completion.
See #24998 for a fix for this (a replacement for the
completions
method fromsagenb.misc.support
).
Has there been a deprecation warning at some point?
comment:380 Changed 5 years ago by
Replying to SimonKing:
Has there been a deprecation warning at some point?
A deprecation for what? Sagenb is sortof deprecated but still supported.
comment:381 followups: 383 395 Changed 5 years ago by
comment:382 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
ANd has matplotlib.font_manager (previous comment) really something to do with this ticket??
No, but it does cause doctest failures. So it must be a dependency.
You mean it causes doctest failures in the cohomology package? Why should that possibly happen? I mean, there is plotting in the barcode.py module, but I don't think it uses matplotlib (unless matplotlib is the default in Sage when calling the plot()
function).
But perhaps you see my frustration:
 3 years ago, after comment:52, I fixed the problem with internet access. At that point, Sage decided to generally block internet access during tests. So, I fixed it, now it comes up again.
 18 months ago (comment:149) I got all tests to pass, but I had some questions, so, I left it as "needs_info".
 Later, changes broke even installation of the package. I fixed it.
 4 days ago, all tests passed. And all of the sudden everything became foobar again, since all of a sudden doctests that easily pass interactively won't reasonably work during the test suite, and since some module from the Sage source tree got relocated.
comment:383 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
it is supposed to fail very quickly
It doesn't. That's your problem.
But what has changed in the last 4 days that could explain it? Sage blocks internet access during tests since at least 3 years, and I coped with it 3 years ago. Is it perhaps the case that Sage can block using Wifi but cannot properly block ethernet? Because that's the only difference I can see: At home, I have Wifi, in my office I have ethernet.
comment:384 Changed 5 years ago by
BTW, what I didn't understand: Do you currently see the same problem than I (namely: hanging tests)? Or is it just on my machine?
I'll repeat the tests at home, and IMHO it would be fine to say that it is required to run the tests without connecting the computer to the ethernet of FriedrichSchillerUniversität Jena.
comment:385 Changed 5 years ago by
PS: Still it is "hooray".
[p_group_cohomology3.0]  [p_group_cohomology3.0] All tests passed! [p_group_cohomology3.0]  [p_group_cohomology3.0] Total time for all tests: 12883.2 seconds [p_group_cohomology3.0] cpu time: 641.8 seconds [p_group_cohomology3.0] cumulative wall time: 22076.6 seconds
The time to run the tests is of course a disaster, but still all tests pass. In particular the tests pass that check against a specific log. The fact that the log is as expected demonstrates that internet access was off, as it should be. So, perhaps it is just Sage not realising that the internet was off.
comment:386 followups: 387 389 Changed 5 years ago by
Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage. Could even be file access.
comment:387 followup: 388 Changed 5 years ago by
Replying to SimonKing:
Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage. Could even be file access.
in your office, do you perhaps have ~/.sage/ on an NFS volume? As e.g. GAP workspaces are kept there, it could be a performance killer.
comment:388 Changed 5 years ago by
Replying to dimpase:
Replying to SimonKing:
Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage. Could even be file access.
in your office, do you perhaps have ~/.sage/ on an NFS volume? As e.g. GAP workspaces are kept there, it could be a performance killer.
No, it is just my usual laptop plugged to ethernet cable.
comment:389 followup: 392 Changed 5 years ago by
comment:390 followup: 393 Changed 5 years ago by
Here is a problem. First I deleted (from my .sage/pGroupCohomology/db
directory) the directories 8gp1
, 8gp2
, 8gp3
, 8gp4
. Then I started Sage and did this:
sage: from pGroupCohomology import CohomologyRing sage: H = CohomologyRing(8,1) sage: H = CohomologyRing(8,2) sage: H = CohomologyRing(8,3) sage: H = CohomologyRing(8,4)
This recreated those directories. I quit Sage and started again:
sage: from pGroupCohomology import CohomologyRing sage: H = CohomologyRing(8,1) BOOM sage: H = CohomologyRing(8,2) BOOM sage: H = CohomologyRing(8,3) sage: H = CohomologyRing(8,4)
The BOOM
means that the first two commands produce errors, but the last two seem to work fine. The error looks like:
IOError Traceback (most recent call last) <ipythoninput2887a1ff3a72e> in <module>() > 1 H = CohomologyRing(Integer(8),Integer(1)) /Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds) 1490 if q.is_prime_power(): 1491 CacheKey = (KEY, os.path.join(root_user_db,GStem,'dat','State')) > 1492 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1493 return OUT 1494 /Users/jpalmier/Desktop/Sage_stuff/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds) 1118 if coho_options.has_key('@use_this_root@'): 1119 del coho_options['@use_this_root@'] > 1120 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_user_db,file_name), msg)) 1121 ## 3. Link with public db and load from there 1122 elif os.access(os.path.join(root_public_db,file_name), os.R_OK) and not from_scratch: IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/8gp1/H8gp1.sobj are not readable: Singular error: ? ...parse error ? error occurred in or before STDIN line 18: `ideal COHO1I;` ? wrong type declaration. type 'help ideal;'
It's weird: I just quit and started again, and CohomologyRing(8,1)
gave me this error the first time, but worked (in the same Sage session) after that, while CohomologyRing(8,2)
gave an error every time I used it.
comment:391 Changed 5 years ago by
If my error is not reproducible for anyone else, it may be due to breakage with the most recent Xcode on OS X: that has at least caused some problems which might be related to Singular. People should test it on other platforms, though.
comment:392 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Note that the cpu time is only a bit more than 10 minutes. So, it must be something outside of Sage.
It's the internet connection, see 381.
OK. Could one tell the user that the test suite should be run without being connected to internet (I mean: REALLY plugging WLAN and ethernet off)?
comment:393 Changed 5 years ago by
Work issues:  → Allow (un)pickling of empty rings 

Dear John,
Replying to jhpalmieri:
Here is a problem. First I deleted (from my
.sage/pGroupCohomology/db
directory) the directories8gp1
,8gp2
,8gp3
,8gp4
. Then I started Sage and did this:sage: from pGroupCohomology import CohomologyRing sage: H = CohomologyRing(8,1) sage: H = CohomologyRing(8,2) sage: H = CohomologyRing(8,3) sage: H = CohomologyRing(8,4)This recreated those directories. I quit Sage and started again:
sage: from pGroupCohomology import CohomologyRing sage: H = CohomologyRing(8,1) BOOM
}}}
Nice catch!
Singular error: ? ...parse error ? error occurred in or before STDIN line 18: `ideal COHO1I;` ? wrong type declaration. type 'help ideal;'It's weird: I just quit and started again, and
CohomologyRing(8,1)
gave me this error the first time, but worked (in the same Sage session) after that, whileCohomologyRing(8,2)
gave an error every time I used it.
I tried twice, and I got the same error twice.
Here is how to work around the problem:
 Delete the folders
 Start Sage and create the rings
 COMPUTE the rings (with H.make())
 quit and restart sage
 create the rings again, and they are correctly loaded.
I also tried to just save the empty rings instead of computing the ring structure, but this didn't help.
So, I can reproduce it. I guess the reason is that the pickle of a freshly created (i.e., empty) ring structure constitutes a corner case in which certain autogenerated definitions in Singular become syntax errors. I'll investigate what is happening here. Certainly it SHOULD work!
comment:394 followup: 396 Changed 5 years ago by
Would it not be possible to put all files needed for the testsuite in the package? That way, you avoid the internet connection.
comment:395 followup: 398 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
it is supposed to fail very quickly
It doesn't. That's your problem.
I still don't understand what is your exact statement. So, let me ask again.
 Do you observe similar slowness when you run the test on your machine?
 Or do you just observe that internet is accessed, but without slowness?
By the way, I do have reason to believe that there is something weird going on in the combination of my laptop with my office: If my laptop is connected with the ethernet in my office and I try to shut the laptop down, then the shutdown of ubuntu sometimes takes ages. So, before shutting down my laptop I have to disconnect from ethernet, wait a few seconds till the network manager confirms that it is disconnected, and then I can start to shut down.
At the moment I am trying to run the tests at home.
comment:396 Changed 5 years ago by
Replying to jdemeyer:
Would it not be possible to put all files needed for the testsuite in the package? That way, you avoid the internet connection.
Theoretically I could put more rings into the "public database" that is shipped with the spkg.
However, I have reason to believe that this will not solve the problem.
Namely, in my office, I have seen that even H = CohomologyRing(8,3)
hangs. Now the point is that this particular ring is contained in the "public database" (this is something else than the tarball used to simulate web access in some tests). Hence, H = CohomologyRing(8,3)
is supposed to NEVER access internet,vwith only one exception: If CohomologyRing.set_public_db()
was used to redefine the location of the public database.
So, if this hangs then it isn't the internet. And if this hangs then putting more stuff into the public database will not solve it.
By the way: While I was writing these lines, the spkg installed and passed all its tests in a total of less than 20 minutes walltime.
comment:397 followup: 399 Changed 5 years ago by
Let me review how the package behaves in different internet settings. It is all on the same ubuntu laptop.
 With no internet connection whatsoever:
 All tests passed!  Total time for all tests: 729.6 seconds cpu time: 1127.5 seconds cumulative wall time: 2119.5 seconds
 With WLan at my home:
 All tests passed!  Total time for all tests: 624.6 seconds cpu time: 850.7 seconds cumulative wall time: 1764.2 seconds
 With ethernet in my office (note that in that setting interactive use of the package does not show slowness!)
 All tests passed!  Total time for all tests: 12883.2 seconds cpu time: 641.8 seconds cumulative wall time: 22076.6 seconds
Conclusions
 The tests pass under all circumstances. Therefore, it is not the case that the tests "depend on internet" or "are broken by internet".
 The comparison "no internet" vs. "WLan" shows that the tests work faster if data can be found in internet. I don't see that as a problem. What matters is that the eventual outcome is the same.
 Very likely the package is not to blame for the slowness of the tests with ethernet connection. Evidence: In that setting I have all sorts of trouble (slowness in shutting down my computer, and also thunderbird sometimes doesn't properly work in my office).
 Therefore, I do not think that the internet issue needs to be addressed.
TODO
 I need to check that preexisting user data have no influence on the doctests. I.e., I will remove
~/.sage/pGroupCohomology
andSAGE_SHARE/pGroupCohomology
before reinstalling and retesting the package. Both with and without internet connection.  The bug found by John ("creation of empty rings breaks pickling") needs to be fixed. But other than that, I still think the package is fine.
comment:398 followup: 403 Changed 5 years ago by
Replying to SimonKing:
 Do you observe similar slowness when you run the test on your machine?
Yes:
[p_group_cohomology3.0] Total time for all tests: 11383.1 seconds [p_group_cohomology3.0] cpu time: 1155.4 seconds [p_group_cohomology3.0] cumulative wall time: 23145.5 seconds
 Or do you just observe that internet is accessed, but without slowness?
It's trying to access the internet but failing since (as you correctly noted) internet is blocked during package installation.
comment:399 Changed 5 years ago by
Replying to SimonKing:
 Therefore, I do not think that the internet issue needs to be addressed.
I strongly disagree. The testsuite spends 3 hours doing absolutely nothing (waiting for an internet connection that will never succeed).
comment:400 followup: 405 Changed 5 years ago by
Note that there are several easy ways to overcome the internet problem:
 Never access the internet for easy groups: just do the computation locally.
 Simply add more data to the database such that internet access won't be needed during the tests.
 Have some option (environment variable or whatever) to disable internet lookup and use that during testing.
comment:401 followup: 404 Changed 5 years ago by
Replying to SimonKing:
If I recall correctly,
8gp3.tar.gz
was specifically added in order to be able to simulate a web database by local file access in doctests. So, rather not a bug.
If that file is only needed for the testsuite, it shouldn't be installed. You could ship it in the sources in some testdata
subdirectory or whatever.
comment:402 Changed 5 years ago by
Replying to SimonKing:
The binding=True option led to 20% performance loss, that's why I wouldn't use it even if the bug with default options was fixed.
Offtopic, but I'm working on this: https://www.python.org/dev/peps/pep0575/
comment:403 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
 Do you observe similar slowness when you run the test on your machine?
Yes:
Thank you. You hadn't been clear enough (for me at least) about that before.
 Or do you just observe that internet is accessed, but without slowness?
It's trying to access the internet but failing since (as you correctly noted) internet is blocked during package installation.
Aha! Actually I incorrectly noted that internet was blocked during doctests. So, it is about package installation, not about doctests.
comment:404 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
If I recall correctly,
8gp3.tar.gz
was specifically added in order to be able to simulate a web database by local file access in doctests. So, rather not a bug.If that file is only needed for the testsuite, it shouldn't be installed. You could ship it in the sources in some
testdata
subdirectory or whatever.
Good idea!
comment:405 followup: 413 Changed 5 years ago by
Replying to jdemeyer:
Note that there are several easy ways to overcome the internet problem:
 Never access the internet for easy groups: just do the computation locally.
I have to think about it. "Small" probably means "up to order 125". In orders 1 to 127, only the groups of order 64 take a serious amount of time to compute, but order 64 is provided by the public database, so they are available locally anyway.
 Have some option (environment variable or whatever) to disable internet lookup and use that during testing.
Or:
 Have a function that quickly determines whether
sage t
is running. Use internet if that function returns "False", otherwise do not use internet.
Would it be difficult to check for sage t
?
comment:406 Changed 5 years ago by
Work issues:  Allow (un)pickling of empty rings → Internet; (un)pickling of empty rings; test altering private database 

All tests pass when one first removes all existing cohomology data.
However, I noticed that after running the test suite there are 274 rings in the user's private database DOT_SAGE/pGroupCohomology/db/
. That's another issue to be fixed.
comment:407 Changed 5 years ago by
Work issues:  Internet; (un)pickling of empty rings; test altering private database → Internet; empty rings; tests alter private database; do not install test data files 

comment:408 Changed 5 years ago by
As I stated on sagedevel, I consider adding a function CohomologyRing.doctest_setup()
that I would insert at the beginning of each doctest. It would
 disallow internet access
 setup a temporary directory for cohomology computations (I am doing this in most tests anyway)
 clear the cache (I am doing that in some tests already to prevent sideeffects of tests)
In that way, two work issues ("internet" and "tests alter private database") would be resolved.
comment:409 Changed 5 years ago by
Work issues:  Internet; empty rings; tests alter private database; do not install test data files → Internet; empty rings; tests alter private database; do not install test data files; use #24998 

comment:410 Changed 5 years ago by
Dependencies:  #25079, #25106 → #25079, #25106, #24998 

comment:411 Changed 5 years ago by
Summary:  Upgrade of group cohomology spkg → Upgrade of p_group_cohomology spkg 

comment:412 followups: 414 415 Changed 5 years ago by
Can PLEASE some expert for trac finally make it so that the browser isn't insanely jumping its focus while typing a comment?!?
Lectures are starting tomorrow, hence I currently only have little time to work on the spkg.
But at least I'd like to recall some design choices and then outline the next steps I'm planning.
So far I distinguished user_db
, public_db
and web_db
, where db
in all cases means "database". However, I think I should distinguish the roles that all these databases are playing:
 A
user_db
is located inDOT_SAGE
and is where new cohomology data are by default written to. It is formed by a bunch of folders, one folder for one ring. ALL computations are done there, even if the computation starts with data obtained from apublic_db
.  A
public_db
is a source of cohomology data located inSAGE_SHARE
; it is formed by folders located in a common root directory, one folder for one cohomology ring. Loading a ring from apublic_db
means to create links fromuser_db
topublic_db
.  A
web_db
is some url providing cohomology rings in the form of.tar.gz
files, one file for one ring, that when unpacked results in a folder suitable foruser_db
.
Now I plan to do a little relabelling:
 A more suitable label for
user_db
would beworkspace
. In most cases a single cohomology ring relies on computing various other cohomology rings. It makes sense to have a single workspace for all rings that the user is doing computations with. So, there should be a global option defining the location of theworkspace
(such a global option exists already).  In contrast, both
user_db
andweb_db
are local resp. web based data sources. It should be totally fine to have several data sources simultaneously. So, there should be global optionslocal_sources
andweb_sources
providing a tuple of paths resp. URLs (the current global options only allow a single location). Or, with the unifications outlined below, it would be enough to have a single global optionsources
providing both kinds of sources.  It should be possible to declare the optional data sources in the cohomology ring constructor, so that it will be used for the ringtobeconstructed and the other rings that it depends on.
The relabelled global options would be used as follows:
 The cohomology ring constructor first tries to find a ring in
workspace
. If it isn't there, it would try to load it (in order) from the availablelocal_sources
. If it isn't in any of them, it would try to download it (in order) from the availableweb_sources
.  I mentioned a function
CohomologyRing.doctest_setup()
. That function would setworkspace=tmp_dir()
andweb_sources=()
, thus solving the "internet in doctests" issue.
All of the above could be done soonish (modulo the time that I need to prepare lectures). In particular, once the above is done, it would probably be in a sufficient shape for review.
After that (or when the review takes too long), I plan to rework the internal logic by which the file locations are managed. It currently works, but I find it messy.
And now I have a question for you: I have mentioned above that local databases provide one folder per ring, whereas web databases provide the .tar.gz
versions of such folders. It would be tempting to unify the format, so that I can unify the process of (down)loading a cohomology ring. So, what shall I do?
 Change the web databases so that they provide folders? Downside would be that downloading a ring would need more bandwith.
 Change the local databases (not the workspace, though!) so that they provide
.tar.gz
balls? Downsides: Loading would take longer, and it wouldn't be possible to preserve storage space on disk by using links from workspace to local database.  Keep the current formats, but still unify the routine that loads a cohomology ring? I.e., if there is a folder in the database, then use it, otherwise resort to the tarball.
 None of the above?
comment:413 followup: 416 Changed 5 years ago by
Replying to SimonKing:
I have to think about it. "Small" probably means "up to order 125". In orders 1 to 127, only the groups of order 64 take a serious amount of time to compute, but order 64 is provided by the public database, so they are available locally anyway.
What kind of time are we talking about to compute one group? If it's less than 100ms, I would compute it locally because opening an internet connection typically takes longer than that. That would not only help doctests, but also actual users.
comment:414 followup: 417 Changed 5 years ago by
Replying to SimonKing:
 It should be possible to declare the optional data sources in the cohomology ring constructor
This sounds strange to me, a single global option makes more sense. Is there any reason why you would want to use different data sources for different rings?
comment:415 followup: 418 Changed 5 years ago by
Replying to SimonKing:
 Change the web databases so that they provide folders? Downside would be that downloading a ring would need more bandwith.
 Change the local databases (not the workspace, though!) so that they provide
.tar.gz
balls? Downsides: Loading would take longer, and it wouldn't be possible to preserve storage space on disk by using links from workspace to local database. Keep the current formats, but still unify the routine that loads a cohomology ring? I.e., if there is a folder in the database, then use it, otherwise resort to the tarball.
 None of the above?
I would go for for "None of the above" and keep the current implementation. Is there any downside to that?
comment:416 Changed 5 years ago by
Replying to jdemeyer:
What kind of time are we talking about to compute one group? If it's less than 100ms, I would compute it locally because opening an internet connection typically takes longer than that. That would not only help doctests, but also actual users.
Hi Jeroen,
it very much depends on the group. The cohomology of group 299 of order 256 couldn't be computed in one month.
But 150 ms for a computation from scratch is probably impossible for any group.
One should also consider what cohomology rings can really be found in the web database, and currently the smallest groups are those of order 128 (except for the dihedral group of order 8, but that's in the local database, too). Also, it is only prime power order.
From that point of view, it really makes no sense to access the web for something that isn't prime power order and is less than order 128.
comment:417 followup: 419 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
 It should be possible to declare the optional data sources in the cohomology ring constructor
This sounds strange to me, a single global option makes more sense. Is there any reason why you would want to use different data sources for different rings?
It was a suggestion that I recently got on sagedevel. And it makes sense because a computation can very well simultaneously involve a group of order 128 (to be found in the web) and order 64 (to be found in the public database) and of order 32 (computed locally). So, they already are different sources used simultaneously. So, the only change would be that theoretically you could use several web sources where currently you can use only one.
It wouldn't make the logic of the ring constructor more complicated.
And last but not least: The location that you'll do computations with is the workspace
 regardless of the original source of the ring. So, the source really doesn't matter for the computations, i.e., there is no need to make the source globally unique.
comment:418 Changed 5 years ago by
Replying to jdemeyer:
I would go for for "None of the above" and keep the current implementation. Is there any downside to that?
A unification of the database formats would mean that I'd only need a single function to load a ring where I currently need two. But that's all. And I pointed out that a unification of formats would have its downsides, too (bandwidth in web resp. time to unpack stuff locally). So, I'll keep that.
To be clear: That point is orthogonal to whether or not to allow multiple web sources simultaneously.
comment:419 followup: 420 Changed 5 years ago by
Replying to SimonKing:
And it makes sense because a computation can very well simultaneously involve a group of order 128 (to be found in the web) and order 64 (to be found in the public database) and of order 32 (computed locally). So, they already are different sources used simultaneously.
But weren't you talking about allowing to specify a tuple of sources? In that case, this example uses a fixed tuple of sources, namely (local database, public database, web database).
comment:420 Changed 5 years ago by
Replying to jdemeyer:
But weren't you talking about allowing to specify a tuple of sources? In that case, this example uses a fixed tuple of sources, namely (local database, public database, web database).
Right, it would be a fixed tuple of sources, and the program would search them in the order given by the tuple.
Hm. On the one hand, it would be easy to allow overriding the global tuple of sources by an optional argument. On the other hand, personally I would probably never use that, except in doctests.
So, better keep it simple, i.e., with just a global option.
comment:421 Changed 5 years ago by
Let's say that there should at least be a global option to specify the sources. I let you decide whether you want an additional argument when creating the ring.
comment:422 Changed 5 years ago by
I am basically done with the changes and will soon post an update of the package. I have yet to run the tests with an ethernet connection in my office (recall that this used to be a problem), but will post the new package before that.
 I decided to keep a single local source (formerly known as "public db"), which on the one hand is due to details in old pickles and which on the other hand is reasonable: I think no sane user will create additional local databases (i.e., in addition to the one in SAGE_SHARE/pGroupCohomolog/).
 What was formerly known as "private db" became "workspace"  I guess that's more descriptive.
 The remote sources are defined as a tuple of URLs. In particular, if that tuple is empty, there will be no web access.
 There is a new function
doctest_setup()
which clears the cache, puts the workspace into a temporary directory and defines "remote sources" as empty tuple.
The above is all about global options. I decided to keep the possibility to define a websource for a single cohomology ring, temporarily overriding the global setting.
 The file
8gp3.tar.gz
, that is used to provide a local version of a web repository, will not be installed. Instead, it is shipped in a foldertest_data
in the package.
 I have fixed the bug on reading data for an "empty" cohomology ring (i.e., no generators, no relations). The problem was that I tried to create an empty ideal in Singular, which failed as there was no basering. I also added a test for that fix.
(MINOR) PROBLEM:
There is the following warning:
[p_group_cohomology3.0] Warning: This package has a badlybehaved setup.py which outputs [p_group_cohomology3.0] more than the package name for 'setup.py name'; using the last
I tried to google for that warning, but couldn't find what's wrong with my setup.py. Can you help me with it?
comment:423 followup: 424 Changed 5 years ago by
That is apparently something sagespecific as part of sagepipinstall
. This has only apparently appeared only on one other ticket #22756, but that doesn't seem to have any useful information about the warning message. Might be worthwhile to look at what the precise output of your setup.py
.
comment:424 followup: 425 Changed 5 years ago by
Replying to tscrim:
That is apparently something sagespecific as part of
sagepipinstall
. This has only apparently appeared only on one other ticket #22756, but that doesn't seem to have any useful information about the warning message. Might be worthwhile to look at what the precise output of yoursetup.py
.
$ python setup.py name Compiling pGroupCohomology/resolution.pyx because it changed. Compiling pGroupCohomology/cochain.pyx because it changed. Compiling pGroupCohomology/cohomology.pyx because it changed. Compiling pGroupCohomology/modular_cohomology.pyx because it changed. Compiling pGroupCohomology/dickson.pyx because it changed. [1/5] Cythonizing pGroupCohomology/cochain.pyx [2/5] Cythonizing pGroupCohomology/cohomology.pyx [3/5] Cythonizing pGroupCohomology/dickson.pyx [4/5] Cythonizing pGroupCohomology/modular_cohomology.pyx [5/5] Cythonizing pGroupCohomology/resolution.pyx pGroupCohomology
So, the problem is that the innocent python setup.py name
isn't just returning the name but results in a compilation. Why is that?
comment:425 followup: 430 Changed 5 years ago by
Replying to SimonKing:
$ python setup.py name Compiling pGroupCohomology/resolution.pyx because it changed. Compiling pGroupCohomology/cochain.pyx because it changed. Compiling pGroupCohomology/cohomology.pyx because it changed. Compiling pGroupCohomology/modular_cohomology.pyx because it changed. Compiling pGroupCohomology/dickson.pyx because it changed. [1/5] Cythonizing pGroupCohomology/cochain.pyx [2/5] Cythonizing pGroupCohomology/cohomology.pyx [3/5] Cythonizing pGroupCohomology/dickson.pyx [4/5] Cythonizing pGroupCohomology/modular_cohomology.pyx [5/5] Cythonizing pGroupCohomology/resolution.pyx pGroupCohomologySo, the problem is that the innocent
python setup.py name
isn't just returning the name but results in a compilation. Why is that?
I have no idea as I have never dealt with these things. Jeroen?
comment:426 Changed 5 years ago by
Commit:  441ee4250588968bd122c8ff54989e91c75344f8 → 6a65dec4648d10bbeed67d876a4f8890e7259b73 

Branch pushed to git repo; I updated commit sha1. New commits:
cdfe056  incorporate one function from sagenb

039799b  adding a doctest

da202c8  simplify the completion method

9afe56f  Merge branch 'u/chapoton/24998' in 8.2.rc1

dc2e6f0  trac 24998 removing ignored argument

1277686  Merge branch 't/24998/24998' into p_group_cohomology3.0

72671ad  Merge branch 'develop' into p_group_cohomology3.0

6a65dec  p_group_cohomology3.0: Fix unpickling the empty cohomology ring; cleaner doctest setup

comment:427 Changed 5 years ago by
Status:  needs_work → needs_review 

I have updated the documentation at http://users.minet.unijena.de/cohomology/documentation/ and the source tarball at http://users.minet.unijena.de/cohomology/p_group_cohomology3.0.tar.xz
The with the latest changes, the doctests pass in a reasonable time:
[p_group_cohomology3.0] Doctesting 12 files using 3 threads. [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/cochain.pxd [p_group_cohomology3.0] [0 tests, 0.00 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/modular_cohomology.pyx [p_group_cohomology3.0] [450 tests, 460.34 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/cochain.pyx [p_group_cohomology3.0] [1459 tests, 525.24 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/factory.py [p_group_cohomology3.0] [216 tests, 80.46 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/cohomology.pyx [p_group_cohomology3.0] [1371 tests, 606.90 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/resolution.pyx [p_group_cohomology3.0] [892 tests, 18.69 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/resolution.pxd [p_group_cohomology3.0] [0 tests, 0.00 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/barcode.py [p_group_cohomology3.0] [143 tests, 23.43 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/dickson.pyx [p_group_cohomology3.0] [28 tests, 2.07 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/resolution_bindings.pxd [p_group_cohomology3.0] [0 tests, 0.00 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/auxiliaries.py [p_group_cohomology3.0] [69 tests, 3.29 s] [p_group_cohomology3.0] sage t long pyxsources/pGroupCohomology/__init__.py [p_group_cohomology3.0] [349 tests, 189.75 s] [p_group_cohomology3.0]  [p_group_cohomology3.0] All tests passed! [p_group_cohomology3.0]  [p_group_cohomology3.0] Total time for all tests: 650.5 seconds [p_group_cohomology3.0] cpu time: 965.3 seconds [p_group_cohomology3.0] cumulative wall time: 1910.2 seconds [p_group_cohomology3.0] [p_group_cohomology3.0] real 10m57.949s [p_group_cohomology3.0] user 16m44.248s [p_group_cohomology3.0] sys 2m30.648s [p_group_cohomology3.0] Successfully installed p_group_cohomology3.0
I have yet to test for the "web access problem", that I only encountered in my office. But I am confident that the remote sources are set to the empty tuple in all tests that might possibly access the internet.
So, for now it is "needs review"!
comment:428 Changed 5 years ago by
Commit:  6a65dec4648d10bbeed67d876a4f8890e7259b73 → 1cf4d6c5a812af021ae26f88f36592481191a383 

Branch pushed to git repo; I updated commit sha1. New commits:
1cf4d6c  p_group_cohomology3.0: Some fixes

comment:429 Changed 5 years ago by
In the latest changes, I fix some typos in the documentation, change some method names to something more descriptive (from_workspace
, from_local_sources
, from_remote_sources
), and fix from_workspace
so that it behaves according to what one would expect, namely: The ring is taken from the workspace if it is there and is computed from scratch if it isn't, but certainly it is not copied resp. downloaded from other sources.
And tests pass, so, it remains "needs review".
comment:430 Changed 5 years ago by
Replying to tscrim:
I have no idea as I have never dealt with these things. Jeroen?
Personally, I would just ignore that warning for now. There is no official proper way to package Cython projects.
comment:431 Changed 5 years ago by
As with earlier versions, if I take a vanilla copy of Sage and do ./sage i c p_group_cohomology
, I get test failures because of
Exception raised: [snip] import sage.libs.meataxe ImportError: No module named meataxe
I don't care about this. Note that when I ran ./sage i c ...
, meataxe was installed and then the Sage library was built, so I don't know why the meataxe extension wasn't built then.
After running ./sage b
, all tests pass, and they do so pretty quickly (11 minutes on a slowish computer).
comment:432 followup: 433 Changed 5 years ago by
That is a general Sage issue: we do not compile optional extensions after /sage i
.
comment:433 followup: 435 Changed 5 years ago by
Replying to tscrim:
That is a general Sage issue: we do not compile optional extensions after
/sage i
.
Making $(SAGERUNTIME)
an orderonly dependency was supposed to address this (see the earlier discussion here).
comment:434 Changed 5 years ago by
Right...hmm...IDK. I don't have my laptop in front of me to check in more detail. I will do so later today.
comment:435 followup: 436 Changed 5 years ago by
Replying to jhpalmieri:
Making
$(SAGERUNTIME)
an orderonly dependency was supposed to address this (see the earlier discussion here).
No, that was to address a different issue.
I don't see a solution for the sage b
problem, but suggestions are welcome...
comment:436 Changed 5 years ago by
Replying to jdemeyer:
Replying to jhpalmieri:
Making
$(SAGERUNTIME)
an orderonly dependency was supposed to address this (see the earlier discussion here).No, that was to address a different issue.
I don't see a solution for the
sage b
problem, but suggestions are welcome...
I have a suggestion for the problem here (but it wouldn't solve the general problem): p_group_cohomology's spkginstall
should check whether sage.libs.meataxe
can be imported. If it can't then either the installation should be aborted with a message advising the user to do sage b
, or spkginstall
should run sage b
automatically.
comment:437 followup: 439 Changed 5 years ago by
The problem is in running the tests, so would it make more sense for p_group_cohomology's spkgcheck
to test whether meataxe can be imported?
comment:438 Changed 5 years ago by
In any case, I'm not sure that this is an obstacle to giving a positive review here.
comment:439 Changed 5 years ago by
Replying to jhpalmieri:
The problem is in running the tests, so would it make more sense for p_group_cohomology's
spkgcheck
to test whether meataxe can be imported?
The problem is that without sage b, even the simplest operations in the package won't work. So, if sage b was forgotten then the package isn't properly installed. That's why I'd be in favour of solving it in spkginstall. Note that not all users will run the tests, and for them a solution involving spkgcheck won't be a solution.
comment:440 followups: 441 442 Changed 5 years ago by
If there are no good solutions, I am willing to give this a positive review. (Although I like the idea of testing to see if sage.libs.meataxe
can be imported at the end of spkginstall
and printing a helpful message if not.)
comment:441 followup: 443 Changed 5 years ago by
Replying to jhpalmieri:
If there are no good solutions, I am willing to give this a positive review. (Although I like the idea of testing to see if
sage.libs.meataxe
can be imported at the end ofspkginstall
and printing a helpful message if not.)
Do I understand correctly that you only needed to do sage b
and did not need to reinstall the package in order to make it work?
In that case, I think a warning should be given at the end of spkginstall and an error schould be raised at the beginning of spkgcheck, if sage.libs.meataxe cannot be imported.
But if a reinstallation of the package is needed after sage b
, then I'd prefer an error being raised at the beginning of spkginstall.
comment:442 Changed 5 years ago by
Replying to jhpalmieri:
printing a helpful message if not
Unfortunately, my experience is that users typically don't read such helpful messages.
comment:443 followup: 444 Changed 5 years ago by
Replying to SimonKing:
Do I understand correctly that you only needed to do
sage b
and did not need to reinstall the package in order to make it work?
Which "the package"?
You need to build the Sage library, that's all.
comment:444 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Do I understand correctly that you only needed to do
sage b
and did not need to reinstall the package in order to make it work?Which "the package"?
p_group_cohomology
You need to build the Sage library, that's all.
So, you install meataxe, forget to do sage b, install p_group_cohomology, then finally do sage b, and all is good?
You said that helpful messages are often not read. In that case, I'd suggest that at the end of p_group_cohomology
's spkginstall it is checked whether sage.libs.meataxe can be imported, and if it can't then sage b is done automatically.
comment:445 Changed 5 years ago by
Commit:  1cf4d6c5a812af021ae26f88f36592481191a383 → 4d13b9e9257f58dafaf475c627f5f6a19aa0342b 

Branch pushed to git repo; I updated commit sha1. New commits:
4d13b9e  p_group_cohomology3.0: Do sage b when meataxe can't be imported

comment:446 followup: 447 Changed 5 years ago by
Is my change to spkginstall acceptable? I cannot test it at the moment, as I am running a computation that I wouldn't like to interrupt (it will fun for a week or so).
PS: I'm doing the test for proper installation of meataxe the beginning of spkginstall.
comment:447 followup: 449 Changed 5 years ago by
Milestone:  sage8.2 → sage8.3 

Status:  needs_review → needs_work 
comment:448 Changed 5 years ago by
Commit:  4d13b9e9257f58dafaf475c627f5f6a19aa0342b → 843545fffbc89961d90890b5131ea46b1a8aaa23 

Branch pushed to git repo; I updated commit sha1. New commits:
843545f  p_group_cohomology3.0: Do not do sage b, but let the user do it

comment:449 Changed 5 years ago by
comment:450 followup: 453 Changed 5 years ago by
Can you please suggest make sagelib
instead of sage b
? The comand sage b
is really a more lowlevel command that you should only use if you know what you are doing. I would never recommend it like that.
comment:451 Changed 5 years ago by
Also the error message MeatAxe cannot be imported
is not really right. It should be the Sage interface to MeatAxe cannot be imported
or something like that.
comment:452 Changed 5 years ago by
Commit:  843545fffbc89961d90890b5131ea46b1a8aaa23 → a33b1b26a3010495099aa454de097bc2c8f3d9b4 

Branch pushed to git repo; I updated commit sha1. New commits:
a33b1b2  p_group_cohomology3.0: Recommend 'make sagelib' instead of 'sage b'

comment:453 followup: 454 Changed 5 years ago by
Replying to jdemeyer:
Can you please suggest
make sagelib
instead ofsage b
? The comandsage b
is really a more lowlevel command that you should only use if you know what you are doing. I would never recommend it like that.
Actually I thought sage b
is the recommended thing to do. Is it really better to do make sagelib
? I was totally unaware of it. Anyway I changed it.
Replying to jdemeyer:
Also the error message
MeatAxe cannot be imported
is not really right. It should bethe Sage interface to MeatAxe cannot be imported
or something like that.
Done. What cannot be imported is sage.libs.meataxe
comment:454 Changed 5 years ago by
Replying to SimonKing:
Actually I thought
sage b
is the recommended thing to do.
In this specific case, sage b
does make sense. But I don't want to recommend it because the various make
targets like make sagelib
or make build
should be what people use by default. The reason is that those do dependency checking while sage b
is more lowlevel command which does not.
comment:455 Changed 5 years ago by
For me personally, the standard command that I use is make build
because it's fast (unlike plain make
which also builds the doc) and it works well. The only time that I run sage b
is indirectly by running sage bt ...
(combining sage b
and sage t
).
comment:456 followup: 457 Changed 5 years ago by
The error message is not properly formatted: you don't want Do `make sagelib` and retry
– the backquotes should just be regular quotes: Do 'make sagelib' and retry
. (In fact I just ran ./sage i p_group_cohomology
twice in a row without make sagelib
first and the second time it first built sagelib, so it succeeded in building p_group_cohomology.)
comment:457 followup: 459 Changed 5 years ago by
Replying to jhpalmieri:
The error message is not properly formatted: you don't want
Do `make sagelib` and retry
– the backquotes should just be regular quotes:
I don't see why it is a problem, but I'll change it in a few minutes.
(In fact I just ran
./sage i p_group_cohomology
twice in a row withoutmake sagelib
first and the second time it first built sagelib, so it succeeded in building p_group_cohomology.)
How came that? Do you say that the second sage i p_group_cohomology
did automatically do "make sagelib"? It shouldn't!
comment:458 Changed 5 years ago by
Commit:  a33b1b26a3010495099aa454de097bc2c8f3d9b4 → a38b9bbec1f0c05dba20fef8520f65af563269d9 

Branch pushed to git repo; I updated commit sha1. New commits:
a38b9bb  p_group_cohomology3.0: Change backtick into quote

comment:459 Changed 5 years ago by
Replying to SimonKing:
Replying to jhpalmieri:
The error message is not properly formatted: you don't want
Do `make sagelib` and retry
– the backquotes should just be regular quotes:I don't see why it is a problem, but I'll change it in a few minutes.
It's a problem because when the shell prints a string, it tries to evaluate whatever is in backquotes: try echo `ls`
compared to echo 'ls'
.
(In fact I just ran
./sage i p_group_cohomology
twice in a row withoutmake sagelib
first and the second time it first built sagelib, so it succeeded in building p_group_cohomology.)How came that? Do you say that the second
sage i p_group_cohomology
did automatically do "make sagelib"? It shouldn't!
This may be a mistake on my part. I'm trying again (is there a quicker way than make distclean
and rebuilding from scratch?).
New commits:
a38b9bb  p_group_cohomology3.0: Change backtick into quote

comment:460 Changed 5 years ago by
Status:  needs_work → needs_review 

Work issues:  Internet; empty rings; tests alter private database; do not install test data files; use #24998 
Done.
Probably the backtick explains why the second sage i p_group_cohomology
has built sagelib: The backticked make sagelib
got evaluated!
comment:462 Changed 5 years ago by
Commit:  a38b9bbec1f0c05dba20fef8520f65af563269d9 → ca212cdaab90cc21cc85d4268fcc4bc21bd4a923 

Branch pushed to git repo; I updated commit sha1. New commits:
ca212cd  p_group_cohomology3.0: Fix a typo in an error message of spkginstall

comment:463 Changed 5 years ago by
Replying to jdemeyer:
Typo:
sage.lib.meataxe
>sage.libs.meataxe
In the error message, you mean. Fixed!
comment:464 followup: 467 Changed 5 years ago by
Something I just realized after reading the recent comments here: you can actually simplify
Do 'make sagelib' and retry installation of p_group_cohomology
to
Retry installation of p_group_cohomology
This will work because the reinstallation of p_group_cohomology
will trigger a rebuild of the Sage library because of the $(SAGERUNTIME)
dependency.
comment:465 followup: 466 Changed 5 years ago by
Status:  needs_review → needs_work 

comment:466 followup: 468 Changed 5 years ago by
Replying to jdemeyer:
Why did you change it to "needs_work"? I thought that I had addressed all remaining points (considering comment:464 as a nonissue).
comment:467 followup: 469 Changed 5 years ago by
Replying to jdemeyer:
Something I just realized after reading the recent comments here: you can actually simplify
Do 'make sagelib' and retry installation of p_group_cohomologyto
Retry installation of p_group_cohomologyThis will work because the reinstallation of
p_group_cohomology
will trigger a rebuild of the Sage library because of the$(SAGERUNTIME)
dependency.
Why is that the case? I.e., what is different in the second installation attempt?
comment:468 Changed 5 years ago by
Replying to SimonKing:
considering comment:464 as a nonissue
It seems strange to tell a user "do A and B" when "do B" is sufficient.
comment:469 Changed 5 years ago by
Replying to SimonKing:
what is different in the second installation attempt?
The difference is that meataxe
is installed before starting the build of p_group_cohomology
.
Maybe I should explain the problem again in full detail: there are 3 relevant packages: meataxe (M), sagelib (S) and p_group_cohomology (P). The fact that S is a special package doesn't really matter much: the build system mostly treats S like any other package. One exception is that it is never considered uptodate: it will always rebuild S if it occurs as a dependency.
Whenever S is built, it will compile the optional module sage.libs.meataxe
provided that M was built before building S. So we need some build of S after the build of M.
There is a dependency of P on both M and S. The problem is that M and S are built in an unspecified order. So the build order could be M, S, P (good) or S, M, P (bad).
If we are in the bad case S, M, P and we build P again, then the full build order will be S, M, P, S, P which works because it contains the good subsequence M, S, P.
comment:470 Changed 5 years ago by
Commit:  ca212cdaab90cc21cc85d4268fcc4bc21bd4a923 → d77ab39269bf5525c1a8cc154b3adb333d0b61e3 

Branch pushed to git repo; I updated commit sha1. New commits:
d77ab39  p_group_cohomology3.0: Simplify build instructions in case of failing import

comment:471 Changed 5 years ago by
Status:  needs_work → needs_review 

Thank you for the explanation. I have changed the error message accordingly.
comment:472 followup: 474 Changed 5 years ago by
I am a bit worried: From a discussion on sagedevel, I got the impression that a recently merged ticket broke several optional packages, including database_gap, which also affects p_group_cohomology. So, what's the status? Is it safe to use the current Sage beta release to work with optional packages?
Also: Is there more than #25079 that prevents completion of the review of this ticket?
comment:473 Changed 5 years ago by
comment:474 Changed 5 years ago by
comment:476 Changed 5 years ago by
comment:477 Changed 5 years ago by
Dependencies:  #25079, #25106, #24998 → #25079, #24998 

comment:478 Changed 5 years ago by
Status:  needs_review → needs_work 

comment:479 Changed 5 years ago by
comment:480 followup: 481 Changed 5 years ago by
comment:481 followups: 484 485 Changed 5 years ago by
Replying to jdemeyer:
Note that #25106 doesn't need to be dependency but then 362 needs to be fixed in a different way.
In 362 you did in fact NOT explain the problem. You just showed that something went wrong, but didn't give a hint how to reproduce it.
And since you didn't tell, I can only guess: You tried to install the p_group_cohomology package before you completed building Sage. But that's not what one is supposed to do in the case of an optional package: First, one has to have a FULLY FUNCTIONAL Sage installation (i.e., with all standard packages built and installed) and only then should one start with installation of optional packages.
And if Sage is fully functional, matplotlib is available (it is standard package) and the doctesting framework is available too.
comment:482 followup: 483 Changed 5 years ago by
Since #25106 has a positive review, what difference does it make now? Can we move on?
comment:483 Changed 5 years ago by
Replying to jhpalmieri:
Since #25106 has a positive review, what difference does it make now? Can we move on?
+1
I'll take care of #25079 soon.
comment:484 Changed 5 years ago by
Dependencies:  #25079, #24998 → #25079, #24998, #25106 

Status:  needs_work → needs_review 
Replying to SimonKing:
You tried to install the p_group_cohomology package before you completed building Sage.
Indeed and there is (despite what you think) nothing wrong with that.
comment:485 followup: 486 Changed 5 years ago by
comment:486 followups: 487 488 Changed 5 years ago by
Replying to jdemeyer:
Replying to SimonKing:
In 362 you did in fact NOT explain the problem.
That is true but I linked to #25106 where I did explain. Maybe I explained badly, but I certainly tried to be helpful (even replying to your comments on that ticket).
Thank you. The missing bit of information for me was indeed that optional packages are supposed to be buildable before Sage is fully functional (as long as the package's explicitly listed dependencies are available, of course).
Anyway, since #25106 has positive review, I suppose we can consider that issue as being fixed, right?
Do I understand correctly that I do not need to post here a merge commit with #25106?
comment:487 followup: 489 Changed 5 years ago by
Replying to SimonKing:
Anyway, since #25106 has positive review, I suppose we can consider that issue as being fixed, right?
Indeed.
Do I understand correctly that I do not need to post here a merge commit with #25106?
Indeed. Nothing needs to change, just keep the dependency.
Apologies if my posts from yesterday were rude, I was a bit frustrated by the sudden removal of that dependency.
comment:488 Changed 5 years ago by
Replying to SimonKing:
The missing bit of information for me was indeed that optional packages are supposed to be buildable before Sage is fully functional (as long as the package's explicitly listed dependencies are available, of course).
You really should think of "standard" packages and "optional" as being exactly the same, except that optional packages are not installed by default. In particular, they follow the same rules about dependencies.
This rule makes sense because it makes it easy to change optional packages to standard (in theory at least).
comment:489 Changed 5 years ago by
Replying to jdemeyer:
Apologies if my posts from yesterday were rude, I was a bit frustrated by the sudden removal of that dependency.
In fact I am sorry because my own posts have been too rude. My frustration: There have been too many occasions where the old "official" package versions or the "inofficial" new releases of the package broke because of changes in Sage that were more or less trivial but nonetheless made it necessary to change the package, with each change involving a lengthy peer review process (this ticket is 3 years old) during which further trivial changes in Sage led to further necessary adaptions of the package. And I thought we would have yet another iteration for it, under circumstances that from my biased perspective are close to "wrong usage".
comment:490 Changed 5 years ago by
So one problem (solved by #25106) is that, although $(SAGERUNTIME)
is a dependency for this package, $(SAGERUNTIME)
does not require matplotlib
, so it is possible that the Sage library will build before matplotlib
. And pre #25106, the doctesting framework required matplotlib
, leading to problems when testing the pgroup cohomology package before completing a toplevel make
command. Do I have all that right?
comment:491 followup: 492 Changed 5 years ago by
comment:492 Changed 4 years ago by
comment:493 Changed 4 years ago by
Dependencies:  #25079, #24998, #25106 → #25079, #24998, #25106, #25511 

comment:494 followup: 495 Changed 4 years ago by
comment:495 followup: 496 Changed 4 years ago by
Replying to jdemeyer:
Simon: do you mind if I rewrite git history on this ticket and its dependencies?
Why would it be needed to rewrite the git history? I thought "dependency of a ticket" means that there is no need to post an explicit merge commit, as long as the branches from the two tickets merge cleanly.
The dependencies are becoming a mess with various tickets depending on each other. I'd like to make the dependency graph linear:
Is that currently not the case? Here, when we omit the alreadymerged tickets #24998 and #25106, we have dependency #25511, which currently depends on #25079, which depends on #25476.
So, it should be fine to state on #25511 that the ONLY dependency is #25079 and to state here that the ONLY dependency is #25511. But I don't see what that has to do with changing git history.
comment:496 followup: 498 Changed 4 years ago by
Replying to SimonKing:
Why would it be needed to rewrite the git history?
Just to make things simpler.
I thought "dependency of a ticket" means that there is no need to post an explicit merge commit, as long as the branches from the two tickets merge cleanly.
Right, it is not strictly required. But then you need to do those merges anyway whenever working on that ticket or a dependency of it.
comment:497 followup: 500 Changed 4 years ago by
I just saw this commit here:

src/sage/matrix/matrix_gfpn_dense.pyx
diff git a/src/sage/matrix/matrix_gfpn_dense.pyx b/src/sage/matrix/matrix_gfpn_dense.pyx index e8490e5..25a4d6c 100644
a b cdef class Matrix_gfpn_dense(Matrix_dense): 390 390 self.Data = MatLoad(filename) 391 391 FfSetField(self.Data.Field) 392 392 B = GF(self.Data.Field, 'z') 393 parent = MatrixSpace(B, self.Data.Nor, self.Data.Noc )393 parent = MatrixSpace(B, self.Data.Nor, self.Data.Noc, implementation=Matrix_gfpn_dense) 394 394 self._is_immutable = False 395 395 self._parent = parent 396 396 self._base_ring = B
I guess that should be moved to #25511.
comment:498 followup: 499 Changed 4 years ago by
Replying to jdemeyer:
Right, it is not strictly required. But then you need to do those merges anyway whenever working on that ticket or a dependency of it.
Then I'd say: Go ahead. I.e., rebase this ticket on top of #25511, making it the only dependency of this ticket. But then please also tell me how to smoothly pull from the git branch of this ticket after a forced push: Is there a "forced pull" command?
comment:499 Changed 4 years ago by
Replying to SimonKing:
But then please also tell me how to smoothly pull from the git branch of this ticket after a forced push: Is there a "forced pull" command?
I typically do as follows:
$ git fetch $ git trac checkout 18514 $ git reset hard COMMIT_HASH_ON_TICKET
where COMMIT_HASH_ON_TICKET is the value of the "Commit" field on Trac.
comment:500 Changed 4 years ago by
comment:501 Changed 4 years ago by
There is also a conflict of sorts with #25490 because both add the MPB
field.
comment:502 Changed 4 years ago by
Branch:  u/SimonKing/upgrade_of_group_cohomology_spkg → u/jdemeyer/upgrade_of_group_cohomology_spkg 

comment:503 Changed 4 years ago by
Commit:  d77ab39269bf5525c1a8cc154b3adb333d0b61e3 → 3959b5049e299282708950770b8741a2a4b9b675 

Dependencies:  #25079, #24998, #25106, #25511 → #25511 
Status:  needs_review → needs_work 
New commits:
4eae803  Making matrices use the new _echelon_in_place method.

e2f0550  Specify that _echelon_in_place shall return the pivots

3c4a06d  Enable _mul_long for matrices

1eaed37  More stuff in the meataxe interface, and a meataxe helper function

8a06c0f  Fix docstring formatting

13da208  Clean up creating Matrix_gfpn_dense matrices

3959b50  Add p_group_cohomology3.0 spkg

comment:504 Changed 4 years ago by
comment:505 followup: 508 Changed 4 years ago by
I now try to rework the package on top of #25511. One question arose: Why does the following import statement in resolution.pyx does not work:
from sage.cpython.string cimport str_to_bytes from sage.cpython.string import FS_ENCODING
?
It is the same import statement as in sage.matrix.matrix_gfpn_dense  so, it SHOULD work, shouldn't it?
EDIT: This is the error I get when compiling it:
[1/1] Cythonizing pGroupCohomology/resolution.pyx Error compiling Cython file:  ... # Missing from cpython.unicode in Cython 0.27.3 char* PyUnicode_AsUTF8(object s) cdef inline str char_to_str(const char* c, encoding=None, errors=None): IF PY_MAJOR_VERSION <= 2: ^  /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/cpython/string.pxd:25:7: Compiletime name 'PY_MAJOR_VERSION' not defined Error compiling Cython file:  ... TypeError: expected bytes, list found """ if not isinstance(b, bytes): raise TypeError(f"expected bytes, {type(b).__name__} found") IF PY_MAJOR_VERSION <= 2: ^  /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/cpython/string.pxd:70:7: Compiletime name 'PY_MAJOR_VERSION' not defined Error compiling Cython file:  ... TypeError: expected str ... list found """ cdef const char* err cdef const char* enc IF PY_MAJOR_VERSION <= 2: ^  /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/cpython/string.pxd:106:7: Compiletime name 'PY_MAJOR_VERSION' not defined Error compiling Cython file:  ... from pGroupCohomology.cochain cimport COCH from libc.string cimport memcpy from cysignals.memory cimport sig_free from cysignals.signals cimport sig_check, sig_on, sig_off from sage.cpython.string cimport str_to_bytes ^  pGroupCohomology/resolution.pyx:58:0: 'sage/cpython/string/str_to_bytes.pxd' not found
comment:506 Changed 4 years ago by
I also get this very strange attribute error (unfortunately without a good traceback):
... pGroupCohomology/cochain.pyx in pGroupCohomology.cochain.COCH._mul_() /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/structure/element.pyx in sage.structure.element.Element.__getattr__ (build/cythonized/sage/structure/element.c:4519)() 491 AttributeError: 'LeftZeroSemigroup_with_category.element_class' object has no attribute 'blah_blah' 492 """ > 493 return self.getattr_from_category(name) 494 495 cdef getattr_from_category(self, name): /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/structure/element.pyx in sage.structure.element.Element.getattr_from_category (build/cythonized/sage/structure/element.c:4628)() 504 else: 505 cls = P._abstract_element_class > 506 return getattr_from_other_class(self, cls, name) 507 508 def __dir__(self): /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/cpython/getattr.pyx in sage.cpython.getattr.getattr_from_other_class (build/cythonized/sage/cpython/getattr.c:2536)() 392 dummy_error_message.cls = type(self) 393 dummy_error_message.name = name > 394 raise AttributeError(dummy_error_message) 395 attribute = <object>attr 396 # Check for a descriptor (__get__ in Python) AttributeError: 'Singular' object has no attribute '_strip_prompt'
But '_strip_prompt' appears nowhere in the cohomology package! So, what the heck is going on here?
comment:507 Changed 4 years ago by
I don't understand why Singular gets involved, but the error in the previous comment occurs when trying to call Matrix_gfpn_dense._new
, which apparently got removed in #25511. So, I now know how to work around. But the compiler error in comment:505 still needs explanation.
comment:508 Changed 4 years ago by
Replying to SimonKing:
It is the same import statement as in sage.matrix.matrix_gfpn_dense  so, it SHOULD work, shouldn't it?
I guess the difference is between compiling in Sage and compiling externally. I certainly do consider it a bug.
comment:510 followups: 511 526 Changed 4 years ago by
Replying to jdemeyer:
I opened #25549 for that.
Thank you! For now, I am able to work around that problem, because at the moment it still works to pass a str to MatLoad, but I guess if #25549 gets fixed soon I should use it.
A related question: Can you point me to an explanation on how to deal with strings in a "Python3approved" way? I still do if isinstance(...,basestring):
, cdef str FOO
, and even return <str>PyBytes_FromString(...)
, but I guess that should change.
EDIT: In fact, return <str>PyBytes_FromString(...)
wasn't I. That line is in sage.cpython.string
.
comment:511 followup: 512 Changed 4 years ago by
Replying to SimonKing:
Can you point me to an explanation on how to deal with strings in a "Python3approved" way?
That's a very general question with no easy answer. It all depends on what you want to do precisely...
comment:512 followups: 513 514 Changed 4 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Can you point me to an explanation on how to deal with strings in a "Python3approved" way?
That's a very general question with no easy answer. It all depends on what you want to do precisely...
Well, my understanding is that in Python 3, bytes
takes the role of what in Python 2 is str
. And a string in Python 3 always comes with an encoding.
In fact, my questions ARE very general and unprecise, as I heard that there ARE differences between Python 2 and 3 regarding strings, but I do not have the faintest idea how they differ and what it means for coding.
 How to test whether a function argument is a string? I recall that
isinstance(FOO,basestring)
is depreciated.  Is there a change when talking with a C library function? I.e., currently a Python str is converted into a
char*
; is there any pitfall in Python 3 (such as "one is supposed to use a proper string with encoding in Python code, but when talking with C one needs to convert tobytes
"?  Example: What does
do that
if type(filename) is not bytes: filename = str_to_bytes(filename, FS_ENCODING, 'surrogateescape')
if not isinstance(filename, basestring): <do some conversion>
doesn't
comment:513 Changed 4 years ago by
First of all, the answers are different between Python and Cython. Since you're mostly dealing with Cython code, I'll give the Cython answer.
There are 2 classes which are unambiguous: bytes
and unicode
. Here, bytes
is a sequence of bytes (corresponding to char *
in C) and unicode
is a sequence of unicode characters. You can also think of unicode
as an abstract string, while bytes
is an encoded string (say, with UTF8 encoding).
There is also the str
class, which is an alias for bytes
on Python 2 but an alias of unicode
on Python 3. But it's best to think of bytes
, str
and unicode
as three distinct classes (that's more or less what Cython does internally)
isinstance(foo, basestring)
matches str
and unicode
(so it matches bytes
on Python 2 but not on Python 3).
For literals, the following rules apply:
b"foo"
>bytes
"foo"
>str
u"foo"
>unicode
Replying to SimonKing:
 How to test whether a function argument is a string?
Depends what you mean with "a string".
 Is there a change when talking with a C library function?
Absolutely, I would say that this is the main reason of existence of sage.cpython.string
. C functions take a char *
which maps to bytes
. But the Python wrappers typically take a str
as argument. So there needs to be a conversion between str
and bytes
(which is a noop in Python 2).
comment:514 Changed 4 years ago by
Replying to SimonKing:
if type(filename) is not bytes: filename = str_to_bytes(filename, FS_ENCODING, 'surrogateescape')
Let me just explain what this does and why it's the right thing to do. For filenames, many Python functions accept both bytes
and str
. So it's natural to allow this too in Sage.
If filename
has type bytes
, then nothing needs to be done because C functions can accept that (Cython does the conversion bytes
> char *
for you).
If not, str_to_bytes
is called which converts either str
or unicode
to bytes
(it raises an error for other input). So that piece of code accepts bytes
, str
and unicode
on all Python versions. The result is that filename
is of type bytes
.
The arguments FS_ENCODING, 'surrogateescape'
are what Python uses by default for converting filenames.
The only thing that slightly puzzles me is why the check is type(filename) is not bytes
instead of not isinstance(filename, bytes)
(i.e. allowing subclasses of bytes
). Maybe custom subclasses of bytes
cannot easily be converted to char *
?
comment:515 Changed 4 years ago by
Branch:  u/jdemeyer/upgrade_of_group_cohomology_spkg → u/SimonKing/upgrade_of_group_cohomology_spkg 

comment:516 followups: 517 518 Changed 4 years ago by
Commit:  3959b5049e299282708950770b8741a2a4b9b675 → 0631029b380acdc82ac195d553b42b5c2bfa42d4 

Status:  needs_work → needs_review 
I have reworked upstream so that the recent additions from #25511 are used. The advantage is that this ticket's branch only adds changes to build/pkgs, but not to the Sage library; these changes are in the only new commit of the branch that I just forcepushed.
Of course, I have updated the .tar.xzfile at the location given in the ticket description.
Result:
[p_group_cohomology3.0]  [p_group_cohomology3.0] All tests passed! [p_group_cohomology3.0]  [p_group_cohomology3.0] Total time for all tests: 541.7 seconds [p_group_cohomology3.0] cpu time: 883.3 seconds [p_group_cohomology3.0] cumulative wall time: 1575.8 seconds [p_group_cohomology3.0] [p_group_cohomology3.0] real 9m4.813s [p_group_cohomology3.0] user 15m25.364s [p_group_cohomology3.0] sys 2m18.384s [p_group_cohomology3.0] Successfully installed p_group_cohomology3.0 [p_group_cohomology3.0] Deleting temporary build directory [p_group_cohomology3.0] /home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.0 [p_group_cohomology3.0] Finished installing p_group_cohomology3.0.spkg make[1]: Leaving directory '/home/king/Sage/git/sage/build/make' real 17m7.539s user 22m55.884s sys 2m23.596s Sage build/upgrade complete!
So, ready for review!
Strangely, only the branch field of this ticket was automatically updated, but not the "commit" field. So, I changed it manually.
comment:517 Changed 4 years ago by
Replying to SimonKing:
Strangely, only the branch field of this ticket was automatically updated, but not the "commit" field.
comment:518 followup: 519 Changed 4 years ago by
Replying to SimonKing:
Of course, I have updated the .tar.xzfile at the location given in the ticket description.
I am having problems downloading the file: I get
Fehler 403 Zugriff auf /cohomology/p_group_cohomology3.0.tar.xz verweigert. Wenn Sie dies für einen Fehler in der Serverkonfiguration halten, so senden Sie bitte eine Nachricht an den Webmaster oder an die Autorin der Seite, über die Sie hierher gelangt sind. Bitte fügen Sie Ihrer Nachricht die folgenden Zeilen an: Wed Jun 13 19:47:02 2018 (75.172.196.95) Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0 Error 403 You don't have permission to access /cohomology/p_group_cohomology3.0.tar.xz on this server. If you feel that this is a configuration error, please contact the administrators or the author of the referring page. Please include the following lines in your message: Wed Jun 13 19:47:02 2018 (75.172.196.95) Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0
comment:519 Changed 4 years ago by
Description:  modified (diff) 

Replying to jhpalmieri:
Replying to SimonKing:
Of course, I have updated the .tar.xzfile at the location given in the ticket description.
I am having problems downloading the file:
Thanks for spotting it. I have informed our administrator.
Temporarily, I have posted it at my homepage (https://users.fmi.unijena.de/~king/p_group_cohomology3.0.tar.xz), but that will not be permanent, as I have a data limit there.
And can PLEASE finally someone fix trac, so that the browser isn't insanely jumping to odd places (for example it was just centering at comment:223 while I tried to write the new comment)
comment:520 Changed 4 years ago by
The checksums didn't work for me, but once I fixed those, all tests in the test suite passed. Is there anything else that needs to be done here?
comment:521 Changed 4 years ago by
Commit:  0631029b380acdc82ac195d553b42b5c2bfa42d4 → a330d39c12ea918970ca3ac9678e02e3441dd760 

Branch pushed to git repo; I updated commit sha1. New commits:
a330d39  p_group_cohomology3.0: Fix checksum

comment:522 Changed 4 years ago by
Sorry, I forgot to push the checksum update. So, here it is...
What *could* be reasonable is to check the package on different platforms. But I guess Jeroen has a more elaborate opinion on what needs to be done to finish the review.
New commits:
a330d39  p_group_cohomology3.0: Fix checksum

comment:523 Changed 4 years ago by
Description:  modified (diff) 

comment:524 Changed 4 years ago by
I prefer to do a final review after all dependencies have been merged. You can remind me when 8.3.beta6 is released.
comment:525 Changed 4 years ago by
Since there are no longer any changes to Sage here, it's just a matter of checking that the package builds and passes its testsuite, ideally on several platforms.
comment:526 Changed 4 years ago by
Replying to SimonKing:
For now, I am able to work around that problem, because at the moment it still works to pass a str to MatLoad
But that won't work on Python 3, right?
comment:527 Changed 4 years ago by
Branch:  u/SimonKing/upgrade_of_group_cohomology_spkg → u/jdemeyer/upgrade_of_group_cohomology_spkg 

comment:528 Changed 4 years ago by
Commit:  a330d39c12ea918970ca3ac9678e02e3441dd760 → 7bf9f76405e664d32ce7c01f43afccc6f4b52a60 

Dependencies:  #25511 
comment:529 followup: 530 Changed 4 years ago by
I tried this on 8.3.beta7
and it was not able to pass its test suite. It seems to be coming from this:
sage: from pGroupCohomology import CohomologyRing sage: H = CohomologyRing(8,1)  TypeError Traceback (most recent call last) <ipythoninput3887a1ff3a72e> in <module>() > 1 H = CohomologyRing(Integer(8),Integer(1)) /home/travis/sagebuild/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds) 1531 if q < 128: 1532 extras['websource'] = False > 1533 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1534 return OUT 1535 /home/travis/sagebuild/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1244 OUT = COHO(gap(KEY[0]), **extras) 1245 else: > 1246 OUT = COHO(KEY[0],KEY[1], **extras) 1247 _gap_init(OUT.group().parent()) 1248 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.G_ALG.r_action() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.makeMTX() /home/travis/sagebuild/local/lib/python2.7/sitepackages/sage/matrix/matrix0.pyx in sage.matrix.matrix0.Matrix.__cinit__ (build/cythonized/sage/matrix/matrix0.c:3925)() 95 [2] 96 """ > 97 def __cinit__(self, parent, *args, **kwds): 98 """ 99 The initialization routine of the ``Matrix`` base class ensures TypeError: __cinit__() takes at least 1 positional argument (0 given)
which was given by a relatively recent change to matrices __cinit__
(I don't remember the exact ticket). Once this is fixed, I am willing to set a positive review.
comment:530 followup: 532 Changed 4 years ago by
Replying to tscrim:
I tried this on
8.3.beta7
and it was not able to pass its test suite. It seems to be coming from this ... relatively recent change to matrices__cinit__
(I don't remember the exact ticket).
Yet again.
Frankly, it happens way too often that the group cohomology package gets broken because of changes in sagelib. So, I believe that the package should eventually be moved to the sage library, too, in order to make it easier to prevent this reoccurring mishap.
comment:531 Changed 4 years ago by
Status:  needs_review → needs_work 

This is no longer allowed:
cdef MTX M = MTX.__new__(MTX)
Since #25505, it is mandatory to construct matrices with parent
as first argument. So you need
cdef MTX M = MTX.__new__(MTX, MatrixSpace(...))
Even better, use the function new_mtx
from src/sage/matrix/matrix_gfpn_dense.pyx
which more or less does the same thing.
comment:532 followup: 533 Changed 4 years ago by
Replying to SimonKing:
Replying to tscrim:
I tried this on
8.3.beta7
and it was not able to pass its test suite. It seems to be coming from this ... relatively recent change to matrices__cinit__
(I don't remember the exact ticket).Yet again.
Frankly, it happens way too often that the group cohomology package gets broken because of changes in sagelib. So, I believe that the package should eventually be moved to the sage library, too, in order to make it easier to prevent this reoccurring mishap.
I would be happy with that. I think it would be easier to maintain in the longrun. Is there a particular reason why this is not being done?
comment:533 followup: 536 Changed 4 years ago by
Replying to tscrim:
... So, I believe that the package should eventually be moved to the sage library, too, in order to make it easier to prevent this reoccurring mishap.
I would be happy with that. I think it would be easier to maintain in the longrun. Is there a particular reason why this is not being done?
It would already be possible to implement it with optional extension modules. But it has been argued at some point that optional extension modules should only be used for standard functionality of Sage that can also be provided by a nonstandard backend.
So, when I talk about "putting it into the Sage library", I mean "putting it into the Sage library as plain extension modules that depend only on standard packages". Then:
 It would be needed to make SharedMeatAxe a standard package. Given that its matrix arithmetic is vastly superior to what vanilla Sage has to offer for matrices over small nonprime fields, this should happen anyway.
 It would be needed to make the Clibrary part of the group cohomology package (aka modular_resolution1.0) a standard package.
 It would be needed to make SmallGroups (via database_gap) standard. This used to be impossible because of licencing, but that's fixed now.
comment:534 Changed 4 years ago by
Why does "git trac checkout 18514" followed by "git trac pull" NOT yield the current branch?
comment:535 Changed 4 years ago by
I guess using new_mtx instead of makeMTX is a good idea, partly because in many cases there will be a template, so that the creation of the new matrix will be even faster.
comment:536 followup: 537 Changed 4 years ago by
comment:537 followup: 538 Changed 4 years ago by
Replying to jdemeyer:
Replying to SimonKing:
But it has been argued at some point that optional extension modules should only be used for standard functionality of Sage that can also be provided by a nonstandard backend.
See 337 and following.
OK. But what I forgot to add: I care about documentation, and if I understand correctly, the documentation of an optional extension module will not be built (indeed, the documentation of matrix_gfpn_dense is not available). And I believe that's a show stopper for any complex package.
In any case, I would somehow prefer to postpone this to version 3.1. My plan for version 3.1 is to make the package independent of SmallGroups, and while I am at it, I can put the package into the Sage library  provided that the documentation problem can be solved.
comment:538 Changed 4 years ago by
Replying to SimonKing:
OK. But what I forgot to add: I care about documentation, and if I understand correctly, the documentation of an optional extension module will not be built (indeed, the documentation of matrix_gfpn_dense is not available). And I believe that's a show stopper for any complex package.
Well, I am guessing you have the manually build and host the documentation yourself currently. So it would not be so different than being included in Sage. So I do not see it as really being that different than being a separate package. The interactive doc is still the same, and by being included, it will also get picked up by, e.g. search_src
.
I am in Zaragoza with Jeroen, and we just a brief conversion about how to get Sphinx and Sage to handle optional extensions. He believes it should be something that Sphinx should be able to do, in which case, we could have the official Sage doc be built with all optional packages. However, this is something that I think will likely not be quick to get done.
In any case, I would somehow prefer to postpone this to version 3.1. My plan for version 3.1 is to make the package independent of SmallGroups, and while I am at it, I can put the package into the Sage library  provided that the documentation problem can be solved.
If you are able to do that and MeatAxe also becomes standard, then the doc issues will be fixed because it will be a usual pyx
file.
comment:539 Changed 4 years ago by
Branch:  u/jdemeyer/upgrade_of_group_cohomology_spkg → u/SimonKing/upgrade_of_group_cohomology_spkg 

comment:540 Changed 4 years ago by
Commit:  7bf9f76405e664d32ce7c01f43afccc6f4b52a60 → e59e1dc79070fe7ae722e30acfd3629372a87496 

Description:  modified (diff) 
Status:  needs_work → needs_review 
Good news: I managed to replace !makeMTX by !new_mtx.
Bad news:
 Strange enough, the "commit" field of the ticket has not been correctly set, I needed to edit it manually.
 THE PACKAGE DOES NOT EVEN START TO BUILD. ARRGHH!!!
I have posted the tar ball in its original location (see ticket description). The attempt to install the package ends in an infinite loop:
... [p_group_cohomology3.0] Installing p_group_cohomology3.0 [p_group_cohomology3.0] Warning: This package has a badlybehaved setup.py which outputs [p_group_cohomology3.0] more than the package name for 'setup.py name'; using the last [p_group_cohomology3.0] line as the package name: pGroupCohomology [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. [p_group_cohomology3.0] Skipping pGroupCohomology as it is not installed. ...
Of course it is not installed! That's why I want to install it, stupid!
Can you explain what is going wrong here?
I noticed that the python setup.py name
results in a compilation of the cython files, which certainly shouldn't happen. How to prevent it?
But if I understand correctly, the installation went past that point and did correctly obtain the package name. So, why does it REPEATEDLY complain that the package to be installed is not installed.
comment:541 followup: 542 Changed 4 years ago by
Note that today I upgraded my pip installation. It is now
$ pip version pip 10.0.1 from /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pip (python 2.7)
Can this be related? I am no totally sure, though, because installing the package directly from the unpacked sources did work.
So, bug in Sage's package builder?
comment:542 followup: 543 Changed 4 years ago by
Replying to SimonKing:
Note that today I upgraded my pip installation. It is now
$ pip version pip 10.0.1 from /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pip (python 2.7)Can this be related?
Absolutely! I can confirm that Sage does not work with pip 10.
comment:543 Changed 4 years ago by
Replying to jdemeyer:
Replying to SimonKing:
Note that today I upgraded my pip installation. It is now
$ pip version pip 10.0.1 from /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pip (python 2.7)Can this be related?
Absolutely! I can confirm that Sage does not work with pip 10.
Ouch.
So, at least two questions arise:
 Why is it possible to install the package from its sources by doing
python setup.py build
followed bypip install . upgrade
? Note that plainly doingpip install .
did not suffice.  How to downgrade pip? Note that systemwide I have pip 8. So, if you tell me how to remove pip from the sage installation, I would do so.
comment:544 followup: 552 Changed 4 years ago by
I tried "pip uninstall pip", which first seemed to work. But now it seems that I have to rebuild Sage from scratch.
comment:545 followups: 546 547 Changed 4 years ago by
After downgrading pip, installing the package works. But alas, there are two independent errors in its test suite:
1.
[p_group_cohomology3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 2457, in pGroupCohomology.cohomology.COHO [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] H4.make() [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 983, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology.cohomology.COHO[91]>", line 1, in <module> [p_group_cohomology3.0] H4.make() [p_group_cohomology3.0] File "pGroupCohomology/cohomology.pyx", line 12099, in pGroupCohomology.cohomology.COHO.make [p_group_cohomology3.0] File "pGroupCohomology/cohomology.pyx", line 11559, in pGroupCohomology.cohomology.COHO.next [p_group_cohomology3.0] File "pGroupCohomology/resolution.pyx", line 2194, in pGroupCohomology.resolution.RESL.makeAutolift [p_group_cohomology3.0] TypeError: Cannot convert sage.matrix.matrix_mod2_dense.Matrix_mod2_dense to sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense
If I understand correctly that's when testing the option to use a different matrix backend for some internal computations. Hence, at some point a conversion is needed, which presumably fails because of internal changes in Sage.
2.
[p_group_cohomology3.0] File "pyxsources/pGroupCohomology/barcode.py", line 465, in pGroupCohomology.barcode.BarCode2d.plot [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] plot(B) # render [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 983, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology.barcode.BarCode2d.plot[5]>", line 1, in <module> [p_group_cohomology3.0] plot(B) # render [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/decorators.py", line 567, in wrapper [p_group_cohomology3.0] return func(*args, **options) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/plot/plot.py", line 1942, in plot [p_group_cohomology3.0] G = funcs.plot(*args, **original_opts) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/barcode.py", line 469, in plot [p_group_cohomology3.0] from sage.plot.plot import circle, line [p_group_cohomology3.0] ImportError: cannot import name circle
If I understand correctly, it means that there is a changed import location, i.e., yet another internal change in Sage that creates trouble.
When I developed the first package version in 2009, Sage was an ideal platform. Frankly, if I would create a similar package today, I wouldn't know what platform to choose.
comment:546 Changed 4 years ago by
Replying to SimonKing:
1.
[p_group_cohomology3.0] File "pyxsources/pGroupCohomology/cohomology.pyx", line 2457, in pGroupCohomology.cohomology.COHO [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] H4.make() [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 983, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology.cohomology.COHO[91]>", line 1, in <module> [p_group_cohomology3.0] H4.make() [p_group_cohomology3.0] File "pGroupCohomology/cohomology.pyx", line 12099, in pGroupCohomology.cohomology.COHO.make [p_group_cohomology3.0] File "pGroupCohomology/cohomology.pyx", line 11559, in pGroupCohomology.cohomology.COHO.next [p_group_cohomology3.0] File "pGroupCohomology/resolution.pyx", line 2194, in pGroupCohomology.resolution.RESL.makeAutolift [p_group_cohomology3.0] TypeError: Cannot convert sage.matrix.matrix_mod2_dense.Matrix_mod2_dense to sage.matrix.matrix_gfpn_dense.Matrix_gfpn_denseIf I understand correctly that's when testing the option to use a different matrix backend for some internal computations. Hence, at some point a conversion is needed, which presumably fails because of internal changes in Sage.
No, I was mistaken: When providing a template for new_mtx
, I accidentally chose a matrix that with the option "nouseMTX" is not a meataxe matrix.
comment:547 Changed 4 years ago by
Replying to SimonKing:
2.
[p_group_cohomology3.0] File "pyxsources/pGroupCohomology/barcode.py", line 465, in pGroupCohomology.barcode.BarCode2d.plot [p_group_cohomology3.0] Failed example: [p_group_cohomology3.0] plot(B) # render [p_group_cohomology3.0] Exception raised: [p_group_cohomology3.0] Traceback (most recent call last): [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 573, in _run [p_group_cohomology3.0] self.compile_and_execute(example, compiler, test.globs) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 983, in compile_and_execute [p_group_cohomology3.0] exec(compiled, globs) [p_group_cohomology3.0] File "<doctest pGroupCohomology.barcode.BarCode2d.plot[5]>", line 1, in <module> [p_group_cohomology3.0] plot(B) # render [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/decorators.py", line 567, in wrapper [p_group_cohomology3.0] return func(*args, **options) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/plot/plot.py", line 1942, in plot [p_group_cohomology3.0] G = funcs.plot(*args, **original_opts) [p_group_cohomology3.0] File "/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/barcode.py", line 469, in plot [p_group_cohomology3.0] from sage.plot.plot import circle, line [p_group_cohomology3.0] ImportError: cannot import name circleIf I understand correctly, it means that there is a changed import location, i.e., yet another internal change in Sage that creates trouble.
Apparently the import location HAS changed. However, I am importing "circle" for absolutely no reason. I only use "line", which still can be imported from sage.plot.plot. However, the official import statement is from sage.plot.line import line
, and I'll use this.
comment:548 Changed 4 years ago by
Status:  needs_review → needs_work 

comment:549 Changed 4 years ago by
Commit:  e59e1dc79070fe7ae722e30acfd3629372a87496 → 0aaeef3f5ce41405e9e1755d62f57046db134db4 

Branch pushed to git repo; I updated commit sha1. New commits:
0aaeef3  p_group_cohomology: Fixing bugs from previous commit

comment:550 Changed 4 years ago by
Status:  needs_work → needs_review 

Hooray...
[p_group_cohomology3.0]  [p_group_cohomology3.0] All tests passed! [p_group_cohomology3.0]  [p_group_cohomology3.0] Total time for all tests: 634.0 seconds [p_group_cohomology3.0] cpu time: 878.8 seconds [p_group_cohomology3.0] cumulative wall time: 1557.3 seconds [p_group_cohomology3.0] [p_group_cohomology3.0] real 10m36.533s [p_group_cohomology3.0] user 15m26.060s [p_group_cohomology3.0] sys 2m23.668s [p_group_cohomology3.0] Successfully installed p_group_cohomology3.0
So, things should now work, with the package that I have posted at the location indicated in the ticket description.
comment:551 followup: 553 Changed 4 years ago by
It now works for me. I am basically ready to set this to a positive review, except I do have a two questions:
 What are you going to do for documentation?
 How will people know about this inside of Sage? In particular, I think you should have some link to the spkg doc from Sage's doc, wherever you decide to host the spkg doc. Also, I think it would be useful to have a file in the
sage/tests
folder that does some of the tests for this package. That way, it would be more likely that when some behavior changes that your spkg is breaking.
comment:552 Changed 4 years ago by
Replying to SimonKing:
I tried "pip uninstall pip", which first seemed to work. But now it seems that I have to rebuild Sage from scratch.
I probably should have said this earlier, but ./sage f pip
should fix your problem.
comment:553 Changed 4 years ago by
Replying to tscrim:
 What are you going to do for documentation?
The documentation is hosted at my home institute.
 How will people know about this inside of Sage?
I guess that's the same for most optional Sage packages. Or maybe there is one crucial difference: Most optional packages are independent projects that can be installed and used in the Sage ecosystem; so, people become aware without Sage. But p_group_cohomology is a package that *uses* Sage and is thus not independent.
However, is a problem here to solve? I know that some experts for modular cohomology of finite groups know p_group_cohomology, but I doubt that people outside that niche will use that package.
In particular, I think you should have some link to the spkg doc from Sage's doc, wherever you decide to host the spkg doc. Also, I think it would be useful to have a file in the
sage/tests
folder that does some of the tests for this package. That way, it would be more likely that when some behavior changes that your spkg is breaking.
This would only be executed on a machine that has the package installed in the first place, right?
But actually I think this might be a good idea: The current test suite is only executed if the package is newly installed and the user doesn't forget to provide the c
option. But if one has a Python file with optional tests somewhere in the sage library, then these tests would be executed when doing "make test" provided that the package is installed. And in that way, problems of my package resulting from a change in sagelib would be more likely to become visible.
OK, I'll try to cook something up...
comment:554 followup: 556 Changed 4 years ago by
Status:  needs_review → needs_work 

There is an undeclared dependency on matplotlib
: the testsuite plots stuff, so you should add matplotlib
as dependency (orderonly, so after 
).
comment:555 Changed 4 years ago by
Commit:  0aaeef3f5ce41405e9e1755d62f57046db134db4 → 45146d82f5a77402aecbd7ba3ebb30e9dbd76dac 

comment:556 Changed 4 years ago by
Status:  needs_work → needs_review 

Replying to jdemeyer:
There is an undeclared dependency on
matplotlib
: the testsuite plots stuff, so you should addmatplotlib
as dependency (orderonly, so after
).
Is it not part of $SAGE_RUNTIME
? In any case, I added it as a dependency.
Also, I added src/sage/tests/modular_group_cohomology.py
. The tests there take 9 seconds (I hope that's OK), and they cover a wide range of functionality of the package. Hence, if some change in the sage library breaks p_group_cohomology
and a user who has installed the package runs make test
, then the problem is very likely to become visible immediately.
I guess it is back to "needs review", right?
comment:557 Changed 4 years ago by
OK, let me test this new version on a few machines. If everything goes well, I agree that positive review can be given.
comment:558 followups: 559 563 Changed 4 years ago by
So on my machine, I get
sage t warnlong src/sage/tests/modular_group_cohomology.py ********************************************************************** File "src/sage/tests/modular_group_cohomology.py", line 46, in sage.tests.modular_group_cohomology Warning, slow doctest: H.essential_ideal() # optional  p_group_cohomology Test ran for 1.39 s ********************************************************************** File "src/sage/tests/modular_group_cohomology.py", line 62, in sage.tests.modular_group_cohomology Warning, slow doctest: H = CohomologyRing(gap(AlternatingGroup(7)), GroupName="A(7)", prime=2, from_scratch=True) # optional  p_group_cohomology Test ran for 1.51 s ********************************************************************** File "src/sage/tests/modular_group_cohomology.py", line 79, in sage.tests.modular_group_cohomology Warning, slow doctest: H = CohomologyRing(gap(AlternatingGroup(6)), GroupName="A(6)", prime=3, from_scratch=True) # optional  p_group_cohomology Test ran for 1.90 s
So while these tests are longer than 1 second on my machine, I think they are not sufficiently long to warrant a # long time
considering the purpose of this file.
comment:559 followup: 560 Changed 4 years ago by
Replying to tscrim:
So on my machine, I get
sage t warnlong src/sage/tests/modular_group_cohomology.py ********************************************************************** File "src/sage/tests/modular_group_cohomology.py", line 46, in sage.tests.modular_group_cohomology Warning, slow doctest: H.essential_ideal() # optional  p_group_cohomology Test ran for 1.39 s ********************************************************************** File "src/sage/tests/modular_group_cohomology.py", line 62, in sage.tests.modular_group_cohomology Warning, slow doctest: H = CohomologyRing(gap(AlternatingGroup(7)), GroupName="A(7)", prime=2, from_scratch=True) # optional  p_group_cohomology Test ran for 1.51 s ********************************************************************** File "src/sage/tests/modular_group_cohomology.py", line 79, in sage.tests.modular_group_cohomology Warning, slow doctest: H = CohomologyRing(gap(AlternatingGroup(6)), GroupName="A(6)", prime=3, from_scratch=True) # optional  p_group_cohomology Test ran for 1.90 s
Would one get these warnings automatically? Or does one need to provide a special option? The complete log for my test run is this:
$ ./sage t src/sage/tests/modular_group_cohomology.py too many failed tests, not using stored timings Running doctests with ID 20180630203328f134a034. Git branch: t/18514/upgrade_of_group_cohomology_spkg Using optional=database_gap,meataxe,mpir,p_group_cohomology,python2,sage Doctesting 1 file. sage t src/sage/tests/modular_group_cohomology.py [16 tests, 9.26 s]  All tests passed!  Total time for all tests: 9.3 seconds cpu time: 7.3 seconds cumulative wall time: 9.3 seconds
It is of course conceivable that some of the 16 tests actually take longer than one second (given that on average they take 0.58 seconds). But has the policy changed? I seem to remember that when I was young, tests were supposed to be "not longer than a few seconds". But just *one* second seems short to me.
comment:560 Changed 4 years ago by
Replying to SimonKing:
Would one get these warnings automatically? Or does one need to provide a special option?
Sorry, I didn't notice that you used sage t warnlong
.
On my machine, I get warnings for four tests:
H.essential_ideal() ascii_art(H.bar_code('LowerCentralSeries')[2]) H = CohomologyRing(gap(AlternatingGroup(7)), GroupName="A(7)", prime=2, from_scratch=True) H = CohomologyRing(gap(AlternatingGroup(6)), GroupName="A(6)", prime=3, from_scratch=True)
I'll try to come up with shorter tests.
comment:561 Changed 4 years ago by
Commit:  45146d82f5a77402aecbd7ba3ebb30e9dbd76dac → d4a612c7ebf08d5bb48479163099671fd880994f 

Branch pushed to git repo; I updated commit sha1. New commits:
d4a612c  p_group_cohomology3.0: Simplify tests/modular_group_cohomology.py

comment:562 Changed 4 years ago by
I have simplified the tests and still think that they cover a wide range of functionality. This time I didn't get a warning (and I tested 10 times):
$ ./sage t warnlong src/sage/tests/modular_group_cohomology.py Running doctests with ID 201807010021354bcb0f8f. Git branch: t/18514/upgrade_of_group_cohomology_spkg Using optional=database_gap,meataxe,mpir,p_group_cohomology,python2,sage Doctesting 1 file. sage t warnlong src/sage/tests/modular_group_cohomology.py [17 tests, 5.15 s]  All tests passed!  Total time for all tests: 5.2 seconds cpu time: 4.0 seconds cumulative wall time: 5.2 seconds
comment:563 followup: 564 Changed 4 years ago by
Replying to tscrim:
So while these tests are longer than 1 second on my machine, I think they are not sufficiently long to warrant a
# long time
considering the purpose of this file.
Why not? The official policy is that normal tests should take less than 1 second, # long time
tests less than 30 seconds. Remember that 1 second on your machine may be multiple seconds on a slower machine.
comment:564 followup: 566 Changed 4 years ago by
Replying to jdemeyer:
Replying to tscrim:
So while these tests are longer than 1 second on my machine, I think they are not sufficiently long to warrant a
# long time
considering the purpose of this file.Why not? The official policy is that normal tests should take less than 1 second,
# long time
tests less than 30 seconds. Remember that 1 second on your machine may be multiple seconds on a slower machine.
2/3 were less than 1.5 seconds, and my machine is not particularly fast. I am happier to have the shorter tests. Yet, I think this is something that will be tested only by the bots (if they even have this optional package installed) and is meant to catch interface breakage, I did not see as much harm in having longer regular tests.
comment:565 Changed 4 years ago by
Branch:  u/SimonKing/upgrade_of_group_cohomology_spkg → u/jdemeyer/upgrade_of_group_cohomology_spkg 

comment:566 followup: 567 Changed 4 years ago by
Commit:  d4a612c7ebf08d5bb48479163099671fd880994f → 65769150a96fcd8a5fcfad88a4c2173dedb11b42 

Reviewers:  → Travis Scrimshaw, Jeroen Demeyer 
Status:  needs_review → positive_review 
Replying to tscrim:
2/3 were less than 1.5 seconds, and my machine is not particularly fast. I am happier to have the shorter tests. Yet, I think this is something that will be tested only by the bots (if they even have this optional package installed) and is meant to catch interface breakage, I did not see as much harm in having longer regular tests.
On the other hand, those bots typically run tests with long
, so it wouldn't hurt to add # long time
in that case.
I just added a few trivial changes to wrap some long lines, so I guess this is finally good to go...
New commits:
10cdf0e  Add optional package p_group_cohomology3.0

6576915  Wrapping some long lines

comment:567 Changed 4 years ago by
Replying to jdemeyer:
Replying to tscrim:
2/3 were less than 1.5 seconds, and my machine is not particularly fast. I am happier to have the shorter tests. Yet, I think this is something that will be tested only by the bots (if they even have this optional package installed) and is meant to catch interface breakage, I did not see as much harm in having longer regular tests.
On the other hand, those bots typically run tests with
long
, so it wouldn't hurt to add# long time
in that case.
Just for the record: The tests in the final version are all less than a second.
In any case: Thank you for the positive review!
comment:568 Changed 4 years ago by
Branch:  u/jdemeyer/upgrade_of_group_cohomology_spkg → 65769150a96fcd8a5fcfad88a4c2173dedb11b42 

Resolution:  → fixed 
Status:  positive_review → closed 
For me, all but two tests fail. Both tests fail with "connection timed out" during access to a web data base, and both work fine when done in an interactive session.
Has something in the test framework changed, so that web access via
urllib2
fails during tests?