Opened 6 months ago

Closed 4 months ago

Last modified 3 months ago

#28414 closed defect (fixed)

p_group_cohomology does not build on Python3

Reported by: dimpase Owned by:
Priority: major Milestone: sage-9.0
Component: packages: optional Keywords: py3
Cc: SimonKing Merged in:
Authors: Simon King Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: 6c5dafc (Commits) Commit:
Dependencies: #28444 Stopgaps:

Description (last modified by SimonKing)

The aim of this ticket is to provide a new version of the optional p_group_cohomology spkg so that it builds and passes tests both with python-2 and python-3.

Hopefully the version number 3.3 (two times three) is appropriate.

Package documentation

Source ball

p_group_cohomology at Travis CI

Attachments (1)

State.sobj (1.1 KB) - added by SimonKing 6 months ago.
Pickle that isn't readable in Python-3

Download all attachments as: .zip

Change History (167)

comment:1 Changed 6 months ago by jhpalmieri

cmp is also not supported in Python 3, and for a dictionary d, you should do x in d rather than d.has_key(x).

Last edited 6 months ago by jhpalmieri (previous) (diff)

comment:2 Changed 6 months ago by jhpalmieri

Simon, I have made some progress in converting this to Python 3. Let me know and I can email you some diff files.

comment:3 Changed 6 months ago by SimonKing

Hi John, in fact Python-3-compatibility was one of the things that I considered for the next package version. So, if you already have a diff, I'd appreciate to see it!

comment:4 follow-ups: Changed 6 months ago by jhpalmieri

I just sent email to your Jena address. I can get the package to build, but it doesn't work yet. An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  #!/usr/bin/env sage-python23
WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py:623: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
8gp3: Files on disk have been moved - trying to get things right
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-9bd7fbf5ad54> in <module>()
----> 1 load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in __call__(self, result)
    244             self.start_displayhook()
    245             self.write_output_prompt()
--> 246             format_dict, md_dict = self.compute_format_data(result)
    247             self.update_user_ns(result)
    248             self.fill_exec_result(result)

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in compute_format_data(self, result)
    148 
    149         """
--> 150         return self.shell.display_formatter.format(result)
    151 
    152     # This can be set to True by the write_output_prompt method in a subclass

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/display/formatter.py in format(self, obj, include, exclude)
    198         """
    199         # First, use Sage rich output if there is any
--> 200         sage_format, sage_metadata = self.dm.displayhook(obj)
    201         assert PLAIN_TEXT in sage_format, 'plain text is always present'
    202         if not set(sage_format.keys()).issubset(self.default_mime()):

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in displayhook(self, obj)
    809             return
    810         self._backend.set_underscore_variable(obj)
--> 811         plain_text, rich_output = self._rich_output_formatter(obj, dict())
    812         return self._backend.displayhook(plain_text, rich_output)
    813 

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in _rich_output_formatter(self, obj, rich_repr_kwds)
    621         rich_output = None
    622         plain_text = None
--> 623         has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
    624         if has_rich_repr:
    625             rich_output = self._call_rich_repr(obj, rich_repr_kwds)

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getattr__()

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    149 
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)
    153     try:

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    150     ## Load file by absolute filename
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:
    154         X._default_filename = os.path.abspath(filename)

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

comment:5 in reply to: ↑ 4 Changed 6 months ago by SimonKing

Replying to jhpalmieri:

I just sent email to your Jena address. I can get the package to build, but it doesn't work yet.

Thank you very much, I received the patch.

An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

That probably is harmless. I.e., I should use "warning", and apparently in some unpickling method I use "str" where it should be "bytes".

comment:6 Changed 6 months ago by jhpalmieri

Yes, and I would like to track down the str vs. bytes issue. It would be nice if I could get basic functionality like CohomologyRing(8,3) to work.

comment:7 follow-up: Changed 6 months ago by jhpalmieri

Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.

comment:8 in reply to: ↑ 7 Changed 6 months ago by SimonKing

Replying to jhpalmieri:

Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.

I did change .warn( into .warning( everywhere (so, no need to include it in a diff).

Concerning str vs. byte: There is only one occurrence of cdef str, and this gets assigned to the output of <some singular object>.name(). Is that output really a string in python-3? If not, it should be changed to cdef bytes. However, so far the error in unpickling stuff looks mysterious to me, as the traceback doesn't mention where the problem actually occurs in my code.

comment:9 follow-up: Changed 6 months ago by jhpalmieri

Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?

comment:10 in reply to: ↑ 9 ; follow-up: Changed 6 months ago by SimonKing

Replying to jhpalmieri:

Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?

You mean Python 3 is severely backwards incompatible and cannot deal with old pickles? That would be a show stopper for Sage, I think.

Note that at some point in the past I inserted code to regenerate very old pickles and translate them automatically to a new format. However, I would certainly not like to regenerate several gigabytes of data in the web repository.

Note that the problem occurs together with a warning:

WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
...
8gp3: Files on disk have been moved - trying to get things right

Maybe that indicates where the problem comes from. So, let's recall how storing the modular cohomology ring of a prime power group is solved.

Very early in the package development I found that is is terribly slow to pack all data of a ring in a single file. It simply takes so long and requires so much memory (at least in some cases) that it is not feasible to automatically store intermediate results of a lengthy computation (which evidently is a reasonable thing to do).

My solution was to not store a ring in a single file but distribute the data in a folder. So, the actual .sobj pickle file only contains basic information but is not stand-alone. In particular, the p_group_cohomology spkg needs to know where the data folder is located, the .sobj file has to be in the data folder and also it needs write permission for otherwise it couldn't do a continuation of the cohomology ring's computation. What is stored in the web repository is a compressed tar file of the data folder (i.e., when the web repository is accessed, then the tar file is downloaded, unpacked, and then the cohomology ring is loaded from there).

By default, the data folder's location is composed from the location of the current workspace (which are global variables) and from the identifier of the group (which is stored in the .sobj file).

The warning mentioned above occurs when the data folder's default location cannot be accessed. And in that case, I do some trickery --- and perhaps that trickery doesn't work in Python 3? Namely:

  • When an object is obtained from reading a pickle file, then Python (at least Python 2) stores the original file location as an attribute of the object. However, that attribute is only available after successful unpickling.
  • Therefore, when initially the location of the data folder cannot be determined, an unitialised cohomology ring is returned.
  • As soon as the uninitialised cohomology ring is touched (I think constructing its repr is enough), initialisation is triggered. Namely, recall that the location of the .sobj file is stored as an attribute of the uninitialised ring and that the .sobj file is supposed to be located in the data folder -- voilà, you can now determine the data folder and finish initialisation of the cohomology ring.

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

comment:11 follow-up: Changed 6 months ago by SimonKing

Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?

comment:12 in reply to: ↑ 11 ; follow-up: Changed 6 months ago by SimonKing

Replying to SimonKing:

Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?

Here is what I found in the "internetz" concerning how to test if something is a string in a way that works in python 2 and 3:

import six

if isinstance(obj, six.string_types):
    print('obj is a string!')

So, I think I should fix this.

comment:13 in reply to: ↑ 10 ; follow-up: Changed 6 months ago by SimonKing

Replying to SimonKing:

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

Hooray:

# in Sage-with-python-2
sage: P = CombinatorialFreeModule(ZZ,'x')
sage: save(P,'../../pickle')
# BOTH with python-2 and python-3
sage: B = load('../../pickle.sobj')
sage: B._default_filename
'/home/king/Sage/pickle.sobj'

So, the old trick should still work.

comment:14 Changed 6 months ago by SimonKing

Currently, I get the following behaviour:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__()

TypeError: expected bytes, str found

and

sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3/dat/State.sobj'
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1154                 coho_options['@use_this_root@'] = root_workspace
-> 1155                 OUT = load(os.path.join(root_workspace,file_name)) # realpath here?
   1156                 if '@use_this_root@' in coho_options:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO_unpickle()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-4-927128be23f4> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1159                 if '@use_this_root@' in coho_options:
   1160                     del coho_options['@use_this_root@']
-> 1161                 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_local_sources,file_name), msg))
   1162         ## 4. Search web repository
   1163         elif kwds.get('websource')!=False and (not from_scratch):

OSError: Saved data at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj are not readable: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

So, that's something I can now try to debug.

comment:15 follow-up: Changed 6 months ago by chapoton

in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3

comment:16 in reply to: ↑ 15 Changed 6 months ago by SimonKing

Replying to chapoton:

in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3

Thank you! I don't know yet whether or not I'll need them, but certainly it is good to know.

comment:17 Changed 6 months ago by SimonKing

I found that this fails in RESL.__init__:

        tmprstem = os.path.join(res_folder,rstem)
        tmpgstem = os.path.join(gps_folder,gstem)
        self.Data = newResolWithGroupLoaded(tmprstem,tmpgstem,1)

Apparently tmprstem or tmpgstem are strings, but the C function newResolWithGroupLoaded expects char * --- is there no automatic conversion from py3-string to char*?

comment:18 follow-up: Changed 6 months ago by chapoton

Maybe try

self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)

..with no guarantee..

comment:19 in reply to: ↑ 18 Changed 6 months ago by SimonKing

Replying to chapoton:

Maybe try

self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)

..with no guarantee..

That is likely to work. Another potential solution would be to generally work with bytes. I.e., let self.gstem be a bytes and then do os.path.join(self.res_folder, b'A'+self.gstem+b'.sobj')) etc.

What would be the preferred way?

comment:20 follow-up: Changed 6 months ago by chapoton

beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes

EDIT:

indeed:

sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components
Last edited 6 months ago by chapoton (previous) (diff)

comment:21 in reply to: ↑ 20 Changed 6 months ago by SimonKing

Replying to chapoton:

beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes

EDIT:

indeed:

sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components

This only shows that it cannot mix both types. In fact, if both are bytes then things work:

sage: os.path.join(b'home', b'foo')
b'home/foo'

Probably it would be a long and tedious work to change everything to bytes. In principle I would be willing to do that work, but if there is a different solution involving less work and only a slight overhead, I'd prefer that.

Currently, using str_to_bytes when passing a string to a C function does seem to work.

comment:22 Changed 6 months ago by SimonKing

WTF? I just see that I have the following argument types of COCH.__init__:

def __init__(self, PARENT, int n, char *Nick, L, ydeg=None, rdeg=None, is_polyrep=False)

I suppose Nick shouldn't be char* but (if anything) str. Or better not type it at all.

comment:23 Changed 6 months ago by SimonKing

I think I should also fix the following compiler warning (there are several similar):

pGroupCohomology/cohomology.c: In function ‘__pyx_pf_16pGroupCohomology_10cohomology_12explore_one_parameter’:
pGroupCohomology/cohomology.c:17678:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (__pyx_t_7 = 0; __pyx_t_7 < __pyx_t_16; __pyx_t_7+=1) {

comment:24 Changed 6 months ago by SimonKing

I guesss getting the following error is a progress, although it seems a bit strange to me:

sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
> export action matrices
Unable to save basic ring setup
Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
    obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)
    obj.save(filename=filename, compress=compress, **kwargs)
  File "sage/structure/sage_object.pyx", line 424, in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)
    filename = _base_save(self, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
  File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
TypeError: can't pickle dict_items objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
    obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py", line 1254, in _get_p_group_from_scratch
    safe_save(OUT, OUT.autosave_name())
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
  File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
TypeError: can't pickle dict_items objects
Computing next term
> rk P_02 =   1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'dict_items' object has no attribute 'save'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/structure/sage_object.pyx in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)()
    423 
--> 424         filename = _base_save(self, filename, compress=compress)
    425 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     84         os.unlink(path)
---> 85     save (obj, path)
     86 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     84         os.unlink(path)
---> 85     save (obj, path)
     86 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

TypeError: can't pickle dict_items objects

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'dict_items' object has no attribute 'save'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.make()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.next()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

TypeError: can't pickle dict_items objects

What are dict_items, and why can't they be pickled?

comment:25 follow-up: Changed 6 months ago by chapoton

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

comment:26 in reply to: ↑ 25 ; follow-ups: Changed 6 months ago by SimonKing

Replying to chapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.

comment:27 follow-up: Changed 6 months ago by SimonKing

There is a considerable progress now (after putting list around .items()):

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
> export action matrices
Computing next term
> rk P_02 =   1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
Determine degree 2 standard monomials
We got 1 standard monomials
Monomial 0
> Candidate: c_1_0^2
> Express monomial as a Cochain
Compute c_1_0*c_1_0
Compose chain maps R_2 -> R_1 -> R_0
Lift with Urbild Groebner basis in degree 1
load Urbild Groebner basis
Decomposable cochain found

There is no new generator in degree 2
Summary: 0 relations and 0 generators in degree 2
Ring approximation computed out to degree 2!
Storing approximation data
Inserting SmallGroup(4,2) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
> export action matrices
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

KeyError: 'autoliftElAb'

The good news is that for the first time cohomology rings were computed in py-3 (namely cohomology rings of elementary abelian groups).

comment:28 in reply to: ↑ 26 Changed 6 months ago by jhpalmieri

Replying to SimonKing:

Replying to chapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.

You could use a try ... except block to deal with the Python 2 case quickly. Sage will soon (I hope) default to Python 3, so I don't think this is worth it, though.

comment:29 in reply to: ↑ 27 Changed 6 months ago by SimonKing

Replying to SimonKing:

There is a considerable progress now (after putting list around .items()): ...

 KeyError: 'autoliftElAb'

Apparently I forgot to create a shallow copy in some places in which I temporarily stored the global options -- it doesn't work to recreate a dict from its items in Py-3, and that's why I got the key error above:

sage: D = {1:2}
sage: I = D.items()
sage: D.clear()
sage: D.update(I)
sage: D
{}

In Py-2, one would get

sage: D
{1: 2}

comment:30 Changed 6 months ago by SimonKing

Hooray!

I am now in a position to completely compute CohomologyRing(8,3) from scratch. Unfortunately reading old data doesn't work yet, but certainly it is a progress :-)

comment:31 in reply to: ↑ 12 ; follow-up: Changed 6 months ago by jdemeyer

Replying to SimonKing:

import six

Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.

comment:32 in reply to: ↑ 26 ; follow-up: Changed 6 months ago by jdemeyer

Replying to SimonKing:

Replying to chapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2?

Use list(d.iteritems()) in Cython.

comment:33 in reply to: ↑ 31 ; follow-up: Changed 6 months ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

import six

Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.

I am not dealing with incompatibilities in that case. I import six in order to be able to safely test for string types. How to, if I don't use six?

comment:34 in reply to: ↑ 32 Changed 6 months ago by SimonKing

Replying to jdemeyer:

Use list(d.iteritems()) in Cython.

Would this also work fast if Cython doesn't know that d is a dict? Should I do list(<dict>(self._D).iteritems()), for instance?

Last edited 6 months ago by SimonKing (previous) (diff)

comment:35 in reply to: ↑ 33 Changed 6 months ago by jdemeyer

Replying to SimonKing:

I import six in order to be able to safely test for string types. How to, if I don't use six?

First of all, you should check whether you really need to test for "string types". Can you use duck typing instead? If for some reason, duck typing is hard, then it depends what you mean precisely with "string types".

Since you were mentioning filenames: isinstance(s, (bytes, unicode)) are valid types for filenames both in Python 2 and Python 3. Note that Python 3 also has https://docs.python.org/3/library/pathlib.html, a custom class to deal specifically with filenames.

Last edited 6 months ago by jdemeyer (previous) (diff)

Changed 6 months ago by SimonKing

Pickle that isn't readable in Python-3

comment:36 in reply to: ↑ 4 Changed 6 months ago by SimonKing

Replying to jhpalmieri:

I can get the package to build, but it doesn't work yet. An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
... 
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

I tried to track down that problem, and currently it seems to me that it is due to a very bad backwards incompatibility in Python. You can reproduce it as follows:

  • Download attachment:State.sobj
  • Install the optional meataxe spkg (this may involve sage -b, so that some optional module in the Sage library is built)
  • Define some unpickle helper:
    sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle
    sage: class unpickle_old_mtx:
    ....:     def __call__(self, *args, **kwds):
    ....:         return mtx_unpickle(*args, **kwds)
    ....:     
    sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
    

Now, in Sage-with-Py-2, loading the file works:

sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
sage: X
([[1,
   [(2, 1),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_2']
]]],
  [2,
   [(4, 2),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1]                                                                                                                   
[0 1 0 0 0 0 0 0]                                                                                                                   
[0 0 0 0 0 1 0 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_2']
]]],
  [3,
   [(4, 2),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1]                                                                                                                   
[0 0 1 0 0 0 0 0]                                                                                                                   
[0 0 0 0 0 0 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_2']
]]]],
 [2, 1, 1],
 1,
 1,
 '8gp3/',
 ['b_1_0*b_1_1'],
 '8gp3/dat/',
 [((4, 2), '/home/king/.sage/pGroupCohomology/db/4gp2/H4gp2'),
  ((2, 1), '/home/king/.sage/pGroupCohomology/db/2gp1/H2gp1')],
 '8gp3/dat/',
 [],
 '8gp3/sgp/',
 2,
 [2, 3],
 [2, 2],
 2,
 2,
 ['b_1_0*b_1_1'],
 [[2, 'c_2_2', [0 0 1], 1, 0],
  [1, 'b_1_0', [1 0], 0, 0],
  [1, 'b_1_1', [0 1], 0, 0]],
 [[0, [['1', ['1']]]],
  [1, [['b_1_0', ['b_1_0']], ['b_1_1', ['b_1_1']]]],
  [2,
   [['c_2_2', ['c_2_2']],
    ['b_1_0', ['b_1_0^2']],
    ['b_1_1', ['b_1_0*b_1_1', 'b_1_1^2']]]]],
 {},
 0.24,
 True,
 'Monomials',
 2,
 None,
 '',
 0,
 '8gp3',
 '8gp3/dat/R8gp3.sobj',
 1,
 ['0'],
 ['b_1_0 + b_1_1'],
 None,
 [('_max_module_deg', 0),
  ('sgpDickson', [((4, 2), [[1, 'c_1_1', [0 1]]])]),
  ('useFactorization', True),
  ('useElimination', False),
  ('GroupDescr', 'Dihedral group of order 8'),
  ('_key', ((8, 3), '/home/king/SPKG/database/8gp3/dat/State')),
  ('auto', 4),
  ('Restriction_2', ['c_1_0*c_1_1+c_1_0^2', 'c_1_1', '0']),
  ('GroupName', 'D8'),
  ('DicksonExp', 3),
  ('_parameters_do_exist', True),
  ('KeepBases', None),
  ('completeGroebner', True),
  ('_SymondsTestdata', ['c_2_2', 'b_1_1', 'b_1_0']),
  ('_parameters_for_criterion', ['c_2_2', 'b_1_1', 'b_1_0']),
  ('root', '/home/king/SPKG/database/'),
  ('Restriction_3', ['c_1_0*c_1_1+c_1_0^2', '0', 'c_1_1']),
  ('_method', 'Symonds')],
 [(('dependent_parameters',), [['c_2_2', 'b_1_1', 'b_1_0'], 2])])

However, Sage-with-Py-3 yields the error found by John:

sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-4-74e7e91f9969> in <module>()
----> 1 X = load('/home/king/Projekte/coho/tests/State.sobj')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    149 
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    150     ## Load file by absolute filename
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:
    154         X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

And the point is: Python-3 even is unable to read the pickle file as a string! In Python-2, reading the file is no problem at all.

king@klap:~$ ~/Sage/git/py3/sage -python
Python 3.7.3 (default, Aug 27 2019, 23:22:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/king/Sage/git/py3/local/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte
>>> 
king@klap:~$ ~/Sage/git/sage/sage --python
Python 2.7.15 (default, Jul 26 2019, 11:49:43) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
'x\x9c\x95U\x8bZ\x1bE\x14&!\x91\x92J\x8b\x82V\xbc\x94x\xedXl\x96\x04\xaah\x95B\x0b\x05]\xc1\x16\x18\xad\x97\xe9:\xd9\x1dv\xb7d3{v7\nU,^\xd8B\xbd\xd6\xfb\xfd\xae\xcf\xd0\xa7\xf0E|\t\xcfl\xb2\x08\xf5\xab_I\xbe\x9d\xefd\xe6?\xe7\xfc\xe7?g6kY\xc2 \x83O\x96\xe8\x19\x06\xed\xc4\x0c\xb9-J\x81[\xb7\xc3\x92[\x8f\x84-\x82\x82\xc7\x97\x84\x91\xfe\x80\x1c\xcdT\xd6\xe7 \xef\xa0QF\xe3\xa6\x18:\x18\xec!\xc4\xf4\xa7\x02\xd9\xf0OJGz\xb2&\xed\x95\x92\x17-\x17f\x16\xce\x1a\x8d\xba\xef\x9aK5a\x985\x1e\x86\x05\xe8\x94PX\x85\xbd\xf4\x80a\x84|Q\x18\x8b2H1\x98\xd80\xe0\xe6\x8d\xb0J\xf4,~\xf7\xd0\xecZ\xffF4\x07]\x0c\xf6\x11\xa7\xa0v3\xb8\x9bY\xdb\xc4\xcd\xfdt@\xc3lB[B?m\xfe\xb4>\xa5Y<\xe2U\x1e\nm\xc4\xf6\x87\xb4\xd0\xf6\x13#\xb4\xcbF\x19\xbaw\x03\xaf\xc0-\x02?\x0cn\xc5\x9c\x0cz\x88*x\x18\x0b\xeeuZ\x12\xdc\x16\xc3\xed\x0c\x0e\xb4X\xe5\x90Un\xad\x7f,\xaf\xd8\xde\xc1\xa0\xef?l\xef\xbc\xf1\xf4\x15d{\xd7n\xe0\x15\xb8\xbb\xc9\xf6\x1e\xa2\xb738\xb8\xc5\xb6?e[\x8c\xe1^\x06\xf7\xedd[lWl\xefg\xf0\x00qvAo\x08\xe9=\xb8\x1bx\x05\x1e\x12M~\x87\x9a\x9adD:?$5\x1e\xa6\xf9\xc4\x8f\xc1a\xba\xb7j\x94\x8d\xc1\xc3j-\xc3\x00\xa7\x9d\xc9\t\xc6\xc7\xd3G\x88\xd3\xeb\xa0\xf6G\xa8\xb6\x8d@I\r\xaev\xed\x08jVU\x1b\xb6\xfd\x8a6\xad\xd6\x18JN\xde\xc1\x81\xd5n\xd0\xb5b\xfbemZ\xad1\x0c\x8a\xed,Z\xb6\xaaQMG\x99\xa42W\x941\x84\xc6\x10\x16;\xbc\xb5}45\x1e\x15\xceQ\xe5\xf1\xd8\xce\x1aG8\x83\xc7\xf1\x1e>\x81\xf2\xd0\xbci\xa8\x8e\x1eK\x07\xa8\x9df\x8aj\x80\x9eD\xbb\r\xc3>\x85\x97\x95\xe6\x13w\x18MA\xd9\xd6\x94\x1d\xd7\xdb\x12\xd0\xd8\x16\xa8\x0c\xe3\xdb@c\nt"\x01!\xea$\xe6\x9c z\x1b\x83I\x06\xa7\x08v\x82\xc1\x14\xae\\p<\x9dN\xde\nO#\xe6\x19\xe2\x8c2\xd0\xd3\xac\xcf\xaa\xd3\x19\xe2\x8c3\x98M\x93<\xc7\x93\x06\x9fN\xae\xcb\x19\xf4\x99#\xce1\x06\xf3i9\x0b\xca\x87\x924\xc4\xf3\x0c^\xa0\x1d\x89}\xae\x02g\xd5\xe1\x8b$\x8d\xf5\x12\x83\x97\xc9N\x85^i\x82\xcb\x08f\xc9,\xad\xc2\xb9\xa9\xe3\x7f]=\xf3\xf7\xfa\xc1\xab\x1b\xb4sF\xd6\xa5\xe7\xf2Z\x08F\xaa\xf5\xab\xb3\x14\xcb\xa49\xd5*\xe0\xb4g\xab}s\xca*\x85\xb2z^\xd5W\xa5\x99A\x94\xdf\xa4]I\xbe\xe2@\xb1\x99\xd1\xe2\xb3\x0c\x04\xa1\xfb\r\x8f/\x1b\x9e\xb4\x1a\xf8\n\xb3\x84\r\x8b*\xc1 &\xb0cph\x01\x87`\x02\xdf]\xa1\xac\x83\xcb\xe0\xbcs\x84\xc1\x12\x83Z\xa2\xbf\x99D\xf2\xae\xd5\xbf.x\x0c\x12\x1f\x9fv7Bq\x8a\x9b\x91\x0c\xdc\x0b<r1\x08l\xc4\x10\xd0}\xb8?Ys=\xb7\xde\xdc\r7c\x88h!\x99\xd2\t\x11\x9a\x014h\xdf\x84\xeb\x08+\xe0\xb5\xa2\xad\xb6\x8br\xb1(\x03K\x04\xc5\x11x-\x86\xd7i\xceX\x12+\xb0\xac\xe8\x8e \xdd\x95t6/\xc4\xf0\x06=\xf4\xff\xd7W\xe94\x1f\xf1H\xc4\xf0f\x0c\xab4\xc7\x1b\x91\x84\xb7\xf4\\\x0c\x17i\xd7\x9c\x08\xa3\xc05\x153l\xed\x1a\x83\xb7\t\xed1\x93v%%\x0f\x98\xad\xc6\xbe\x93j\xf0.J\x86\xb1\xde\xa3\x9dI\t\xb3\xdc\x13\xb0N\xb3\x13#\x10\xc7p\x89\x16Z\x12N.\xfb\xb0\xa1\xb7\xc7\xb0I{\r\x9f\x07\x08\x8bD\x10\x1a\x964\xc4\xb2\x1bFp\x19\xd5y\x9fv\xeaB\xf8\'\x90l\x08\x1f\xcc\xc6\xf0!\xed6\xa5\xe7\xd7\x10\x8b\xd1E\xb5.\x02\xf8\x08\x81\x1f\xd3nc~\xc5\x93u+\\@\xc6\xaaB\xf8\x84\xc1\x15\x9cLg\xdc\x19E>\x9f\xd2\xbe\xedi\xd4_\x91\x19\xb8h+\xcd?c\xf0\xf9\xbf\xd0/h.\x902\x82/i\xdfu\xa5\x83\xafb\xf8z\xa7<C\xf0\r\x83o\xaf#\xcfw\xa8J\xaa\xd0\xf7\x98\xe3\x07\xdaa \x15GZ\xf0#\xedhQ\x87\x9fb\xf8\x19\xef\xc7/\xb4\xd7\x12\xbe\xa8[\xa2\x1em#\r\xbf\xae\xc3o\x0c~\'\xce\x15=\x8b1\xfe\xe0\x11\xfcY\xfa\x07\x14\x1f\x95\xac'

comment:37 follow-up: Changed 6 months ago by chapoton

This works fine in python3:

sage: with open('State.sobj', 'rb') as f:
....:     f.read()

comment:38 in reply to: ↑ 37 Changed 6 months ago by SimonKing

Replying to chapoton:

This works fine in python3:

sage: with open('State.sobj', 'rb') as f:
....:     f.read()

Do I understand that it works because b means "binary"? But of course the real question is why Python 3 cannot unpickle from that file.

I tried to dig a bit deeper into Sage's load function (I guess the error is hidden there), and indeed load() uses with open(filename, 'rb') as fobj:. So, the unpickling problem is something else:

sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle
sage: filename = '/home/king/Projekte/coho/tests/State.sobj'
sage: compress = True
sage: class unpickle_old_mtx:
....:     def __call__(self, *args, **kwds):
....:         return mtx_unpickle(*args, **kwds)
....:     
sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
# The following is copied from Sage's load() function.
sage:     ## Load file by absolute filename
....:     with open(filename, 'rb') as fobj:
....:         X = loads(fobj.read(), compress=compress)
....:     try:
....:         X._default_filename = os.path.abspath(filename)
....:     except AttributeError:
....:         pass
....: 
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-654643a51ef6> in <module>()
      1 ## Load file by absolute filename
      2 with open(filename, 'rb') as fobj:
----> 3     X = loads(fobj.read(), compress=compress)
      4 try:
      5     X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

So, now we see that the error occurs in SageUnpickler(io.BytesIO(x)).load().

Last edited 6 months ago by SimonKing (previous) (diff)

comment:39 follow-up: Changed 6 months ago by SimonKing

I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.

So, I am now trying to find if this is really the case or not.

comment:40 in reply to: ↑ 39 ; follow-up: Changed 6 months ago by SimonKing

Replying to SimonKing:

I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.

So, I am now trying to find if this is really the case or not.

It isn't. I inserted a print statement at the beginning of mtx_unpickle, but it isn't shown before the error triggered by load('/home/king/Projekte/coho/tests/State.sobj') is raised.

I'll open a ticket for that problem.

comment:41 in reply to: ↑ 40 Changed 6 months ago by SimonKing

Replying to SimonKing:

I'll open a ticket for that problem.

It is #28444. I am now convinced that the problem is in Python-3 resp. in Sage's unpickling, but not in MeatAxe.

comment:42 Changed 6 months ago by SimonKing

  • Dependencies set to #28444

comment:43 Changed 5 months ago by SimonKing

It seems that #28444 fixes the problem for p_group_cohomology on Python-3. With the current unpublished code for the package, I get

$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta9, Release Date: 2019-09-02               │
│ Using Python 3.7.3. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Get original root and GStem
/home/king/SPKG/database 8gp3
now, StateFile is /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj
Try to get (8, 3) from cache
Group data are rooted at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj'
Setting state
> export action matrices
Differential reloaded
> rk P_02 =   3
Differential reloaded
> rk P_03 =   4
Import monomials
Checking compatibility of SmallGroups library and stored cohomology ring
sage: print(H)

Cohomology ring of Dihedral group of order 8 with coefficients in GF(2)

Computation complete
Minimal list of generators:
[c_2_2: 2-Cocycle in H^*(D8; GF(2)),
 b_1_0: 1-Cocycle in H^*(D8; GF(2)),
 b_1_1: 1-Cocycle in H^*(D8; GF(2))]
Minimal list of algebraic relations:
[b_1_0*b_1_1]

Hooray...

comment:44 Changed 5 months ago by SimonKing

Strangely, while I can compute some cohomology rings, the following computation fails in py3:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: H = CohomologyRing(32,4)
sage: H.make()

Reason: Automatic saving does not work, and apparently it is because H.__getstate__() returns Singular elements, which cannot be pickled. It seems I did take care of it, as H.__getstate__() in py2 only returns the class sage.interfaces.singular.SingularElement and not the instance, but for some reasons it doesn't in py3.

comment:45 Changed 5 months ago by SimonKing

Question: Has the mechanism of tab completion changed between py2 and py3? I notice that tab completion on a cohomology ring works in py2, but doesn't work (no error, but no completion either) in py3.

comment:46 Changed 5 months ago by dimpase

tab completion is done by ipython, which should be upgraded (they've switched to Python 3 only, so we're using an old version, which in turn might lag in its Python 3 support).

I suggest not to worry about it now.

comment:47 Changed 5 months ago by SimonKing

Aha! The culprit for pickle problem is pGroupCohomology.cohomology.pickle_gap_data, which inspite of its name also takes care of singular elements. In Py2:

sage: self = CohomologyRing(32,4)
sage: self.make()
sage: G = self._decorator_cache['_parameter_restrictions',frozenset({'c_2_2'}),('radical', True)]
sage: G
[[(polynomial ring, over a field, global ordering
   // coefficients: ZZ/2
   // number of vars : 2
   //        block   1 : ordering M
   //                  : names    c_1_0 c_1_1
   //                  : weights      1     1
   //                  : weights     -1     0
   //        block   2 : ordering C,
   <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca690>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
   <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca500>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
 2]
sage: pickle_gap_data(G)
[[(<class 'sage.interfaces.singular.SingularElement'>,
   <class 'sage.interfaces.singular.SingularElement'>,
   <class 'sage.interfaces.singular.SingularElement'>)],
 2]

In Py3:

sage: pickle_gap_data(G)
[[(polynomial ring, over a field, global ordering
   // coefficients: ZZ/2
   // number of vars : 2
   //        block   1 : ordering M
   //                  : names    c_1_0 c_1_1
   //                  : weights      1     1
   //                  : weights     -1     0
   //        block   2 : ordering C,
   <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86a20>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
   <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86dc8>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
 2]

comment:48 Changed 5 months ago by SimonKing

Ouch. A nasty thing.

In pickle_gap_data, I test if the input is an iterable class, and if it is and if the class is a builtin, then the pickle_gap_data traverses the items of the input and creates from the individual results a modified copy of the input. In code:

    try:
        I = iter(G)
    except:
        return G
    if isinstance(G,dict):
        return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items())
    if getattr(type(G), '__module__', None) == '__builtin__':
        return type(G)(pickle_gap_data(X) for X in I)
    return G

The problem is that in Py3 we get

sage: getattr(list, '__module__', None)
'builtins'

Question: How to test in a py2/py3 invariant way whether a type is a builtin?

That asked, are there more iterable builtins than list, tuple, str/bytes and dict? I guess it is safer to just explicitly test for list and tuple. EDIT Like this:

    if isinstance(G, dict):
        return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items())
    if isinstance(G, (list, tuple)):
        return type(G)(pickle_gap_data(X) for X in G)
    return G
Last edited 5 months ago by SimonKing (previous) (diff)

comment:49 follow-up: Changed 5 months ago by dimpase

An iterable is an object with an __iter__ method or/and __getitem__ method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)

comment:50 in reply to: ↑ 49 Changed 5 months ago by SimonKing

Replying to dimpase:

An iterable is an object with an __iter__ method or/and __getitem__ method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)

You misunderstood. The problem was not how to test if something is iterable but how to test if that iterable is a builtin. But, as I said in my previous post, probably this isn't the actual problem anyway. What I want is to deal with strings, lists, tuples and dicts. Period. So, I could be more explicit (and I already tested that the error from comment:44 can be fixed in that way).

comment:51 Changed 5 months ago by SimonKing

In Python-2, one implemented c.__nonzero__() and then bool(c) would work. It seems that in Python-3, something else needs to be done. Is it perhaps the case that I need to implement __bool__?

What should I do to make stuff work both in py-2 and py-3? Implement both __bool__ and __nonzero__? Or would it be enough to replace __nonzero__ with __bool__?

comment:52 Changed 5 months ago by SimonKing

I found this

class test:
    def __bool__(self):
        return False
    __nonzero__=__bool__

here.

comment:53 follow-up: Changed 5 months ago by SimonKing

There is a problem with the logging module: In Python-2, I get

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: P = CohomologyRing(9,2,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/19448/dir_0MUJxK/'
Computing basic setup for Small Group number 1 of order 3
Computing basic setup for Small Group number 2 of order 9
Resolution of GF(3)[9gp2]: 
          > export action matrices
H^*(SmallGroup(9,2); GF(3)): 
          Initialising maximal p-elementary abelian subgroups

whereas in Python-3 I get

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: P = CohomologyRing(9,2,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/18649/dir_9bz1_9bh/'
Computing basic setup for Small Group number 1 of order 3
Computing basic setup for Small Group number 2 of order 9
> export action matrices
Initialising maximal p-elementary abelian subgroups

So, the indentation is gone, and (whats worse) the logging doesn't indicate what object the log comes from (which I found very useful for debugging, as in the computation of a single cohomology ring often many other rings are involved).

Can someone please give me a pointer to a page explaining the differences between py-2 and py-3 in the logging module?

comment:54 in reply to: ↑ 53 Changed 5 months ago by SimonKing

Replying to SimonKing:

Can someone please give me a pointer to a page explaining the differences between py-2 and py-3 in the logging module?

By looking at the code of the logging module in Python-2 and Python-3, I see where the problem comes from. In the following, self denotes an instance of pGroupCohomology.auxiliary.CohoFormatter, which sub-classes loggin.Formatter.

In Python-2, logging.Formatter.format(self, record) uses self._fmt to format the output. In Python-3, self._fmt is not used. Instead, self._style._fmt is consulted, where self._style is one of the styles in logging._STYLES.

The point is: So far, my custom self.format modifies self._fmt and then calls the default .format(). However, in Python-3, when calling the default .format(), the attribute self._fmt plays no role, but this is where I currently put the format into.

Simple way out: Do not call logging.Formatter.format in CohoFormatter.format.

But I wonder: Is there a recommended way to make the logging format change from call to call?

By this, I mean: Make the output of CohoFormatter.format(self, record) depend on record.args[0], in the sense that the output makes visible whether record.args[0] is the same object as in the previous call to CohoFormatter.format.

Or, more radically: What kind of logging approach would you recommend, what do you find most easy to comprehend/parse as a human (I am not interested in automatic parsing of the logging messages)?

I somehow like the current approach, which is: Assume that certain objects (say, ZZ and RR) are involved in a computation, and each object may issue a logging message. Then, stream of messages is arranged in blocks of messages coming from a single object, each block being prepended by an identifier of the object. I.e., if ZZ gives two messages, then RR gives three messages and then again ZZ gives a message, the output would be

Integer Ring:
    message 1
    message 2
Rational Field:
    message 3
    message 4
    message 5
Integer Ring:
    message 6
  1. Do you like this thematic arrangement of messages (which is provided by coho_logger in Python 2, but fails in Python-3)?
  2. If yes: Can you recommend a clean and easy way to achieve it both in Python 2 and Python 3? Ideally, one should still be able to include time information in the logging, which currently is not supported.
  3. If no: What logging format would you recommend?

comment:55 follow-up: Changed 5 months ago by jhpalmieri

Instead of setting self._fmt, can you instead run self.basicConfig(format=FORMAT)? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#user-defined-objects), so maybe you can make use of that somehow.

comment:56 in reply to: ↑ 55 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

Instead of setting self._fmt, can you instead run self.basicConfig(format=FORMAT)? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#user-defined-objects), so maybe you can make use of that somehow.

From these links, I get the impression that each call to the logger results in the creation of a new formatter. Is that really the case?

comment:57 follow-up: Changed 5 months ago by jhpalmieri

I don't know, I only looked at the links, too. It's worth trying.

(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)

comment:58 in reply to: ↑ 57 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

I don't know, I only looked at the links, too. It's worth trying.

(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)

Sure. But I am unsure how a change of formatter could solve the problem without impact on performance.

If logging is in "warn" mode, then there are of course still silent calls to coho_logger.debug. I want that (1) these calls are cheap (i.e., they shouldn't involve creation of Python objects) and (2) the message blocks should not take into account silent messages, i.e., a block shouldn't be closed if a silent message comes from a different object.

Of course, it should be that what I do with the formatter should rather be done with a handler, or maybe a filter. So, I will try to understand what exactly what handlers and filters do.

And, as in point 3. of comment:54: Would you recommend a different formatting of log messages?

comment:59 Changed 5 months ago by SimonKing

Currently I am reading logging HOWTO and logging Cookbook, and at the moment it seems to me that I want to use logging.LoggingAdapter, because it allows to add contextual information (which is what I'd like to use, unless people tell me a nicer way log format).

comment:60 Changed 5 months ago by SimonKing

I think I now know what to do: The customisation should be done in a Formatter, i.e., the existing solution was a good approach. Only the format() method should be different: Before calling super(CohoFormatter, self).format(record), I should add some attribute to the record that is then used in the formatter's format string.

comment:61 Changed 5 months ago by SimonKing

I think I solved the problem. I can now optionally add time information to the log, such as here:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: CohomologyRing.global_options('time')
sage: H = CohomologyRing(720,763,prime=3)
    2019-09-18 20:54:20,277: Try to compute a Sylow 3-subgroup
    2019-09-18 20:54:20,283: Try to find the SmallGroups address of the Sylow subgroup
    2019-09-18 20:54:20,288: We compute this cohomology ring from scratch
    2019-09-18 20:54:20,289: Group data are rooted at '/home/king/.sage/temp/klap/962/dir_tnhvqeq7/'
    2019-09-18 20:54:20,293: Computing basic setup for Small Group number 1 of order 3
    2019-09-18 20:54:20,405: Computing basic setup for Small Group number 2 of order 9
Resolution of GF(3)[9gp2]:
    2019-09-18 20:54:20,466: > export action matrices
H^*(SmallGroup(9,2); GF(3)):
    2019-09-18 20:54:20,517: Initialising maximal p-elementary abelian subgroups
    2019-09-18 20:54:20,555: Computing intermediate subgroup
    2019-09-18 20:54:20,567: Try to find the SmallGroups address of the intermediate subgroup
    2019-09-18 20:54:20,603: Computing group order
    2019-09-18 20:54:20,604: The group is of order 72
    2019-09-18 20:54:20,609: Computing intermediate subgroup
Resolution of GF(3)[SmallGroup(9,2)]:
    2019-09-18 20:54:20,625: Computing next term
    2019-09-18 20:54:20,644: > rk P_02 =   3
H^*(SmallGroup(9,2); GF(3)):
    2019-09-18 20:54:20,647: We have to choose 2 new generators in degree 1
    2019-09-18 20:54:20,648: > There are 2 nilpotent generators in degree 1
...

However, what I'd find even nicer if the sum of the cputimes of Sage and Singular since start of logging was shown, not just the walltime, as above.

Note that the indentation has changed, which means that I will have to change various doc tests, but that will be doable.

comment:62 Changed 5 months ago by SimonKing

Sigh.

After solving the log problem, I find that meanwhile the code ONLY works in Python-3, not Python-2.

First problem:

class FOO(object):
    def __bool__(self):
        return something
    __nonzero__ = __bool__

works in Py3, but not in Py2, although it was recommended in some stackoverflow discussion. In Py2 it results in NameError: name '__bool__' is not defined. How to fix it without code duplication?

Second problem:

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.py in <module>()
     47 import re,os
     48 
---> 49 import urllib.request, urllib.error
     50 import tarfile
     51 import logging

ImportError: No module named request

How to deal with urllib in a py2/py3-compatible way?

comment:63 follow-up: Changed 5 months ago by jhpalmieri

For the urllib issue: http://python3porting.com/noconv.html#import-errors suggests code like

try:
    from urllib.request import ...
except ImportError:
    from urllib import ...

I can't reproduce the first problem. With Python 2, if I do

sage: class FOO(object):
....:     def __bool__(self):
....:         return something
....:     __nonzero__ = __bool__
....:     
sage: a = FOO()
sage: a.__nonzero__()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-74dab0bd3f82> in <module>()
----> 1 a.__nonzero__()

<ipython-input-1-ec93abb3d539> in __bool__(self)
      1 class FOO(object):
      2     def __bool__(self):
----> 3         return something
      4     __nonzero__ = __bool__
      5 

NameError: global name 'something' is not defined

so a.__nonzero__() is certainly calling __bool__. If I change the return line to something sensible, it works.

comment:64 in reply to: ↑ 63 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

For the urllib issue: http://python3porting.com/noconv.html#import-errors suggests code like

try:
    from urllib.request import ...
except ImportError:
    from urllib import ...

OK, thank you.

I can't reproduce the first problem. With Python 2, if I do

The "something" was of course figuratively. Anyway, the problem seems to be that I am using a cdef class here:. In Sage-with-py2:

sage: cython('''
....: class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')
sage: cython('''
....: cdef class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')
ERROR:root:An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line string', (1, 0))

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-91e48d391924> in <module>()
      4         return True
      5     __nonzero__ = __bool__
----> 6 ''')

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/lazy_import.pyx in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3697)()
    352             True
    353         """
--> 354         return self.get_object()(*args, **kwds)
    355 
    356     def __repr__(self):

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_compile(code, **kwds)
    649     with open(tmpfile,'w') as f:
    650         f.write(code)
--> 651     return cython_import_all(tmpfile, get_globals(), **kwds)

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_import_all(filename, globals, **kwds)
    539       code
    540     """
--> 541     m = cython_import(filename, **kwds)
    542     for k, x in iteritems(m.__dict__):
    543         if k[0] != '_':

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_import(filename, **kwds)
    519     try:
    520         sys.path.append(build_dir)
--> 521         return builtins.__import__(name)
    522     finally:
    523         sys.path = oldpath

/home/king/.sage/temp/klap/23274/spyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0.pyx in init _home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0()
      3     def __bool__(self):
      4         return True
----> 5     __nonzero__ = __bool__

NameError: name '__bool__' is not defined

whereas in Sage-with-py3 I get

sage: cython('''
....: class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')
sage: cython('''
....: cdef class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')

So, what to do?

comment:65 follow-up: Changed 5 months ago by jhpalmieri

I don't know Cython, but replacing __nonzero__ = __bool__ with

def __nonzero__(self):
    return self.__bool__()

seems to work.

comment:66 in reply to: ↑ 65 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

I don't know Cython, but replacing __nonzero__ = __bool__ with

def __nonzero__(self):
    return self.__bool__()

seems to work.

Sure. that's what I basically meant with "code duplication". It is an additional call, thus, if the boolean value of a cochain is tested very often then one will see a slow-down. But I don't know if it actually is tested very often.

Anyway, with this code cuplication, things seem to work. Hooray!

At the moment, it seems that most of the remaining work will be to change the expected output for those tests that use logging.

comment:67 follow-up: Changed 5 months ago by jhpalmieri

My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does __nonzero__ do anything with Python 3? If so, maybe you could do something like

from six import PY2
if PY2:
    def __nonzero__(self):
        ...

to avoid the extra call when using Python 3.

comment:68 in reply to: ↑ 67 ; follow-up: Changed 5 months ago by SimonKing

Replying to jhpalmieri:

My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does __nonzero__ do anything with Python 3? If so, maybe you could do something like

from six import PY2
if PY2:
    def __nonzero__(self):
        ...

to avoid the extra call when using Python 3.

I don't think that __nonzero__ does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current not-yet-published code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for __nonzero__ and for __bool__, but only one of them would be compiled.

comment:69 Changed 5 months ago by SimonKing

One detail: According to unit_test_64, there is no significant speed difference between the py2 and py3 versions. In any case, the cputime obtained as sum of the cputime() and Singular's timer is only half of the walltime. Do I recall correctly that the time used to save data on disk is not measured by cputime()? Is there an easy way to determine that time, too?

comment:70 Changed 5 months ago by dimpase

I don't think it makes much sense measuring time to save to disk, as it varies wildly between different OS's, different hardware, different configuration of buffering, etc etc.

Often saving is completely asyncronous from the computation due to buffering and separate threads run to do buffer flushing, etc.

comment:71 Changed 5 months ago by SimonKing

Now I am totally baffled. Is it not the case that comparison is supposed to be implemented by __richcmp__? Apparently __richcmp__ isn't called anymore.

The following corresponds to the code in pGroupCohomology.dickson, that did work in the past:

sage: cython("""
....: from sage.structure.richcmp import richcmp, op_LT, op_NE, op_GT
....: from sage.all import GF
....: class DICKSON:
....:     def __init__(self,p):
....:         self.K = GF(p)
....:     def __richcmp__(self, other, op):
....:         print("Comparison called")
....:         return richcmp(self.K, other.K, op)
....: """)

Now, rich comparison fails both in py2 and py3:

sage: D = DICKSON(5)
sage: F = loads(dumps(D))
sage: D == F
False
sage: F.K == D.K
True

What has changed, how to fix it?

comment:72 follow-up: Changed 5 months ago by SimonKing

Aha! I see in sage.misc.unknown that a decorator @richcmp_method is used. So, I guess I need to use it, too.

comment:73 in reply to: ↑ 72 Changed 5 months ago by SimonKing

Replying to SimonKing:

Aha! I see in sage.misc.unknown that a decorator @richcmp_method is used. So, I guess I need to use it, too.

Hooray:

sage: cython("""
....: from sage.structure.richcmp import richcmp, richcmp_method
....: from sage.all import GF
....: @richcmp_method
....: class DICKSON(object):
....:     def __init__(self,p):
....:         self.K = GF(p)
....:     def __richcmp__(self, other, op):
....:         print("Comparison called")
....:         return richcmp(self.K, other.K, op)
....: """)
sage: D = DICKSON(5)
sage: F = loads(dumps(D))
sage: D == F
Comparison called
True

comment:74 follow-ups: Changed 5 months ago by SimonKing

At the moment, tests in dickson, auxiliaries, resolution and factory pass both in Py2 and Py3. In isomorphism_test, there are currently two failing tests, and it seems to me that they reveal a Py3-bug in cartesian_product_iterator:

File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 353, in pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms
Failed example:
    T.explore_isomorphisms()
Exception raised:
    Traceback (most recent call last):
      File "sage/misc/cachefunc.pyx", line 1944, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10139)
        return cache[k]
    KeyError: (((3, 5, 4, 6, 7, 1, 2), 0, True), ())

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/mrange.py", line 139, in _xmrange_iter
        curr_elt = [next(i) for i in curr_iters[:-1]] + [None]
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/mrange.py", line 139, in <listcomp>
        curr_elt = [next(i) for i in curr_iters[:-1]] + [None]
    StopIteration

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms[10]>", line 1, in <module>
        T.explore_isomorphisms()
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/isomorphism_test.py", line 402, in explore_isomorphisms
        Cands = self.candidates_of_gens(tuple([p[4] for p in Sizes]),len(self.rigid_generators))
      File "sage/misc/cachefunc.pyx", line 1949, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10273)
        w = self._instance_call(*args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1825, in sage.misc.cachefunc.CachedMethodCaller._instance_call (build/cythonized/sage/misc/cachefunc.c:9758)
        return self.f(self._instance, *args, **kwds)
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/isomorphism_test.py", line 644, in candidates_of_gens
        for Ims1,Ims2 in cartesian_product_iterator([FirstHalf, SecondHalf]):
    RuntimeError: generator raised StopIteration

I will investigate which values of FirstHalf and SecondHalf trigger it.

comment:75 in reply to: ↑ 74 Changed 5 months ago by SimonKing

Replying to SimonKing:

I will investigate which values of FirstHalf and SecondHalf trigger it.

See #28521

EDIT: I am not sure if I'll make #28521 a dependency or simply work around the bug.

Last edited 5 months ago by SimonKing (previous) (diff)

comment:76 in reply to: ↑ 74 Changed 5 months ago by SimonKing

Replying to SimonKing:

At the moment, tests in dickson, auxiliaries, resolution and factory pass both in Py2 and Py3.

When working around the cartesian_product_iterator bug, the tests in isomorphism_test pass both in Py2 and Py3. Yay!

Last edited 5 months ago by SimonKing (previous) (diff)

comment:77 Changed 5 months ago by SimonKing

Two errors in Py3 concern __div__, namely

Running doctests with ID 2019-09-21-07-26-57-20b48344.
Using --optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage
Doctesting 1 file.
sage -t --long --warn-long 47.4 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx", line 3131, in pGroupCohomology.cochain.MODCOCH.__div__
Failed example:
    print(H.1/3) #indirect doctest
Expected:
    (a_6_0)/(3): 6-Cocycle in H^*(SmallGroup(400,206); GF(5))
    defined by
    2*c_2_1*c_2_2*a_1_0*a_1_1
Got:
    (a_6_0)*(2): 6-Cocycle in H^*(SmallGroup(400,206); GF(5))
    defined by
    2*c_2_1*c_2_2*a_1_0*a_1_1
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx", line 3139, in pGroupCohomology.cochain.MODCOCH.__div__
Failed example:
    H.2/H.1
Expected:
    Traceback (most recent call last):
    ...
    TypeError: Can not divide by <class 'pGroupCohomology.cochain.MODCOCH'>
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.cochain.MODCOCH.__div__[6]>", line 1, in <module>
        H.gen(2)/H.gen(1)
      File "sage/structure/element.pyx", line 1718, in sage.structure.element.Element.__truediv__ (build/cythonized/sage/structure/element.c:12780)
        return (<Element>left)._div_(right)
      File "sage/structure/element.pyx", line 2654, in sage.structure.element.RingElement._div_ (build/cythonized/sage/structure/element.c:18184)
        frac = self._parent.fraction_field()
    TypeError: 'NoneType' object is not callable
**********************************************************************
1 item had failures:
   2 of   8 in pGroupCohomology.cochain.MODCOCH.__div__
    [1490 tests, 2 failures, 465.54 s]
----------------------------------------------------------------------
sage -t --long --warn-long 47.4 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx  # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 470.6 seconds
    cpu time: 249.3 seconds
    cumulative wall time: 465.5 seconds

The first error seems to indicate that H.1/3 is automatically translated to H.1*2, which does make sense modulo 5, but which I did not implement like this. And the second error is nothing that I implemented.

In fact I overrode __div__ for COCH and MODCOCH. Is it perhaps the case that overriding __div__ won't work in Py3?

comment:78 follow-up: Changed 5 months ago by chapoton

In py3, there is __truediv__ and __floordiv__

comment:79 in reply to: ↑ 78 ; follow-up: Changed 5 months ago by SimonKing

Replying to chapoton:

In py3, there is __truediv__ and __floordiv__

I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement __truediv__, __floordiv__ and __div__?

On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.

So, given a RingElement, what methods do I need to implement for providing division by an element of the base field, and what do I need to implement for making clear that division of two ring elements doesn't work?

I guess I should implement a fraction_field method for cohomology rings. Of course, since cohomology rings can have zero divisors, a fraction field will not always exist. So, should in that situation fraction_field return NotImplemented, or raise a ValueError, or what else?

Do I need to provide a single-underscore _div_ method, in addition to the fraction_field?

How to implement division by a base field element (I guess _div_ is for division of two ring elements)?

comment:80 in reply to: ↑ 79 Changed 5 months ago by SimonKing

Replying to SimonKing:

Replying to chapoton:

In py3, there is __truediv__ and __floordiv__

I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement __truediv__, __floordiv__ and __div__?

On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.

I think I'll proceed as follows:

  • Implement a fraction_field method for cohomology rings raising a TypeError (note that ZZ.quo(6).fraction_field() raises a TypeError, too)
  • Do NOT implement _div_ -- the attempt to divide two elements of a cohomology ring will result in the TypeError raised by the fraction_field() method
  • Division of a cohomology element by an element of the base field will automatically work, by multiplying the inverse of the field element with the cohomology element, using _lmul_/_rmul_. I should add an example/test in the documentation of the _lmul_/_rmul_ methods

comment:81 in reply to: ↑ 68 Changed 5 months ago by SimonKing

Replying to SimonKing:

I don't think that __nonzero__ does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current not-yet-published code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for __nonzero__ and for __bool__, but only one of them would be compiled.

This did work, to some extent. But not really reliably. Therefore, to be on the safe side, I will implement both methods in both python versions, since the code duplication is only small.

comment:82 Changed 5 months ago by SimonKing

I am currently dealing with different but mathematically equivalent behaviour: Apparently, previously Singular did not always do tail reduction, i.e., some rings in the database do not have a fully reduced representation of the relation ideal. For example, here are relations for the cohomology of SmallGroup(64,14):

sage: H.rels()   # not tail reduced, result obtained with py2
['a_1_0^2',
 'a_1_0*a_1_1',
 'a_1_1^3',
 'a_2_1*a_1_0',
 'a_2_1^2+a_2_1*a_1_1^2',
 'a_1_1*a_3_3+a_2_1^2',
 'a_1_0*a_3_3+a_2_1^2',
 'a_2_1*a_3_3',
 'a_3_3^2']

versus

sage: H.rels()   # tail reduced, obtained with Py3
['a_1_0^2',
 'a_1_0*a_1_1',
 'a_1_1^3',
 'a_2_1*a_1_0',
 'a_2_1^2+a_2_1*a_1_1^2',
 'a_1_0*a_3_3+a_2_1*a_1_1^2',
 'a_1_1*a_3_3+a_2_1*a_1_1^2',
 'a_2_1*a_3_3',
 'a_3_3^2']

As you can see, the relation a_2_1^2+a_2_1*a_1_1^2 could be used to reduce the tail of a_1_1*a_3_3+a_2_1^2. Also, the sorting of the relations is different.

The question is how to deal with it. I guess I could proceed as follows:

sage: if (2, 8) < sys.version_info:
....:     expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0',
....:           'a_2_1^2+a_2_1*a_1_1^2', 'a_1_0*a_3_3+a_2_1*a_1_1^2',
....:           'a_1_1*a_3_3+a_2_1*a_1_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
....: else:
....:     expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0',
....:           'a_2_1^2+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1^2',
....:           'a_1_0*a_3_3+a_2_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
....:     
sage: H.rels() == expected_rels
True

comment:83 Changed 5 months ago by SimonKing

Sigh. I am suffering from "interesting" debugging sessions. At the moment, it seems to me that some errors in the py2-version only occur, when I first install p_group_cohomology in the py2-install of Sage, then install p_group_cohomology in the py3-install of Sage, and only then do tests in the py2-install.

comment:84 follow-up: Changed 5 months ago by dimpase

I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.

comment:85 in reply to: ↑ 84 ; follow-up: Changed 5 months ago by SimonKing

Replying to dimpase:

I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.

You mean by testing the p_group_cohomology package, or something else?

Indeed I had the same impression concerning ~/.sage/. Fortunately most of the trouble could be solved by putting str() around some data (because I got unicode where a string was expected - so, perhaps a rogue py3-pickle got misinterpreted in py2).

comment:86 in reply to: ↑ 85 Changed 5 months ago by dimpase

Replying to SimonKing:

Replying to dimpase:

I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.

You mean by testing the p_group_cohomology package, or something else?

Indeed I had the same impression concerning ~/.sage/. Fortunately most of the trouble could be solved by putting str() around some data (because I got unicode where a string was expected - so, perhaps a rogue py3-pickle got misinterpreted in py2).

I was getting different output from a test, IIRC it was releated to unicode, and the problem went away after I removed ~/.sage/. I thought it might have been related to ipython, but who knowns.

comment:87 Changed 5 months ago by SimonKing

I tried to solve it by doing an explicit conversion to str whenever there is data that is expected of this type. And instead of if isinstance(bla, str), I now do if isinstance(bla, (str, unicode)), where I define unicode=str in py3.

So far, this seems to work. And after all, most users will have only one Sage installation anyway.

comment:88 in reply to: ↑ 13 Changed 5 months ago by SimonKing

Replying to SimonKing:

Replying to SimonKing:

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

Hooray:

# in Sage-with-python-2
sage: P = CombinatorialFreeModule(ZZ,'x')
sage: save(P,'../../pickle')
# BOTH with python-2 and python-3
sage: B = load('../../pickle.sobj')
sage: B._default_filename
'/home/king/Sage/pickle.sobj'

So, the old trick should still work.

I wonder if I shouldn't improve the trick.

Currently, if the initialisation of an unpickled object can not be finished, it is the __getattr__ method that would trigger the completion of initialisation based on the value of self._default_filename that is set by Sage's loads() after unpickling the object and before returning it.

But what if I turn _default_filename into a property? I could make it so that its setter triggers the completion of initialisation, to the effect that the object returned by loads() will be valid. The trickery would be in the property setter, not in __getattr__.

comment:89 Changed 5 months ago by SimonKing

I am puzzled about the appearance of unicode. For example, consider these two lines in my code:

        root = str(kwds.get('root', COHO.workspace))
        coho_logger.debug("Group data are rooted at %r", None, root)

The logging appears in the following session (in Sage-with-py2, where I first installed the package in Sage-with-py3 and then in Sage-with-py2):

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: from pGroupCohomology.cohomology import COHO
sage: _ = COHO()
<module>:
    Group data are rooted at u'/home/king/.sage/temp/klap/6618/dir_T5Im2o/'

So, apparently the variable root in the code snipped above is a unicode, although I explicitly converted it to a str. Moreover:

sage: COHO.workspace
'/home/king/.sage/temp/klap/6618/dir_T5Im2o/'
sage: type(COHO.workspace)
<type 'str'>

In other words: There is an attribute COHO.workspace of type str, and when I convert it to a str then a unicode results. WTF???

comment:90 follow-up: Changed 5 months ago by dimpase

I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.

It might be that iPython installs some kind of display hook that sets str to convert to unicode. There is code in src/sage/repl/rich_output that might be doing this.

That is to say that unicode you see printed might be happening due to on the fly conversion, set up by DisplayManager.

Last edited 5 months ago by dimpase (previous) (diff)

comment:91 in reply to: ↑ 90 Changed 5 months ago by SimonKing

Replying to dimpase:

I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.

Is there unicode in py3?

The common topic is: In some cases I get a TypeError saying that a str was expected, but got unicode. And a log shows that a variable is of type unicode after an EXPLICIT(!) cast to str.

In all mentioned cases, I proceeded as follows:

  1. Install the package in Sage-with-py2
  2. Install the package in Sage-with-py3
  3. Test in Sage-with-py2

Sage-with-py2 and Sage-with-py3 are of course in different folders. What they share: The git repository (but different git worktrees), and DOT_SAGE.

But it is all very flaky. I just inserted a couple of print statements in my code to investigate more deeply, did the above double-install procedure --- and suddenly the type was str, not unicode.

It might be that iPython installs some kind of display hook that sets str to convert to unicode. There is code in src/sage/repl/rich_output that might be doing this.

It is not only the output but the type inside of the code. In some cases, I get a TypeError.

comment:92 Changed 5 months ago by dimpase

Do you see these errors while running doctests? I gather there is --nodotsage option you can use while running tests, so you may try to see whether you can reproduce this without DOT_SAGE.

Another thing to try is to wipe out DOT_SAGE each time you switch from pyX to py(5-X), and see whether this make the error to go away.

comment:93 Changed 5 months ago by dimpase

another way to work around this might be just to convert all your strings to unicode.

comment:94 Changed 5 months ago by SimonKing

I am afraid that it is currently not possible for anybody else to reproduce the following, because I didn't publish my code yet. But anyway, here is what I did:

  1. Insert type information, so that the mentioned code snippet now is
            root = str(kwds.get('root', COHO.workspace))
            coho_logger.debug("Group data are rooted at %r. Type: %r", None, root, type(root))
    
  2. Install it in py3
  3. Install it in py2
  4. In Py3, run the following:
    sage: from pGroupCohomology import CohomologyRing
    sage: CohomologyRing.doctest_setup()
    sage: H = CohomologyRing(64,14, from_scratch=True)
    sage: H.make()
    
  5. In Py2, run the following:
    sage: from pGroupCohomology import CohomologyRing
    sage: from pGroupCohomology.cohomology import COHO
    sage: CohomologyRing.doctest_setup()
    sage: CohomologyRing.global_options('debug')
    sage: _ = COHO()
    <module>:
        Group data are rooted at u'/home/king/.sage/temp/klap/14917/dir_20mizE/'. Type: <type 'unicode'>
    

So, the type REALLY is unicode, even though I did an EXPLICIT conversion to str.

Then, I continued:

  1. Move the old ~/.sage and create a new one.
  2. Start Sage-with-py2, and repeat the test:
    Setting permissions of DOT_SAGE directory so only you can read and write it.
    sage: from pGroupCohomology import CohomologyRing
    sage: from pGroupCohomology.cohomology import COHO
    sage: CohomologyRing.doctest_setup()
    sage: CohomologyRing.global_options('debug')
    sage: _ = COHO()
    <module>:
        Group data are rooted at u'/home/king/.sage/temp/klap/15112/dir_fYCkwo/'. Type: <type 'unicode'>
    sage: type(COHO.workspace)
    <type 'str'>
    
    

So, removing DOT_SAGE didn't help. For comparison, do the same in Py3:

sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
<module>:
    Group data are rooted at '/home/king/.sage/temp/klap/15493/dir_81mwix3u/'. Type: <class 'str'>

which of course is no surprise, as python-3 has no unicode.

Note that the line _ = COHO() does not involve taking an old pickle from a repository. It merely creates a new instance of COHO and initialises it only to the point where the current workspace is assigned to the attribute .root.

Summary: The class attribute COHO.workspace is of type str. The dictionary kwds is empty. str(kwds.get('root', COHO.workspace))) returns the unicode version of `COHO.workspace. I.e., converting a str to a str results in a unicode.

comment:95 Changed 5 months ago by SimonKing

Just to document that we are talking about a Heisenbug: I wanted to test whether it is perhaps the case that str==unicode. Therefore, I changed the code snippet to

        root = str(kwds.get('root', COHO.workspace))
        coho_logger.debug("Group data are rooted at %r. Type: %r, %r, %r", None, root, type(root), type(COHO.workspace), str)

, installed it in py2 (only), and rerun the test in py2:

sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
<module>:
    Group data are rooted at '/home/king/.sage/temp/klap/16014/dir_M5saF0/'. Type: <type 'str'>, <type 'str'>, <type 'str'>

Hence, no bug. "Hooray".

Start a sage-with-py3-session and do

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: H = CohomologyRing(64,14, from_scratch=True)
sage: H.make()

The hope was of course that it triggers something. But the test in py2 does not have a unicode problem. "Hooray".

But I am sure that the problem will pop up sooner or later.

comment:96 Changed 5 months ago by SimonKing

To make sure that this bug doesn't pop up again, I inserted a test whether the python version is less than (2,8) and root is of type unicode. In this case, I print some information and then raise a runtime error. I guess it's the best that I can do for now.

After recompiling and rerunning the above tests, they went fine again. But sooner or later it will return, and then the diagnostic tests I inserted will perhaps tell more clearly what is happening.

comment:97 Changed 5 months ago by SimonKing

Or another idea: Since it is Cython code, I could do cdef str root = ....

comment:98 Changed 5 months ago by SimonKing

Hooray!

The inserted test already showed something:

sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
Critical bug in python.
kwds.get('root', COHO.workspace) is <type 'str'>
COHO.workspace is <type 'str'>
str=<type 'unicode'>, unicode=<type 'unicode'>
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-a303ce041f27> in <module>()
----> 1 _ = COHO()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

RuntimeError: An explicit cast to <str> resulted in a unicode object

Aha! For some silly reason, we have str == unicode!!

EDIT: The above was printed like this:

        if (2, 8) > sys.version_info and isinstance(root, unicode):
            print("Critical bug in python.")
            print("kwds.get('root', COHO.workspace) is %r"%type(kwds.get('root', COHO.workspace)))
            print("COHO.workspace is %r"%type(COHO.workspace))
            print("str=%r, unicode=%r"%(str,unicode))
            raise RuntimeError("An explicit cast to <str> resulted in a unicode object")
Last edited 5 months ago by SimonKing (previous) (diff)

comment:99 Changed 5 months ago by SimonKing

How to cope with it? Shall I test upon importing of the module whether str == unicode in Python-2, and raise an error whose message tells to recompile?

comment:100 follow-up: Changed 5 months ago by dimpase

So, would this still pop up if right before running these tests you removed ~/.sage/ ?

comment:101 in reply to: ↑ 100 Changed 5 months ago by SimonKing

Replying to dimpase:

So, would this still pop up if right before running these tests you removed ~/.sage/ ?

See comment:94. There, it didn't help. Now, it doesn't either:

king@klap:~/Sage/git$ mv ~/.sage/ ~/.sageBAK
king@klap:~/Sage/git$ mkdir ~/.sage
king@klap:~/Sage/git$ sage/sage 
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta8, Release Date: 2019-08-25               │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Setting permissions of DOT_SAGE directory so only you can read and write it.
sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
Critical bug in python.
kwds.get('root', COHO.workspace) is <type 'str'>
COHO.workspace is <type 'str'>
str=<type 'unicode'>, unicode=<type 'unicode'>
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-a303ce041f27> in <module>()
----> 1 _ = COHO()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

RuntimeError: An explicit cast to <str> resulted in a unicode object

comment:102 Changed 5 months ago by SimonKing

I wonder if the problem occurs because I am not installing the development versions of my package from an actual "package" (using sage -i p_group_cohomology), but directly from the sources. I.e., in the src folder of my development tree, I do

sage -python setup.py build
sage -pip install . --upgrade

I know that sage -python setup.py build does create .c files in the src folder, but since the changed module is recompiled both in py2 and py3, I thought that it doesn't matter. And for efficiency, I want that only modules are recompiled that have changed.

So, I would like to test if the problem persists when separating the py2- and py3 build folders. When doing sage -python setup.py build, is it somehow possible to prescribe in what folder the .c and .pyc files are created? Then, I could choose separate folders for py2 and py3.

comment:103 Changed 5 months ago by SimonKing

I am now putting cythonize(..., build_dir=os.path.join("build","c_files-{}.{}".format(version_info.major, version_info.minor))) into my setup.py. In that way, the c-files will be created in separate directories for both Sage versions.

Let's see if this is enough to solve the problem!

comment:104 Changed 5 months ago by SimonKing

PS: Even if it solves the problem, I will test upon import whether str is unicode in py2, and if it is, raise a RuntimeError advising to recompile. I hope this error will never pop up...

comment:105 Changed 5 months ago by SimonKing

After changing setup.py so that the .c files will be in different folders for different python versions, tests in cohomology.pyx pass both with py2 and py3 and then py2 again. So, I can now turn to modular_cohomology.pyx and finally init.py (which contains a long doc string, namely the "general overview" of the package).

comment:106 Changed 5 months ago by SimonKing

Hooray! When I ran sage -tp 2 --force-lib --long --warn-long 60.1, all tests passed both in python 2 and python 3. I don't know how significant the run time differences are. In any case, here are the details:

Python-3
========
Running doctests with ID 2019-09-25-00-50-30-78efa273.
Using --optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage
Sorting sources by runtime so that slower doctests are run first....
Doctesting 13 files using 2 threads.
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
    [1481 tests, 350.33 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cohomology.pyx
    [1445 tests, 363.25 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
Warning, slow doctest:
    T.explore_isomorphisms()        # long time
Test ran for 122.79 s
    [213 tests, 221.05 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/modular_cohomology.pyx
    [459 tests, 274.13 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/factory.py
    [229 tests, 126.99 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/__init__.py
    [343 tests, 96.76 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/dickson.pyx
    [28 tests, 1.53 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/auxiliaries.py
    [39 tests, 0.79 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/barcode.py
    [143 tests, 10.31 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution_bindings.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pyx
    [882 tests, 22.95 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 765.7 seconds
    cpu time: 939.6 seconds
    cumulative wall time: 1468.1 seconds

Python-2
========
Running doctests with ID 2019-09-25-01-04-47-fa1de7f4.
Git branch: matrix_implementation_keyword
Using --optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,p_group_cohomology,python2,sage
Sorting sources by runtime so that slower doctests are run first....
Doctesting 13 files using 2 threads.
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
    [1481 tests, 437.44 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cohomology.pyx
    [1445 tests, 465.15 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
Warning, slow doctest:
    T.explore_isomorphisms()        # long time
Test ran for 116.79 s
    [213 tests, 229.85 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/modular_cohomology.pyx
    [459 tests, 312.40 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/factory.py
    [229 tests, 124.88 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/__init__.py
    [343 tests, 96.86 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pyx
    [882 tests, 21.38 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/dickson.pyx
    [28 tests, 1.63 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/auxiliaries.py
    [39 tests, 0.78 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution_bindings.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/barcode.py
    [143 tests, 14.90 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 873.5 seconds
    cpu time: 996.8 seconds
    cumulative wall time: 1705.3 seconds

Hence, tomorrow (i.e., in central European time, today in a few hours :-)) I will create a new package version and post it here for review.

comment:107 Changed 5 months ago by SimonKing

Something more to do: I think I should add a test showing what happens when moving the directory associated with a cohomology ring. The package can cope with moved data, based on some trickery in the attribute _default_filename that is defined by Sage's unpickler. I've turned _default_filename into a property (with getter, setter and deleter), and its documentation should contain the new test.

comment:108 Changed 5 months ago by SimonKing

  • Description modified (diff)

I'm afraid that another test failure popped up. So, I cannot post the new version yet.

comment:109 Changed 5 months ago by SimonKing

After fixing the failed test (in which I forgot to save something before loading it), I added a new feature to unit_test_64: Previously it was only tested whether the generator degrees and the Poincaré series of a freshly computed cohomology ring is consistent with what is stored in the database; now I test whether the new ring and the stored ring are isomorphic.

Indeed they are, for all groups of order 64. More precisely: In many cases the generators of the relation ideal are not identical. But in all cases, mapping the i-th generator of the new ring to the i-th generator of the old ring results in an isomorphism.

There is one remaining py3 issue, though: When I try to build the documentation, I get the error "No module named sphinx". The documentation does build, when I use py2. What can I do to fix it in py3?

EDIT: Is it not the case that sphinx is installed in Sage for building its documentation, both in py2 and py3? I am not sure if I ever tried to build the documentation of Sage-with-py3, though. Will sphinx be installed when I do a plain make in Sage (i.e., when I build the docs)?

Last edited 5 months ago by SimonKing (previous) (diff)

comment:110 follow-up: Changed 5 months ago by jhpalmieri

Sphinx should be installed, yes. Look for SAGE_ROOT/local/lib/python3.7/site-packages/sphinx/. If you run Sage, can you import sphinx?

Edit: this is for a Python 3 build of Sage.

Last edited 5 months ago by jhpalmieri (previous) (diff)

comment:111 in reply to: ↑ 110 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

Sphinx should be installed, yes. Look for SAGE_ROOT/local/lib/python3.7/site-packages/sphinx/. If you run Sage, can you import sphinx?

It was a trivial mistake: In the doc's Makefile, I had SPHINXBUILD = python -msphinx, but since python means python-2 even in a py3-install of Sage, I have to use SPHINXBUILD = sage -python -msphinx

comment:112 Changed 5 months ago by SimonKing

  • Branch set to u/SimonKing/p_group_cohomology_in_py_3

comment:113 Changed 5 months ago by SimonKing

  • Authors set to Simon King
  • Commit set to 463806c6cebb3120e20df0f19a216dd529553741
  • Description modified (diff)
  • Status changed from new to needs_review

I checked that the package builds and passes tests in Sage-with-py3. I checked that when running the tests directly on the sources (not during package installation) the tests pass in Sage-with-py2. I recommend that whoever reviews this tries the package both in py2 and in py3.


New commits:

2e0d11ePass unpickling options to pickle.load, default encoding 'latin1'. Accept both str and bytes in mtx_unpickle
8c105cdMake str_to_bytes/bytes_to_str accept both str and bytes input.
c8a0748Add tests for #28444
89ac77aFix keyword for py3-only test
3693024Fix doc strings in sage.misc.persist
f0828eeAdd a comment regarding the expected data type for an unpickle helper
ba41ebeFix two typos in a comment
463806cUpgrade p_group_cohomology to version 3.3

comment:114 follow-up: Changed 5 months ago by SimonKing

Hooray! In Py2, sage -f -c p_group_cohomology results in

...
[p_group_cohomology-3.3] Running the test suite for p_group_cohomology-3.3...
[p_group_cohomology-3.3] make[2]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1'
[p_group_cohomology-3.3] Making check in src
[p_group_cohomology-3.3] make[3]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] make  mnttest.sh
[p_group_cohomology-3.3] make[4]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] test -r ./test.reg || cp ./test.reg .
[p_group_cohomology-3.3] make[4]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] make  check-TESTS
[p_group_cohomology-3.3] make[4]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] make[5]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] PASS: mnttest.sh
[p_group_cohomology-3.3] ============================================================================
[p_group_cohomology-3.3] Testsuite summary for modular_resolution 1.1
[p_group_cohomology-3.3] ============================================================================
[p_group_cohomology-3.3] # TOTAL: 1
[p_group_cohomology-3.3] # PASS:  1
[p_group_cohomology-3.3] # SKIP:  0
[p_group_cohomology-3.3] # XFAIL: 0
[p_group_cohomology-3.3] # FAIL:  0
[p_group_cohomology-3.3] # XPASS: 0
[p_group_cohomology-3.3] # ERROR: 0
[p_group_cohomology-3.3] ============================================================================
[p_group_cohomology-3.3] make[5]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] make[4]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] make[3]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src'
[p_group_cohomology-3.3] make[3]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1'
[p_group_cohomology-3.3] make[3]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1'
[p_group_cohomology-3.3] make[2]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1'
[p_group_cohomology-3.3] Running doctests with ID 2019-09-26-11-19-11-b01ece2b.
[p_group_cohomology-3.3] Git branch: t/28414/p_group_cohomology_in_py_3
[p_group_cohomology-3.3] Using --optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,python2,sage
[p_group_cohomology-3.3] Sorting sources by runtime so that slower doctests are run first....
[p_group_cohomology-3.3] Doctesting 13 files using 3 threads.
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/modular_cohomology.pyx
[p_group_cohomology-3.3]     [459 tests, 456.02 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/cochain.pxd
[p_group_cohomology-3.3]     [0 tests, 0.00 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/cochain.pyx
[p_group_cohomology-3.3]     [1481 tests, 599.35 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/cohomology.pyx
[p_group_cohomology-3.3]     [1465 tests, 606.86 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/isomorphism_test.py
[p_group_cohomology-3.3] **********************************************************************
[p_group_cohomology-3.3] File "pGroupCohomology-3.3/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
[p_group_cohomology-3.3] Warning, slow doctest:
[p_group_cohomology-3.3]     T.explore_isomorphisms()        # long time
[p_group_cohomology-3.3] Test ran for 161.47 s
[p_group_cohomology-3.3]     [213 tests, 307.72 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/__init__.py
[p_group_cohomology-3.3]     [343 tests, 148.33 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/factory.py
[p_group_cohomology-3.3] **********************************************************************
[p_group_cohomology-3.3] File "pGroupCohomology-3.3/pGroupCohomology/factory.py", line 133, in pGroupCohomology.factory.unit_test_64
[p_group_cohomology-3.3] Warning, slow doctest:
[p_group_cohomology-3.3]     L, t = unit_test_64(from_scratch=False)    # long time
[p_group_cohomology-3.3] Test ran for 99.46 s
[p_group_cohomology-3.3]     [229 tests, 209.64 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/resolution.pxd
[p_group_cohomology-3.3]     [0 tests, 0.00 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/resolution.pyx
[p_group_cohomology-3.3]     [882 tests, 26.85 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/dickson.pyx
[p_group_cohomology-3.3]     [28 tests, 1.41 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/barcode.py
[p_group_cohomology-3.3]     [143 tests, 21.04 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/resolution_bindings.pxd
[p_group_cohomology-3.3]     [0 tests, 0.00 s]
[p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/auxiliaries.py
[p_group_cohomology-3.3]     [39 tests, 0.92 s]
[p_group_cohomology-3.3] ----------------------------------------------------------------------
[p_group_cohomology-3.3] All tests passed!
[p_group_cohomology-3.3] ----------------------------------------------------------------------
[p_group_cohomology-3.3] Total time for all tests: 815.2 seconds
[p_group_cohomology-3.3]     cpu time: 1261.3 seconds
[p_group_cohomology-3.3]     cumulative wall time: 2378.1 seconds
[p_group_cohomology-3.3] 
[p_group_cohomology-3.3] real	13m52.051s
[p_group_cohomology-3.3] user	22m48.984s
[p_group_cohomology-3.3] sys	4m1.098s
[p_group_cohomology-3.3] Successfully installed p_group_cohomology-3.3
[p_group_cohomology-3.3] Deleting temporary build directory
[p_group_cohomology-3.3] /home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3
[p_group_cohomology-3.3] Finished installing p_group_cohomology-3.3.spkg
make[1]: Leaving directory '/home/king/Sage/git/sage/build/make'

real	27m19.468s
user	30m7.398s
sys	4m10.930s
Sage build/upgrade complete!

In Py3, I had more warnings about slow tests (six instead of two, if I recall correctly), but perhaps my laptop (or its harddisk) was busy with other things.

Now I am running unit_test_64 in Python-2. Note that in Python-3 I got about 20 (or even more) examples where the freshly computed relation ideal was presented by a different set of generators than what was stored in the database. But, as I have mentioned in comment:82, this seems to be Singular not doing all tail reduction in py2.

comment:115 in reply to: ↑ 114 Changed 5 months ago by SimonKing

Replying to SimonKing:

Now I am running unit_test_64 in Python-2. Note that in Python-3 I got about 20 (or even more) examples where the freshly computed relation ideal was presented by a different set of generators than what was stored in the database.

Just for the record: In the py2 version, the freshly computed results were identical (not only equal up to generator choice) to the stored results.

comment:116 Changed 5 months ago by SimonKing

  • Description modified (diff)

By the way, at the recent Sage Days in Bonn, I learnt how to host the package repository at github and use Travis CI to some extent. I have posted the link to the ticket description.

Note, however, that testing it on Travis CI with python-3 will be difficult: I was told that the current Sage-with-py3 docker image is rather outdated, and also I don't know how I should write my .travis.yml so that the automatic tests involve both py2 and py3. The answer on sage-devel didn't help me much. Is someone reading this who can tell me how I should modify .travis.yml?

comment:117 follow-up: Changed 5 months ago by jhpalmieri

Suppose I delete .sage/pGroupCohomology/, to force the package to reinstall those files. Then

  • if I build with Python 2 first and then Python 3, all tests pass.
  • if I build with Python 3 first and then Python 2, I get failures:
    [p_group_cohomology-3.3]     IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/4gp2/H4gp2.sobj are not readable: expected bytes, unicode found
    

I guess I have a few questions:

  • is it possible (and not too much work to implement) for the Python 3 version to build .sobj files which are readable by Python 2?
  • if not, should the package discard the existing .sobj file and rebuild in this case?
  • more generally, if there is any error reading the .sobj file — maybe somehow the file got corrupted — should it just rebuild?

This particular problem should not happen much: people won't be switching from Python 3 to Python 2 very much. So don't work too hard to get the Python 3 version to produce .sobj files which are readable by Python 2.

Last edited 5 months ago by jhpalmieri (previous) (diff)

comment:118 in reply to: ↑ 117 ; follow-up: Changed 5 months ago by SimonKing

Replying to jhpalmieri:

Suppose I delete .sage/pGroupCohomology/, to force the package to reinstall those files. Then

  • if I build with Python 2 first and then Python 3, all tests pass.
  • if I build with Python 3 first and then Python 2, I get failures:
    [p_group_cohomology-3.3]     IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/4gp2/H4gp2.sobj are not readable: expected bytes, unicode found
    

It was a good idea to do such tests. And I am surprised about the error message: There is no use of bytes in the whole cohomology package. Perhaps the error is about unpickling a MeatAxe matrix (sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense)? But this was supposed to be fixed in #28444.

I guess I have a few questions:

  • is it possible (and not too much work to implement) for the Python 3 version to build .sobj files which are readable by Python 2?

I thought they were.

  • if not, should the package discard the existing .sobj file and rebuild in this case?
  • more generally, if there is any error reading the .sobj file — maybe somehow the file got corrupted — should it just rebuild?

It is already the case that if some data source provides data in legacy format, it will automatically be overwritten by a new pickle.

This particular problem should not happen much: people won't be switching from Python 3 to Python 2 very much. So don't work too hard to get the Python 3 version to produce .sobj files which are readable by Python 2.

But it should still work and needs to be investigated.

I'll follow your instructions (remove .sage/pGroupCOhomology, install (and also test?) with Python 3 and then install with Python 2 and see what is happening.

comment:119 in reply to: ↑ 118 Changed 5 months ago by jhpalmieri

Replying to SimonKing:

I'll follow your instructions (remove .sage/pGroupCOhomology, install (and also test?) with Python 3 and then install with Python 2 and see what is happening.

I installed and tested when I did this, but just now I hit the same error with:

  • delete `.sage/pGroupCohomology'
  • in Python 3, install the package
  • then run Sage:
    sage: from pGroupCohomology import CohomologyRing
    sage: CohomologyRing(16, 2)
    H^*(SmallGroup(16,2); GF(2))
    
  • in Python 2, install the package
  • then run Sage:
    sage: from pGroupCohomology import CohomologyRing
    sage: CohomologyRing(16, 2)
    ---------------------------------------------------------------------------
    IOError                                   Traceback (most recent call last)
    <ipython-input-2-e1924d996e12> in <module>()
    ----> 1 CohomologyRing(Integer(16), Integer(2))
    
    /Users/jpalmier/Desktop/Sage/sage_builds/py2/sage-8.9.rc1/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds)
       1778             if q < 128:
       1779                 extras['websource'] = False
    -> 1780             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
       1781             return OUT
       1782 
    
    /Users/jpalmier/Desktop/Sage/sage_builds/py2/sage-8.9.rc1/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
       1209                 if '@use_this_root@' in coho_options:
       1210                     del coho_options['@use_this_root@']
    -> 1211                 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_workspace,file_name), msg))
       1212         ## 3. Link with local sources and load from there
       1213         elif root_local_sources and os.access(os.path.join(root_local_sources,file_name), os.R_OK) and not from_scratch:
    
    IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj are not readable: NEWOBJ class argument isn't a type object
    

comment:120 follow-up: Changed 5 months ago by SimonKing

Probably the following would work, too (so that it is not always needed to reinstall):

  • Have the package both in py2 and py3
  • Create a permanent workspace and store a cohomology ring created with py3
  • Try to load the cohomology ring from that permanent workspace with py2

Anyway, after doing the py3 test suit, I can confirm what you reported: The stuff in .sage/pGroupCohomology/db/16gp2 cannot be unpickled with py2. In particular, the state descriptor cannot be loaded:

sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-2796f9b01dc5> in <module>()
----> 1 S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2845)()
    157 
    158     ## Load file by absolute filename
--> 159     with open(filename, 'rb') as fobj:
    160         X = loads(fobj.read(), compress=compress, **kwargs)
    161     try:

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2795)()
    158     ## Load file by absolute filename
    159     with open(filename, 'rb') as fobj:
--> 160         X = loads(fobj.read(), compress=compress, **kwargs)
    161     try:
    162         X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:6929)()
   1043 
   1044     unpickler = SageUnpickler(io.BytesIO(s), **kwargs)
-> 1045     return unpickler.load()
   1046 
   1047 

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)()
    655 
    656         def load(self):
--> 657             return self._unpickler.load()
    658 
    659         def noload(self):

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/structure/factory.pyx in sage.structure.factory.lookup_global (build/cythonized/sage/structure/factory.c:4615)()
    748         Integer Ring
    749     """
--> 750     name = bytes_to_str(name, encoding='ASCII')
    751     try:
    752         return factory_unpickles[name]

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/structure/factory.c:5038)()
     55         return b
     56     if type(b) is not bytes:
---> 57         raise TypeError(f"expected bytes, {type(b).__name__} found")
     58 
     59     if PY_MAJOR_VERSION <= 2:

TypeError: expected bytes, unicode found

And I guess this can be fixed in bytes_to_str: After the changes in #28444, bytes_to_str is more tolerant about the input type (it may be str), but it should be even more so. It should (in py2) accept unicode.

comment:121 follow-up: Changed 5 months ago by dimpase

I think it's good enough if py2-created pickles work with py3.

Require the opposite compatibility seems to be an overkill.

comment:122 in reply to: ↑ 121 Changed 5 months ago by SimonKing

Replying to dimpase:

I think it's good enough if py2-created pickles work with py3.

Require the opposite compatibility seems to be an overkill.

I believe the fix (in #28540) would be relatively easy.

If you think that we shouldn't worry so much about reading py3-pickles in py2, then of course the group cohomology package can be reviewed as it is now. If I am right about #28640, then #28540 will fix the unpickling of py3-cohomology in py2 without any change in the package.

comment:123 in reply to: ↑ 120 ; follow-up: Changed 5 months ago by nbruin

Replying to SimonKing:

And I guess this can be fixed in bytes_to_str: After the changes in #28444, bytes_to_str is more tolerant about the input type (it may be str), but it should be even more so. It should (in py2) accept unicode.

I think that at this point it's becoming pretty clear you should use a routine that is not named bytes_to_str, but a routine "ensure_str", which in py3 accepts unicode=str and return the result unchanged, and also accepts py2's "bytes=str" objects and assumes it's "latin1" encoded, because that's what it would have been as a "str" in py2.

You are clearly NOT converting a "bytes" object to a "str" here; you are ensuring something is a "str", doing some conversions if necessary. You should use a routine that reflects that in its name if you need specific behaviour.

The reason why I care is because you are changing routines in sage.cpython.string, which people will expect to contain widely valid; well-defined routines. Stretching them up with hacks for specific purposes will lead to silent erroneous behaviour further down the line, when these routines are used in other contexts because they sound like the right tool (but aren't).

If it gets a py3 "unicode" object in py2, ideally it could just leave it as unicode, because that should be valid in str contexts in py2 for the most time as well. But we may be far from the ideal in py2. A safe choice would be str(<unicode>) to ensure you actually get a py2 str. That uses "ascii" encoding, so it may fail. You could allow for "UTF-8" but that would not round-trip correctly with the "latin1" on the other side. Using "latin1" for encoding unicode-to-str on py2 may also fail, because not all unicode code points have an encoding in "latin1".

comment:124 in reply to: ↑ 123 Changed 5 months ago by SimonKing

Replying to nbruin:

The reason why I care is because you are changing routines in sage.cpython.string, which people will expect to contain widely valid; well-defined routines. Stretching them up with hacks for specific purposes will lead to silent erroneous behaviour further down the line, when these routines are used in other contexts because they sound like the right tool (but aren't).

Agreed. I have already changed the topic of #28540.

If it gets a py3 "unicode" object in py2, ideally it could just leave it as unicode, because that should be valid in str contexts in py2 for the most time as well. But we may be far from the ideal in py2. A safe choice would be str(<unicode>) to ensure you actually get a py2 str. That uses "ascii" encoding, so it may fail.

The application (and the place where #28540 should be changed) is sage.structure.factory.lookup_global. This function used to only accept str. By #28444, it also accepts bytes, which makes it possible to unpickle a py2-pickle in py3.

I suggest in #28540 that it also accepts a unicode. This makes it possible to unpickle a py3-pickle in py2.

Again, I won't make this a dependency for the cohomology package.

comment:125 Changed 5 months ago by SimonKing

#28540 brought an improvement:

sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(16,2,options='debug')
_get_p_group_from_cache_or_db:
    Data found at /home/king/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj
    Group data are rooted at '/home/king/.sage/pGroupCohomology/db'.
Loading a cohomology ring:
    The state descriptor of the to-be-unpickled ring is expected to be provided at u'/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj'
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-2-a62afc150e39> in <module>()
----> 1 H = CohomologyRing(Integer(16),Integer(2),options='debug')

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds)
   1778             if q < 128:
   1779                 extras['websource'] = False
-> 1780             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1781             return OUT
   1782 

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1209                 if '@use_this_root@' in coho_options:
   1210                     del coho_options['@use_this_root@']
-> 1211                 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_workspace,file_name), msg))
   1212         ## 3. Link with local sources and load from there
   1213         elif root_local_sources and os.access(os.path.join(root_local_sources,file_name), os.R_OK) and not from_scratch:

IOError: Saved data at /home/king/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj are not readable: NEWOBJ class argument isn't a type object

So, the previous error arising in lookup_global has gone.

I am afraid the new error looks more like an error that needs to be fixed in the package, not in the Sage library. We will see...

comment:126 Changed 5 months ago by SimonKing

Next step of debugging.

In Py2, where we still have py3-pickles in DOT_SAGE:

sage: from pGroupCohomology.cohomology import COHO
sage: H = COHO()
sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
sage: H.__setstate__(S)
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-4-9c79e2f6e0a6> in <module>()
----> 1 H.__setstate__(S)

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__setstate__()

IOError: Unable to read resolution saved at /home/king/Sage/git/sage/local/share/pGroupCohomology/16gp2/dat/R16gp2.sobj

The same in py3:

sage: from pGroupCohomology.cohomology import COHO
sage: H = COHO()
sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
sage: H.__setstate__(S)
sage: print(H)

Cohomology ring of Small Group number 2 of order 16 with coefficients in GF(2)

Computation complete
Minimal list of generators:
[c_2_1: 2-Cocycle in H^*(SmallGroup(16,2); GF(2)),
 c_2_2: 2-Cocycle in H^*(SmallGroup(16,2); GF(2)),
 c_1_0: 1-Cocycle in H^*(SmallGroup(16,2); GF(2)),
 c_1_1: 1-Cocycle in H^*(SmallGroup(16,2); GF(2))]
Minimal list of algebraic relations:
[c_1_0^2,
 c_1_1^2]

So, the error is now moved to __setstate__. I guess I'll fix it there.

comment:127 follow-up: Changed 5 months ago by SimonKing

Meanwhile I know at what point the error mentioned in comment:125 arises:

sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
<ipython-input-2-855d0a850e70> in <module>()
----> 1 si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/explain_pickle.pyc in unpickle_newobj(klass, args)
   2438         return pers[int(id)]
   2439 
-> 2440     return SageUnpickler.loads(pickle, persistent_load=pers_load)
   2441 
   2442 

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SageUnpickler.loads (build/cythonized/sage/misc/persist.c:6467)()
    962         """
    963 
--> 964         return cls(io.BytesIO(data), **kwargs).load()
    965 
    966 

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)()
    655 
    656         def load(self):
--> 657             return self._unpickler.load()
    658 
    659         def noload(self):

UnpicklingError: NEWOBJ class argument isn't a type object

But I do not have the faintest idea yet what this error means. For comparison, in py3 I get

sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
sage: si1
<pGroupCohomology.resolution.RESL_sparse_unpickle_class object at 0x7f2ee987abe0>

I'm guessing that a class in Python-3 is a type, but in Python-2 it generally isn't. Is that the meaning of the error? But isn't it the case that a class defined in Cython (here: resolution.pyx) automatically is a type?

Anyway, I'll try to turn class RESL_sparse_unpickle_class into class RESL_sparse_unpickle_class(object) and see if it helps.

comment:128 in reply to: ↑ 127 Changed 5 months ago by SimonKing

Replying to SimonKing:

Anyway, I'll try to turn class RESL_sparse_unpickle_class into class RESL_sparse_unpickle_class(object) and see if it helps.

Nope.

sage: from pGroupCohomology.resolution import RESL_sparse_unpickle_class
sage: RESL_sparse_unpickle_class
<class 'pGroupCohomology.resolution.RESL_sparse_unpickle_class'>
sage: isinstance(_, type)
True

but still

sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
Traceback (most recent call last):
...
UnpicklingError: NEWOBJ class argument isn't a type object

comment:129 Changed 5 months ago by SimonKing

Odd. After trying again, the error is gone. In Py2:

sage: R = load("/home/king/.sage/pGroupCohomology/db/16gp2/dat/R16gp2.sobj")
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-1-c703eea54ef8> in <module>()
----> 1 R = load("/home/king/.sage/pGroupCohomology/db/16gp2/dat/R16gp2.sobj")

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2845)()
    157 
    158     ## Load file by absolute filename
--> 159     with open(filename, 'rb') as fobj:
    160         X = loads(fobj.read(), compress=compress, **kwargs)
    161     try:

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2795)()
    158     ## Load file by absolute filename
    159     with open(filename, 'rb') as fobj:
--> 160         X = loads(fobj.read(), compress=compress, **kwargs)
    161     try:
    162         X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:6929)()
   1043 
   1044     unpickler = SageUnpickler(io.BytesIO(s), **kwargs)
-> 1045     return unpickler.load()
   1046 
   1047 

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)()
    655 
    656         def load(self):
--> 657             return self._unpickler.load()
    658 
    659         def noload(self):

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL_sparse_unpickle_class.__call__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__()

OSError: Bad file format in file pgroup.c (line 404)

So, now complain about NEWOBJ any more.

comment:130 Changed 5 months ago by SimonKing

Spoke too soon. Immediately after this different error (i.e., in the same session), I again got the "NEWOBJ" error.

comment:131 Changed 5 months ago by SimonKing

sage: from pGroupCohomology.resolution import RESL_sparse_unpickle_class
sage: pg_RESL_sparse_unpickle_class is RESL_sparse_unpickle_class
False

In a different session, I got the answer "True".

Anyway, the error about "NEWOBJ" comes from python. It seems that python decided to kill data persistency. I tried to work around to the extent that py2-pickles of cohomology rings seem to be legible in py3. But I think I will not work further on making py3-pickles legible in py2.

Another question: Why is the branch of this ticket shown red?

comment:132 follow-up: Changed 5 months ago by SimonKing

The oddity is: Sometimes pg_RESL_sparce_unpickle_class is RESL_sparse_unpickle_class and sometimes it isn't. When the error occurs, I get

sage: pg_RESL_sparse_unpickle_class
<class pGroupCohomology.auxiliaries.unpickle_old_resl at 0x7fb8d4f549a8>

It is not nice that unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class') sometimes returns something else. But perhaps it is worth trying to make this class inherit from object.

comment:133 in reply to: ↑ 132 Changed 5 months ago by SimonKing

Replying to SimonKing:

But perhaps it is worth trying to make this class inherit from object.

Nope, didn't help. As there doesn't seem to be an easy fix, I do not plan to work further on it.

Do I understand correctly that I have to rebase on the current develop branch?

comment:134 Changed 5 months ago by git

  • Commit changed from 463806c6cebb3120e20df0f19a216dd529553741 to 7ad6a4a35c9af94d1692a17e1b8855f49d60a38e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7ad6a4aUpgrade p_group_cohomology to version 3.3

comment:135 Changed 5 months ago by jhpalmieri

Maybe it would be simpler, if any error arises reading a pickle, to just delete and reproduce the pickle file. Compatibility going from Python 3 to Python 2 may be annoying to debug and to fix, and besides, there may be other errors that we haven't foreseen.

comment:136 Changed 5 months ago by jhpalmieri

Or as you say, to just not work on the Py3 -> Py2 conversion. That's fine with me.

comment:137 Changed 5 months ago by SimonKing

I am using unpickle_override for RESL_sparse_unpickle_class, but I am not sure if that's needed at all.

comment:138 Changed 5 months ago by SimonKing

Bummer. Apparently I have destroyed the git repository for my py3 installation. It says:

king@klap:~/Sage/git/py3$ git lg
fatal: your current branch 't/28444/fix_backwards_incompatibility_of_unpickling_in_python_3' does not have any commits yet

and ALL files in the Sage tree are untracked. How has that happened, and how to fix it?

comment:139 follow-up: Changed 5 months ago by dimpase

Hopefully John has a version of this branch just before you did that fateful forced push.

There could be copies left on patchbots, maybe LinuxMint?/18.2/x86_64/4.8.0-53-generic/pc78-math patchbot does have it...

comment:140 Changed 5 months ago by jhpalmieri

Unfortunately I don't, but there are the commits in comment:113 which are still accessible.

comment:141 Changed 5 months ago by dimpase

yes, indeed, these commits are available in the full "git fetch trac" - I have them locally, in case they get gc'd.

comment:142 in reply to: ↑ 139 Changed 5 months ago by SimonKing

Replying to dimpase:

Hopefully John has a version of this branch just before you did that fateful forced push.

  1. What's wrong with the forced push? IMHO, that has nothing to to with the problem whatsoever. The forced push is nothing more than a rebasing of the previously committed branch. After pushing it, I could still work.
  2. The branch is still present in my git repository. Perhaps I didn't make it clear. I have a single git repository with two worktrees, one for py2, one for py3. The py2 worktree is fine (AFAIC), in particular I can checkout any branch. However, it seems that I deleted the branch on which the py3 worktree was living, with git branch -d. As a result, all files in the py3 worktree are now untracked, as is shown by git status, and git log says that there has never been a commit. Again, I am talking about only one of my two worktrees.
  3. The deleted branch is the branch from #28444, not from here. That branch is merged in "develop", that's why I did git branch -d on it.

So, from my perspective, "solving the problem" means "allowing me to do git checkout <branchname> in the py3-worktree" -- this currently is refused, because git claims that I should first commit or stash the whole Sage library.

Last edited 5 months ago by SimonKing (previous) (diff)

comment:143 Changed 5 months ago by SimonKing

Would git checkout -f <branch_name> possibly help? Or would I possibly destroy even more?

comment:144 follow-up: Changed 5 months ago by jhpalmieri

You could try git stash and then git checkout BRANCH and then git stash COMMAND for some value of COMMAND: git stash pop? git stash apply? git stash branch NEW?

comment:145 in reply to: ↑ 144 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

You could try git stash and then git checkout BRANCH and then git stash COMMAND for some value of COMMAND: git stash pop? git stash apply? git stash branch NEW?

Certainly not. I do in fact not have any uncommitted changes! Instead, git believes that I have uncommitted changes (indeed it believes that the whole sage library is completely new), because git is convinced that I am on a branch with an empty commit history -- namely, it has allowed me to delete the branch on which I was checked out.

comment:146 follow-up: Changed 5 months ago by jhpalmieri

You could first also make a copy of the whole directory, in case git stash etc. screws things up even more. How much can you recover by doing diff -r /path/to/py2/worktree /path/to/py3/worktree, or maybe git diff /path/to/py2/worktree /path/to/py3/worktree?

comment:147 in reply to: ↑ 146 ; follow-up: Changed 5 months ago by SimonKing

Replying to jhpalmieri:

You could first also make a copy of the whole directory,

Hopefully the other worktree (that is still working!) is the whole directory.

in case git stash etc. screws things up even more. How much can you recover by doing diff -r /path/to/py2/worktree /path/to/py3/worktree, or maybe git diff /path/to/py2/worktree /path/to/py3/worktree?

That would be a mega-patch. It would be the diff of an empty folder and the complete Sage sources.

comment:148 in reply to: ↑ 147 ; follow-up: Changed 5 months ago by jhpalmieri

Replying to SimonKing:

Replying to jhpalmieri:

You could first also make a copy of the whole directory,

Hopefully the other worktree (that is still working!) is the whole directory.

in case git stash etc. screws things up even more. How much can you recover by doing diff -r /path/to/py2/worktree /path/to/py3/worktree, or maybe git diff /path/to/py2/worktree /path/to/py3/worktree?

That would be a mega-patch. It would be the diff of an empty folder and the complete Sage sources.

Why? diff -r ... should just compare the files, and the files exist in both places, regardless of whether they're tracked by git.

By the way, if the branch still exists, can you create a new workspace using that branch?

comment:149 in reply to: ↑ 148 Changed 5 months ago by SimonKing

Replying to jhpalmieri:

Why? diff -r ... should just compare the files, and the files exist in both places, regardless of whether they're tracked by git.

I see. Then the diff is empty.

By the way, if the branch still exists, can you create a new workspace using that branch?

That's my question. The branch for this ticket still exists, in fact it is the branch that is currently checked out in my py2 worktree. However, git won't allow me to do any normal (non-forced) operation on my py3 worktree, as the branch that was checked out there was deleted, thus, git assumes that my py3 worktree has no commit history and has a couple of thousands new files.

comment:150 Changed 5 months ago by jhpalmieri

So if the two worktrees aren't any different from the content point of view, maybe you should just create an entirely new worktree, in a new directory, using the existing branch, and delete the old one. It means you will have to rebuild Sage in the new worktree. I guess that's what you're trying to avoid?

Alternatively, you could rename the old directory (OLD -> TEMP), create a new worktree where the old one used to be (OLD). Then in the shell, but without telling git, you could delete that newly created directory and rename TEMP back to OLD. Then see what git says. I think git keeps track of its worktree information in the original folder, not in the newly created one. By the way, is there anything in .git/worktrees/? If so, you might need to remove that before you go through this procedure.

comment:151 follow-up: Changed 4 months ago by jhpalmieri

I'm getting a few doctest failures with a Python 3 build of Sage, version 9.0.beta0. All failures have the form

File "pGroupCohomology-3.3/pGroupCohomology/cohomology.pyx", line 4591, in pGroupCohomology.cohomology.COHO.__dir__
Failed example:
    [a for a in dir(H) if a.startswith('f')]    #indirect doctest
Expected:
    ['filter_degree_type',
     'filter_regular_gready_parameters',
     'filter_regular_parameters',
     'find_dickson',
     'find_dickson_in_subgroup',
     'find_small_last_parameter',
     'firstOdd',
     'fraction_field',
     'from_base_ring']
Got:
    ['filter_degree_type',
     'filter_regular_gready_parameters',
     'filter_regular_parameters',
     'find_dickson',
     'find_dickson_in_subgroup',
     'find_small_last_parameter',
     'firstOdd',
     'fraction_field',
     'free_module',
     'from_base_ring']

comment:152 Changed 4 months ago by jhpalmieri

Same with Python 2 and Sage 9.0.beta0.

comment:153 in reply to: ↑ 151 Changed 4 months ago by SimonKing

Replying to jhpalmieri:

     'free_module',

So, if I understand correctly, the problem is both in py2 and py3., and it is about an additional entry in the dir. That should be solvable...

comment:154 follow-up: Changed 4 months ago by jhpalmieri

Is that kind of doctest too fragile? How important is it? You could certainly change it to startswith('fi') or add 'free_module' to the list, but maybe there are more robust options. 'fraction_field' in dir(H)? I guess I don't know what it's supposed to be testing.

Last edited 4 months ago by jhpalmieri (previous) (diff)

comment:155 in reply to: ↑ 154 Changed 4 months ago by SimonKing

Replying to jhpalmieri:

Is that kind of doctest too fragile? How important is it? You could certainly change it to startswith('fi') or add 'free_module' to the list, but maybe there are more robust options. 'fraction_field' in dir(H)? I guess I don't know what it's supposed to be testing.

It is supposed to test introspection and autocompletion (which doesn't work in Py-3, if I understand correctly).

On the other hand, it is true that all methods in that list that do not start with "fi" are inherited and are thus easily changed. Perhaps it is best to insert an ellipsis.

comment:156 Changed 4 months ago by git

  • Commit changed from 7ad6a4a35c9af94d1692a17e1b8855f49d60a38e to c247f13fd6e6ae3d3feeef41ebba1a78346744da

Branch pushed to git repo; I updated commit sha1. New commits:

c247f13Fix some introspection tests

comment:157 Changed 4 months ago by SimonKing

I have updated the tarball and the checksum


New commits:

c247f13Fix some introspection tests

comment:158 Changed 4 months ago by SimonKing

Hooray, with some long tests:

$ ./sage -f -c p_group_cohomology
...
[p_group_cohomology-3.3] ----------------------------------------------------------------------
[p_group_cohomology-3.3] All tests passed!
[p_group_cohomology-3.3] ----------------------------------------------------------------------
...
[sagelib-8.9.rc1] real	0m18.161s
[sagelib-8.9.rc1] user	0m5.983s
[sagelib-8.9.rc1] sys	0m1.083s
make[2]: Leaving directory '/home/king/Sage/git/py3/build/make'
make[1]: Leaving directory '/home/king/Sage/git/py3/build/make'

real	0m18.438s
user	0m6.252s
sys	0m1.126s
Sage build/upgrade complete!

So, it seems to work!

Last edited 4 months ago by SimonKing (previous) (diff)

comment:159 Changed 4 months ago by jhpalmieri

I still get one failure:

File "pGroupCohomology-3.3/pGroupCohomology/cohomology.pyx", line 4814, in pGroupCohomology.cohomology.COHO.__getattr__
Failed example:
    tc.completions('H.f',globals())
Expected:
    ['H.filter_degree_type',
     'H.filter_regular_gready_parameters',
     'H.filter_regular_parameters',
     'H.find_dickson',
     'H.find_dickson_in_subgroup',
     'H.find_small_last_parameter',
     'H.firstOdd',
     'H.fraction_field',
     'H.from_base_ring']
Got:
    ['H.filter_degree_type',
     'H.filter_regular_gready_parameters',
     'H.filter_regular_parameters',
     'H.find_dickson',
     'H.find_dickson_in_subgroup',
     'H.find_small_last_parameter',
     'H.firstOdd',
     'H.fraction_field',
     'H.free_module',
     'H.from_base_ring']

comment:160 Changed 4 months ago by SimonKing

Too bad that those tests work for me, i.e., it is easy to miss one of those tests.

Trying again (in a few minutes...).

comment:161 Changed 4 months ago by git

  • Commit changed from c247f13fd6e6ae3d3feeef41ebba1a78346744da to 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0

Branch pushed to git repo; I updated commit sha1. New commits:

6c5dafcSilence another flaky doctest

comment:162 Changed 4 months ago by SimonKing

Should be fixed now.


New commits:

6c5dafcSilence another flaky doctest

New commits:

6c5dafcSilence another flaky doctest

comment:163 Changed 4 months ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review
All tests passed!

comment:164 Changed 4 months ago by chapoton

  • Milestone changed from sage-8.9 to sage-9.0

comment:165 Changed 4 months ago by vbraun

  • Branch changed from u/SimonKing/p_group_cohomology_in_py_3 to 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:166 Changed 3 months ago by dimpase

  • Commit 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 deleted

sdh_X scripts call sdh_die themselves, so doing sdh_X || sdh_die ... is equivalent to simply sdh_X.

The intention of calling sdh_die to produce a meaninful error message is commendable, but these after || just won't be called.

It's better to modify sdh_X to allow for custiom error messages, I suppose.

Note: See TracTickets for help on using tickets.