Opened 2 years ago

Last modified 2 years ago

#28414 closed defect

p_group_cohomology does not build on Python3 — at Version 108

Reported by: dimpase Owned by:
Priority: major Milestone: sage-9.0
Component: packages: optional Keywords: py3
Cc: SimonKing Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #28444 Stopgaps:

Status badges

Description (last modified by SimonKing)

The aim of this ticket is to provide a new version of the optional p_group_cohomology spkg so that it builds and passes tests both with python-2 and python-3.

Hopefully the version number 3.3 (two times three) is appropriate.

Change History (109)

comment:1 Changed 2 years ago by jhpalmieri

cmp is also not supported in Python 3, and for a dictionary d, you should do x in d rather than d.has_key(x).

Last edited 2 years ago by jhpalmieri (previous) (diff)

comment:2 Changed 2 years ago by jhpalmieri

Simon, I have made some progress in converting this to Python 3. Let me know and I can email you some diff files.

comment:3 Changed 2 years ago by SimonKing

Hi John, in fact Python-3-compatibility was one of the things that I considered for the next package version. So, if you already have a diff, I'd appreciate to see it!

comment:4 follow-ups: Changed 2 years ago by jhpalmieri

I just sent email to your Jena address. I can get the package to build, but it doesn't work yet. An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  #!/usr/bin/env sage-python23
WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py:623: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
8gp3: Files on disk have been moved - trying to get things right
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-9bd7fbf5ad54> in <module>()
----> 1 load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in __call__(self, result)
    244             self.start_displayhook()
    245             self.write_output_prompt()
--> 246             format_dict, md_dict = self.compute_format_data(result)
    247             self.update_user_ns(result)
    248             self.fill_exec_result(result)

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in compute_format_data(self, result)
    148 
    149         """
--> 150         return self.shell.display_formatter.format(result)
    151 
    152     # This can be set to True by the write_output_prompt method in a subclass

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/display/formatter.py in format(self, obj, include, exclude)
    198         """
    199         # First, use Sage rich output if there is any
--> 200         sage_format, sage_metadata = self.dm.displayhook(obj)
    201         assert PLAIN_TEXT in sage_format, 'plain text is always present'
    202         if not set(sage_format.keys()).issubset(self.default_mime()):

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in displayhook(self, obj)
    809             return
    810         self._backend.set_underscore_variable(obj)
--> 811         plain_text, rich_output = self._rich_output_formatter(obj, dict())
    812         return self._backend.displayhook(plain_text, rich_output)
    813 

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in _rich_output_formatter(self, obj, rich_repr_kwds)
    621         rich_output = None
    622         plain_text = None
--> 623         has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
    624         if has_rich_repr:
    625             rich_output = self._call_rich_repr(obj, rich_repr_kwds)

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getattr__()

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    149 
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)
    153     try:

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    150     ## Load file by absolute filename
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:
    154         X._default_filename = os.path.abspath(filename)

/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

comment:5 in reply to: ↑ 4 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

I just sent email to your Jena address. I can get the package to build, but it doesn't work yet.

Thank you very much, I received the patch.

An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

That probably is harmless. I.e., I should use "warning", and apparently in some unpickling method I use "str" where it should be "bytes".

comment:6 Changed 2 years ago by jhpalmieri

Yes, and I would like to track down the str vs. bytes issue. It would be nice if I could get basic functionality like CohomologyRing(8,3) to work.

comment:7 follow-up: Changed 2 years ago by jhpalmieri

Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.

comment:8 in reply to: ↑ 7 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.

I did change .warn( into .warning( everywhere (so, no need to include it in a diff).

Concerning str vs. byte: There is only one occurrence of cdef str, and this gets assigned to the output of <some singular object>.name(). Is that output really a string in python-3? If not, it should be changed to cdef bytes. However, so far the error in unpickling stuff looks mysterious to me, as the traceback doesn't mention where the problem actually occurs in my code.

comment:9 follow-up: Changed 2 years ago by jhpalmieri

Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?

comment:10 in reply to: ↑ 9 ; follow-up: Changed 2 years ago by SimonKing

Replying to jhpalmieri:

Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?

You mean Python 3 is severely backwards incompatible and cannot deal with old pickles? That would be a show stopper for Sage, I think.

Note that at some point in the past I inserted code to regenerate very old pickles and translate them automatically to a new format. However, I would certainly not like to regenerate several gigabytes of data in the web repository.

Note that the problem occurs together with a warning:

WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
...
8gp3: Files on disk have been moved - trying to get things right

Maybe that indicates where the problem comes from. So, let's recall how storing the modular cohomology ring of a prime power group is solved.

Very early in the package development I found that is is terribly slow to pack all data of a ring in a single file. It simply takes so long and requires so much memory (at least in some cases) that it is not feasible to automatically store intermediate results of a lengthy computation (which evidently is a reasonable thing to do).

My solution was to not store a ring in a single file but distribute the data in a folder. So, the actual .sobj pickle file only contains basic information but is not stand-alone. In particular, the p_group_cohomology spkg needs to know where the data folder is located, the .sobj file has to be in the data folder and also it needs write permission for otherwise it couldn't do a continuation of the cohomology ring's computation. What is stored in the web repository is a compressed tar file of the data folder (i.e., when the web repository is accessed, then the tar file is downloaded, unpacked, and then the cohomology ring is loaded from there).

By default, the data folder's location is composed from the location of the current workspace (which are global variables) and from the identifier of the group (which is stored in the .sobj file).

The warning mentioned above occurs when the data folder's default location cannot be accessed. And in that case, I do some trickery --- and perhaps that trickery doesn't work in Python 3? Namely:

  • When an object is obtained from reading a pickle file, then Python (at least Python 2) stores the original file location as an attribute of the object. However, that attribute is only available after successful unpickling.
  • Therefore, when initially the location of the data folder cannot be determined, an unitialised cohomology ring is returned.
  • As soon as the uninitialised cohomology ring is touched (I think constructing its repr is enough), initialisation is triggered. Namely, recall that the location of the .sobj file is stored as an attribute of the uninitialised ring and that the .sobj file is supposed to be located in the data folder -- voilà, you can now determine the data folder and finish initialisation of the cohomology ring.

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

comment:11 follow-up: Changed 2 years ago by SimonKing

Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?

comment:12 in reply to: ↑ 11 ; follow-up: Changed 2 years ago by SimonKing

Replying to SimonKing:

Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?

Here is what I found in the "internetz" concerning how to test if something is a string in a way that works in python 2 and 3:

import six

if isinstance(obj, six.string_types):
    print('obj is a string!')

So, I think I should fix this.

comment:13 in reply to: ↑ 10 ; follow-up: Changed 2 years ago by SimonKing

Replying to SimonKing:

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

Hooray:

# in Sage-with-python-2
sage: P = CombinatorialFreeModule(ZZ,'x')
sage: save(P,'../../pickle')
# BOTH with python-2 and python-3
sage: B = load('../../pickle.sobj')
sage: B._default_filename
'/home/king/Sage/pickle.sobj'

So, the old trick should still work.

comment:14 Changed 2 years ago by SimonKing

Currently, I get the following behaviour:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__()

TypeError: expected bytes, str found

and

sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3/dat/State.sobj'
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1154                 coho_options['@use_this_root@'] = root_workspace
-> 1155                 OUT = load(os.path.join(root_workspace,file_name)) # realpath here?
   1156                 if '@use_this_root@' in coho_options:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO_unpickle()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-4-927128be23f4> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
   1159                 if '@use_this_root@' in coho_options:
   1160                     del coho_options['@use_this_root@']
-> 1161                 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_local_sources,file_name), msg))
   1162         ## 4. Search web repository
   1163         elif kwds.get('websource')!=False and (not from_scratch):

OSError: Saved data at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj are not readable: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

So, that's something I can now try to debug.

comment:15 follow-up: Changed 2 years ago by chapoton

in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3

comment:16 in reply to: ↑ 15 Changed 2 years ago by SimonKing

Replying to chapoton:

in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3

Thank you! I don't know yet whether or not I'll need them, but certainly it is good to know.

comment:17 Changed 2 years ago by SimonKing

I found that this fails in RESL.__init__:

        tmprstem = os.path.join(res_folder,rstem)
        tmpgstem = os.path.join(gps_folder,gstem)
        self.Data = newResolWithGroupLoaded(tmprstem,tmpgstem,1)

Apparently tmprstem or tmpgstem are strings, but the C function newResolWithGroupLoaded expects char * --- is there no automatic conversion from py3-string to char*?

comment:18 follow-up: Changed 2 years ago by chapoton

Maybe try

self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)

..with no guarantee..

comment:19 in reply to: ↑ 18 Changed 2 years ago by SimonKing

Replying to chapoton:

Maybe try

self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)

..with no guarantee..

That is likely to work. Another potential solution would be to generally work with bytes. I.e., let self.gstem be a bytes and then do os.path.join(self.res_folder, b'A'+self.gstem+b'.sobj')) etc.

What would be the preferred way?

comment:20 follow-up: Changed 2 years ago by chapoton

beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes

EDIT:

indeed:

sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components
Last edited 2 years ago by chapoton (previous) (diff)

comment:21 in reply to: ↑ 20 Changed 2 years ago by SimonKing

Replying to chapoton:

beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes

EDIT:

indeed:

sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components

This only shows that it cannot mix both types. In fact, if both are bytes then things work:

sage: os.path.join(b'home', b'foo')
b'home/foo'

Probably it would be a long and tedious work to change everything to bytes. In principle I would be willing to do that work, but if there is a different solution involving less work and only a slight overhead, I'd prefer that.

Currently, using str_to_bytes when passing a string to a C function does seem to work.

comment:22 Changed 2 years ago by SimonKing

WTF? I just see that I have the following argument types of COCH.__init__:

def __init__(self, PARENT, int n, char *Nick, L, ydeg=None, rdeg=None, is_polyrep=False)

I suppose Nick shouldn't be char* but (if anything) str. Or better not type it at all.

comment:23 Changed 2 years ago by SimonKing

I think I should also fix the following compiler warning (there are several similar):

pGroupCohomology/cohomology.c: In function ‘__pyx_pf_16pGroupCohomology_10cohomology_12explore_one_parameter’:
pGroupCohomology/cohomology.c:17678:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (__pyx_t_7 = 0; __pyx_t_7 < __pyx_t_16; __pyx_t_7+=1) {

comment:24 Changed 2 years ago by SimonKing

I guesss getting the following error is a progress, although it seems a bit strange to me:

sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
> export action matrices
Unable to save basic ring setup
Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
    obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)
    obj.save(filename=filename, compress=compress, **kwargs)
  File "sage/structure/sage_object.pyx", line 424, in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)
    filename = _base_save(self, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
  File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
TypeError: can't pickle dict_items objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
    obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py", line 1254, in _get_p_group_from_scratch
    safe_save(OUT, OUT.autosave_name())
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
  File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
  File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
    save (obj, path)
  File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
    _base_save(obj, filename, compress=compress)
  File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
    with open(filename, 'wb') as fobj:
  File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
    fobj.write(_base_dumps(obj, compress=compress))
  File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
    gherkin = SagePickler.dumps(obj)
  File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
    pickler.dump(obj)
TypeError: can't pickle dict_items objects
Computing next term
> rk P_02 =   1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'dict_items' object has no attribute 'save'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/structure/sage_object.pyx in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)()
    423 
--> 424         filename = _base_save(self, filename, compress=compress)
    425 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     84         os.unlink(path)
---> 85     save (obj, path)
     86 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     84         os.unlink(path)
---> 85     save (obj, path)
     86 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()

TypeError: can't pickle dict_items objects

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
    239         try:
--> 240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):

AttributeError: 'dict_items' object has no attribute 'save'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.make()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.next()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()

pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
     83     if os.path.islink(path):
     84         os.unlink(path)
---> 85     save (obj, path)
     86 
     87 ###################################

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
    240             obj.save(filename=filename, compress=compress, **kwargs)
    241         except (AttributeError, RuntimeError, TypeError):
--> 242             _base_save(obj, filename, compress=compress)
    243     else:
    244         # Saving an object to an image file.

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
    177     filename = _normalize_filename(filename)
    178 
--> 179     with open(filename, 'wb') as fobj:
    180         fobj.write(_base_dumps(obj, compress=compress))
    181 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
    178 
    179     with open(filename, 'wb') as fobj:
--> 180         fobj.write(_base_dumps(obj, compress=compress))
    181 
    182     return filename

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
    255     """
    256 
--> 257     gherkin = SagePickler.dumps(obj)
    258 
    259     if compress:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
    834         buf = io.BytesIO()
    835         pickler = cls(buf, **kwargs)
--> 836         pickler.dump(obj)
    837         return buf.getvalue()
    838 

TypeError: can't pickle dict_items objects

What are dict_items, and why can't they be pickled?

comment:25 follow-up: Changed 2 years ago by chapoton

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

comment:26 in reply to: ↑ 25 ; follow-ups: Changed 2 years ago by SimonKing

Replying to chapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.

comment:27 follow-up: Changed 2 years ago by SimonKing

There is a considerable progress now (after putting list around .items()):

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
> export action matrices
Computing next term
> rk P_02 =   1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
Determine degree 2 standard monomials
We got 1 standard monomials
Monomial 0
> Candidate: c_1_0^2
> Express monomial as a Cochain
Compute c_1_0*c_1_0
Compose chain maps R_2 -> R_1 -> R_0
Lift with Urbild Groebner basis in degree 1
load Urbild Groebner basis
Decomposable cochain found

There is no new generator in degree 2
Summary: 0 relations and 0 generators in degree 2
Ring approximation computed out to degree 2!
Storing approximation data
Inserting SmallGroup(4,2) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/'
> export action matrices
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
   1707             if q < 128:
   1708                 extras['websource'] = False
-> 1709             OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
   1710             return OUT
   1711 

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
   1246                 OUT = COHO(gap(KEY[0]), **extras)
   1247         else:
-> 1248             OUT = COHO(KEY[0],KEY[1], **extras)
   1249         _gap_reset_random_seed()
   1250         try:

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

KeyError: 'autoliftElAb'

The good news is that for the first time cohomology rings were computed in py-3 (namely cohomology rings of elementary abelian groups).

comment:28 in reply to: ↑ 26 Changed 2 years ago by jhpalmieri

Replying to SimonKing:

Replying to chapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.

You could use a try ... except block to deal with the Python 2 case quickly. Sage will soon (I hope) default to Python 3, so I don't think this is worth it, though.

comment:29 in reply to: ↑ 27 Changed 2 years ago by SimonKing

Replying to SimonKing:

There is a considerable progress now (after putting list around .items()): ...

 KeyError: 'autoliftElAb'

Apparently I forgot to create a shallow copy in some places in which I temporarily stored the global options -- it doesn't work to recreate a dict from its items in Py-3, and that's why I got the key error above:

sage: D = {1:2}
sage: I = D.items()
sage: D.clear()
sage: D.update(I)
sage: D
{}

In Py-2, one would get

sage: D
{1: 2}

comment:30 Changed 2 years ago by SimonKing

Hooray!

I am now in a position to completely compute CohomologyRing(8,3) from scratch. Unfortunately reading old data doesn't work yet, but certainly it is a progress :-)

comment:31 in reply to: ↑ 12 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

import six

Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.

comment:32 in reply to: ↑ 26 ; follow-up: Changed 2 years ago by jdemeyer

Replying to SimonKing:

Replying to chapoton:

If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).

You may need to wrap with list(d.items()) for pickling.

Wouldn't that considerably slow things down, at least in Python-2?

Use list(d.iteritems()) in Cython.

comment:33 in reply to: ↑ 31 ; follow-up: Changed 2 years ago by SimonKing

Replying to jdemeyer:

Replying to SimonKing:

import six

Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.

I am not dealing with incompatibilities in that case. I import six in order to be able to safely test for string types. How to, if I don't use six?

comment:34 in reply to: ↑ 32 Changed 2 years ago by SimonKing

Replying to jdemeyer:

Use list(d.iteritems()) in Cython.

Would this also work fast if Cython doesn't know that d is a dict? Should I do list(<dict>(self._D).iteritems()), for instance?

Last edited 2 years ago by SimonKing (previous) (diff)

comment:35 in reply to: ↑ 33 Changed 2 years ago by jdemeyer

Replying to SimonKing:

I import six in order to be able to safely test for string types. How to, if I don't use six?

First of all, you should check whether you really need to test for "string types". Can you use duck typing instead? If for some reason, duck typing is hard, then it depends what you mean precisely with "string types".

Since you were mentioning filenames: isinstance(s, (bytes, unicode)) are valid types for filenames both in Python 2 and Python 3. Note that Python 3 also has https://docs.python.org/3/library/pathlib.html, a custom class to deal specifically with filenames.

Last edited 2 years ago by jdemeyer (previous) (diff)

Changed 2 years ago by SimonKing

Pickle that isn't readable in Python-3

comment:36 in reply to: ↑ 4 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

I can get the package to build, but it doesn't work yet. An illustration of one problem:

sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
... 
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

I tried to track down that problem, and currently it seems to me that it is due to a very bad backwards incompatibility in Python. You can reproduce it as follows:

  • Download attachment:State.sobj
  • Install the optional meataxe spkg (this may involve sage -b, so that some optional module in the Sage library is built)
  • Define some unpickle helper:
    sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle
    sage: class unpickle_old_mtx:
    ....:     def __call__(self, *args, **kwds):
    ....:         return mtx_unpickle(*args, **kwds)
    ....:     
    sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
    

Now, in Sage-with-Py-2, loading the file works:

sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
sage: X
([[1,
   [(2, 1),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_2']
]]],
  [2,
   [(4, 2),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1]                                                                                                                   
[0 1 0 0 0 0 0 0]                                                                                                                   
[0 0 0 0 0 1 0 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_2']
]]],
  [3,
   [(4, 2),
    [
[1 0 0 0 0 0 0 0]                                                                                                                   
[0 0 0 1 1 1 1 1]                                                                                                                   
[0 0 1 0 0 0 0 0]                                                                                                                   
[0 0 0 0 0 0 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_2']
]]]],
 [2, 1, 1],
 1,
 1,
 '8gp3/',
 ['b_1_0*b_1_1'],
 '8gp3/dat/',
 [((4, 2), '/home/king/.sage/pGroupCohomology/db/4gp2/H4gp2'),
  ((2, 1), '/home/king/.sage/pGroupCohomology/db/2gp1/H2gp1')],
 '8gp3/dat/',
 [],
 '8gp3/sgp/',
 2,
 [2, 3],
 [2, 2],
 2,
 2,
 ['b_1_0*b_1_1'],
 [[2, 'c_2_2', [0 0 1], 1, 0],
  [1, 'b_1_0', [1 0], 0, 0],
  [1, 'b_1_1', [0 1], 0, 0]],
 [[0, [['1', ['1']]]],
  [1, [['b_1_0', ['b_1_0']], ['b_1_1', ['b_1_1']]]],
  [2,
   [['c_2_2', ['c_2_2']],
    ['b_1_0', ['b_1_0^2']],
    ['b_1_1', ['b_1_0*b_1_1', 'b_1_1^2']]]]],
 {},
 0.24,
 True,
 'Monomials',
 2,
 None,
 '',
 0,
 '8gp3',
 '8gp3/dat/R8gp3.sobj',
 1,
 ['0'],
 ['b_1_0 + b_1_1'],
 None,
 [('_max_module_deg', 0),
  ('sgpDickson', [((4, 2), [[1, 'c_1_1', [0 1]]])]),
  ('useFactorization', True),
  ('useElimination', False),
  ('GroupDescr', 'Dihedral group of order 8'),
  ('_key', ((8, 3), '/home/king/SPKG/database/8gp3/dat/State')),
  ('auto', 4),
  ('Restriction_2', ['c_1_0*c_1_1+c_1_0^2', 'c_1_1', '0']),
  ('GroupName', 'D8'),
  ('DicksonExp', 3),
  ('_parameters_do_exist', True),
  ('KeepBases', None),
  ('completeGroebner', True),
  ('_SymondsTestdata', ['c_2_2', 'b_1_1', 'b_1_0']),
  ('_parameters_for_criterion', ['c_2_2', 'b_1_1', 'b_1_0']),
  ('root', '/home/king/SPKG/database/'),
  ('Restriction_3', ['c_1_0*c_1_1+c_1_0^2', '0', 'c_1_1']),
  ('_method', 'Symonds')],
 [(('dependent_parameters',), [['c_2_2', 'b_1_1', 'b_1_0'], 2])])

However, Sage-with-Py-3 yields the error found by John:

sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-4-74e7e91f9969> in <module>()
----> 1 X = load('/home/king/Projekte/coho/tests/State.sobj')

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
    149 
    150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
    152         X = loads(fobj.read(), compress=compress)
    153     try:

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
    150     ## Load file by absolute filename
    151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
    153     try:
    154         X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

And the point is: Python-3 even is unable to read the pickle file as a string! In Python-2, reading the file is no problem at all.

king@klap:~$ ~/Sage/git/py3/sage -python
Python 3.7.3 (default, Aug 27 2019, 23:22:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/king/Sage/git/py3/local/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte
>>> 
king@klap:~$ ~/Sage/git/sage/sage --python
Python 2.7.15 (default, Jul 26 2019, 11:49:43) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
'x\x9c\x95U\x8bZ\x1bE\x14&!\x91\x92J\x8b\x82V\xbc\x94x\xedXl\x96\x04\xaah\x95B\x0b\x05]\xc1\x16\x18\xad\x97\xe9:\xd9\x1dv\xb7d3{v7\nU,^\xd8B\xbd\xd6\xfb\xfd\xae\xcf\xd0\xa7\xf0E|\t\xcfl\xb2\x08\xf5\xab_I\xbe\x9d\xefd\xe6?\xe7\xfc\xe7?g6kY\xc2 \x83O\x96\xe8\x19\x06\xed\xc4\x0c\xb9-J\x81[\xb7\xc3\x92[\x8f\x84-\x82\x82\xc7\x97\x84\x91\xfe\x80\x1c\xcdT\xd6\xe7 \xef\xa0QF\xe3\xa6\x18:\x18\xec!\xc4\xf4\xa7\x02\xd9\xf0OJGz\xb2&\xed\x95\x92\x17-\x17f\x16\xce\x1a\x8d\xba\xef\x9aK5a\x985\x1e\x86\x05\xe8\x94PX\x85\xbd\xf4\x80a\x84|Q\x18\x8b2H1\x98\xd80\xe0\xe6\x8d\xb0J\xf4,~\xf7\xd0\xecZ\xffF4\x07]\x0c\xf6\x11\xa7\xa0v3\xb8\x9bY\xdb\xc4\xcd\xfdt@\xc3lB[B?m\xfe\xb4>\xa5Y<\xe2U\x1e\nm\xc4\xf6\x87\xb4\xd0\xf6\x13#\xb4\xcbF\x19\xbaw\x03\xaf\xc0-\x02?\x0cn\xc5\x9c\x0cz\x88*x\x18\x0b\xeeuZ\x12\xdc\x16\xc3\xed\x0c\x0e\xb4X\xe5\x90Un\xad\x7f,\xaf\xd8\xde\xc1\xa0\xef?l\xef\xbc\xf1\xf4\x15d{\xd7n\xe0\x15\xb8\xbb\xc9\xf6\x1e\xa2\xb738\xb8\xc5\xb6?e[\x8c\xe1^\x06\xf7\xedd[lWl\xefg\xf0\x00qvAo\x08\xe9=\xb8\x1bx\x05\x1e\x12M~\x87\x9a\x9adD:?$5\x1e\xa6\xf9\xc4\x8f\xc1a\xba\xb7j\x94\x8d\xc1\xc3j-\xc3\x00\xa7\x9d\xc9\t\xc6\xc7\xd3G\x88\xd3\xeb\xa0\xf6G\xa8\xb6\x8d@I\r\xaev\xed\x08jVU\x1b\xb6\xfd\x8a6\xad\xd6\x18JN\xde\xc1\x81\xd5n\xd0\xb5b\xfbemZ\xad1\x0c\x8a\xed,Z\xb6\xaaQMG\x99\xa42W\x941\x84\xc6\x10\x16;\xbc\xb5}45\x1e\x15\xceQ\xe5\xf1\xd8\xce\x1aG8\x83\xc7\xf1\x1e>\x81\xf2\xd0\xbci\xa8\x8e\x1eK\x07\xa8\x9df\x8aj\x80\x9eD\xbb\r\xc3>\x85\x97\x95\xe6\x13w\x18MA\xd9\xd6\x94\x1d\xd7\xdb\x12\xd0\xd8\x16\xa8\x0c\xe3\xdb@c\nt"\x01!\xea$\xe6\x9c z\x1b\x83I\x06\xa7\x08v\x82\xc1\x14\xae\\p<\x9dN\xde\nO#\xe6\x19\xe2\x8c2\xd0\xd3\xac\xcf\xaa\xd3\x19\xe2\x8c3\x98M\x93<\xc7\x93\x06\x9fN\xae\xcb\x19\xf4\x99#\xce1\x06\xf3i9\x0b\xca\x87\x924\xc4\xf3\x0c^\xa0\x1d\x89}\xae\x02g\xd5\xe1\x8b$\x8d\xf5\x12\x83\x97\xc9N\x85^i\x82\xcb\x08f\xc9,\xad\xc2\xb9\xa9\xe3\x7f]=\xf3\xf7\xfa\xc1\xab\x1b\xb4sF\xd6\xa5\xe7\xf2Z\x08F\xaa\xf5\xab\xb3\x14\xcb\xa49\xd5*\xe0\xb4g\xab}s\xca*\x85\xb2z^\xd5W\xa5\x99A\x94\xdf\xa4]I\xbe\xe2@\xb1\x99\xd1\xe2\xb3\x0c\x04\xa1\xfb\r\x8f/\x1b\x9e\xb4\x1a\xf8\n\xb3\x84\r\x8b*\xc1 &\xb0cph\x01\x87`\x02\xdf]\xa1\xac\x83\xcb\xe0\xbcs\x84\xc1\x12\x83Z\xa2\xbf\x99D\xf2\xae\xd5\xbf.x\x0c\x12\x1f\x9fv7Bq\x8a\x9b\x91\x0c\xdc\x0b<r1\x08l\xc4\x10\xd0}\xb8?Ys=\xb7\xde\xdc\r7c\x88h!\x99\xd2\t\x11\x9a\x014h\xdf\x84\xeb\x08+\xe0\xb5\xa2\xad\xb6\x8br\xb1(\x03K\x04\xc5\x11x-\x86\xd7i\xceX\x12+\xb0\xac\xe8\x8e \xdd\x95t6/\xc4\xf0\x06=\xf4\xff\xd7W\xe94\x1f\xf1H\xc4\xf0f\x0c\xab4\xc7\x1b\x91\x84\xb7\xf4\\\x0c\x17i\xd7\x9c\x08\xa3\xc05\x153l\xed\x1a\x83\xb7\t\xed1\x93v%%\x0f\x98\xad\xc6\xbe\x93j\xf0.J\x86\xb1\xde\xa3\x9dI\t\xb3\xdc\x13\xb0N\xb3\x13#\x10\xc7p\x89\x16Z\x12N.\xfb\xb0\xa1\xb7\xc7\xb0I{\r\x9f\x07\x08\x8bD\x10\x1a\x964\xc4\xb2\x1bFp\x19\xd5y\x9fv\xeaB\xf8\'\x90l\x08\x1f\xcc\xc6\xf0!\xed6\xa5\xe7\xd7\x10\x8b\xd1E\xb5.\x02\xf8\x08\x81\x1f\xd3nc~\xc5\x93u+\\@\xc6\xaaB\xf8\x84\xc1\x15\x9cLg\xdc\x19E>\x9f\xd2\xbe\xedi\xd4_\x91\x19\xb8h+\xcd?c\xf0\xf9\xbf\xd0/h.\x902\x82/i\xdfu\xa5\x83\xafb\xf8z\xa7<C\xf0\r\x83o\xaf#\xcfw\xa8J\xaa\xd0\xf7\x98\xe3\x07\xdaa \x15GZ\xf0#\xedhQ\x87\x9fb\xf8\x19\xef\xc7/\xb4\xd7\x12\xbe\xa8[\xa2\x1em#\r\xbf\xae\xc3o\x0c~\'\xce\x15=\x8b1\xfe\xe0\x11\xfcY\xfa\x07\x14\x1f\x95\xac'

comment:37 follow-up: Changed 2 years ago by chapoton

This works fine in python3:

sage: with open('State.sobj', 'rb') as f:
....:     f.read()

comment:38 in reply to: ↑ 37 Changed 2 years ago by SimonKing

Replying to chapoton:

This works fine in python3:

sage: with open('State.sobj', 'rb') as f:
....:     f.read()

Do I understand that it works because b means "binary"? But of course the real question is why Python 3 cannot unpickle from that file.

I tried to dig a bit deeper into Sage's load function (I guess the error is hidden there), and indeed load() uses with open(filename, 'rb') as fobj:. So, the unpickling problem is something else:

sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle
sage: filename = '/home/king/Projekte/coho/tests/State.sobj'
sage: compress = True
sage: class unpickle_old_mtx:
....:     def __call__(self, *args, **kwds):
....:         return mtx_unpickle(*args, **kwds)
....:     
sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
# The following is copied from Sage's load() function.
sage:     ## Load file by absolute filename
....:     with open(filename, 'rb') as fobj:
....:         X = loads(fobj.read(), compress=compress)
....:     try:
....:         X._default_filename = os.path.abspath(filename)
....:     except AttributeError:
....:         pass
....: 
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-654643a51ef6> in <module>()
      1 ## Load file by absolute filename
      2 with open(filename, 'rb') as fobj:
----> 3     X = loads(fobj.read(), compress=compress)
      4 try:
      5     X._default_filename = os.path.abspath(filename)

/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
    967 
    968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
    970 
    971 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

So, now we see that the error occurs in SageUnpickler(io.BytesIO(x)).load().

Last edited 2 years ago by SimonKing (previous) (diff)

comment:39 follow-up: Changed 2 years ago by SimonKing

I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.

So, I am now trying to find if this is really the case or not.

comment:40 in reply to: ↑ 39 ; follow-up: Changed 2 years ago by SimonKing

Replying to SimonKing:

I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.

So, I am now trying to find if this is really the case or not.

It isn't. I inserted a print statement at the beginning of mtx_unpickle, but it isn't shown before the error triggered by load('/home/king/Projekte/coho/tests/State.sobj') is raised.

I'll open a ticket for that problem.

comment:41 in reply to: ↑ 40 Changed 2 years ago by SimonKing

Replying to SimonKing:

I'll open a ticket for that problem.

It is #28444. I am now convinced that the problem is in Python-3 resp. in Sage's unpickling, but not in MeatAxe.

comment:42 Changed 2 years ago by SimonKing

  • Dependencies set to #28444

comment:43 Changed 2 years ago by SimonKing

It seems that #28444 fixes the problem for p_group_cohomology on Python-3. With the current unpublished code for the package, I get

$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta9, Release Date: 2019-09-02               │
│ Using Python 3.7.3. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.set_workspace(tmp_dir())
sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Get original root and GStem
/home/king/SPKG/database 8gp3
now, StateFile is /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj
Try to get (8, 3) from cache
Group data are rooted at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj'
Setting state
> export action matrices
Differential reloaded
> rk P_02 =   3
Differential reloaded
> rk P_03 =   4
Import monomials
Checking compatibility of SmallGroups library and stored cohomology ring
sage: print(H)

Cohomology ring of Dihedral group of order 8 with coefficients in GF(2)

Computation complete
Minimal list of generators:
[c_2_2: 2-Cocycle in H^*(D8; GF(2)),
 b_1_0: 1-Cocycle in H^*(D8; GF(2)),
 b_1_1: 1-Cocycle in H^*(D8; GF(2))]
Minimal list of algebraic relations:
[b_1_0*b_1_1]

Hooray...

comment:44 Changed 2 years ago by SimonKing

Strangely, while I can compute some cohomology rings, the following computation fails in py3:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: H = CohomologyRing(32,4)
sage: H.make()

Reason: Automatic saving does not work, and apparently it is because H.__getstate__() returns Singular elements, which cannot be pickled. It seems I did take care of it, as H.__getstate__() in py2 only returns the class sage.interfaces.singular.SingularElement and not the instance, but for some reasons it doesn't in py3.

comment:45 Changed 2 years ago by SimonKing

Question: Has the mechanism of tab completion changed between py2 and py3? I notice that tab completion on a cohomology ring works in py2, but doesn't work (no error, but no completion either) in py3.

comment:46 Changed 2 years ago by dimpase

tab completion is done by ipython, which should be upgraded (they've switched to Python 3 only, so we're using an old version, which in turn might lag in its Python 3 support).

I suggest not to worry about it now.

comment:47 Changed 2 years ago by SimonKing

Aha! The culprit for pickle problem is pGroupCohomology.cohomology.pickle_gap_data, which inspite of its name also takes care of singular elements. In Py2:

sage: self = CohomologyRing(32,4)
sage: self.make()
sage: G = self._decorator_cache['_parameter_restrictions',frozenset({'c_2_2'}),('radical', True)]
sage: G
[[(polynomial ring, over a field, global ordering
   // coefficients: ZZ/2
   // number of vars : 2
   //        block   1 : ordering M
   //                  : names    c_1_0 c_1_1
   //                  : weights      1     1
   //                  : weights     -1     0
   //        block   2 : ordering C,
   <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca690>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
   <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca500>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
 2]
sage: pickle_gap_data(G)
[[(<class 'sage.interfaces.singular.SingularElement'>,
   <class 'sage.interfaces.singular.SingularElement'>,
   <class 'sage.interfaces.singular.SingularElement'>)],
 2]

In Py3:

sage: pickle_gap_data(G)
[[(polynomial ring, over a field, global ordering
   // coefficients: ZZ/2
   // number of vars : 2
   //        block   1 : ordering M
   //                  : names    c_1_0 c_1_1
   //                  : weights      1     1
   //                  : weights     -1     0
   //        block   2 : ordering C,
   <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86a20>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
   <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86dc8>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
 2]

comment:48 Changed 2 years ago by SimonKing

Ouch. A nasty thing.

In pickle_gap_data, I test if the input is an iterable class, and if it is and if the class is a builtin, then the pickle_gap_data traverses the items of the input and creates from the individual results a modified copy of the input. In code:

    try:
        I = iter(G)
    except:
        return G
    if isinstance(G,dict):
        return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items())
    if getattr(type(G), '__module__', None) == '__builtin__':
        return type(G)(pickle_gap_data(X) for X in I)
    return G

The problem is that in Py3 we get

sage: getattr(list, '__module__', None)
'builtins'

Question: How to test in a py2/py3 invariant way whether a type is a builtin?

That asked, are there more iterable builtins than list, tuple, str/bytes and dict? I guess it is safer to just explicitly test for list and tuple. EDIT Like this:

    if isinstance(G, dict):
        return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items())
    if isinstance(G, (list, tuple)):
        return type(G)(pickle_gap_data(X) for X in G)
    return G
Last edited 2 years ago by SimonKing (previous) (diff)

comment:49 follow-up: Changed 2 years ago by dimpase

An iterable is an object with an __iter__ method or/and __getitem__ method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)

comment:50 in reply to: ↑ 49 Changed 2 years ago by SimonKing

Replying to dimpase:

An iterable is an object with an __iter__ method or/and __getitem__ method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)

You misunderstood. The problem was not how to test if something is iterable but how to test if that iterable is a builtin. But, as I said in my previous post, probably this isn't the actual problem anyway. What I want is to deal with strings, lists, tuples and dicts. Period. So, I could be more explicit (and I already tested that the error from comment:44 can be fixed in that way).

comment:51 Changed 2 years ago by SimonKing

In Python-2, one implemented c.__nonzero__() and then bool(c) would work. It seems that in Python-3, something else needs to be done. Is it perhaps the case that I need to implement __bool__?

What should I do to make stuff work both in py-2 and py-3? Implement both __bool__ and __nonzero__? Or would it be enough to replace __nonzero__ with __bool__?

comment:52 Changed 2 years ago by SimonKing

I found this

class test:
    def __bool__(self):
        return False
    __nonzero__=__bool__

here.

comment:53 follow-up: Changed 2 years ago by SimonKing

There is a problem with the logging module: In Python-2, I get

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: P = CohomologyRing(9,2,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/19448/dir_0MUJxK/'
Computing basic setup for Small Group number 1 of order 3
Computing basic setup for Small Group number 2 of order 9
Resolution of GF(3)[9gp2]: 
          > export action matrices
H^*(SmallGroup(9,2); GF(3)): 
          Initialising maximal p-elementary abelian subgroups

whereas in Python-3 I get

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: P = CohomologyRing(9,2,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/18649/dir_9bz1_9bh/'
Computing basic setup for Small Group number 1 of order 3
Computing basic setup for Small Group number 2 of order 9
> export action matrices
Initialising maximal p-elementary abelian subgroups

So, the indentation is gone, and (whats worse) the logging doesn't indicate what object the log comes from (which I found very useful for debugging, as in the computation of a single cohomology ring often many other rings are involved).

Can someone please give me a pointer to a page explaining the differences between py-2 and py-3 in the logging module?

comment:54 in reply to: ↑ 53 Changed 2 years ago by SimonKing

Replying to SimonKing:

Can someone please give me a pointer to a page explaining the differences between py-2 and py-3 in the logging module?

By looking at the code of the logging module in Python-2 and Python-3, I see where the problem comes from. In the following, self denotes an instance of pGroupCohomology.auxiliary.CohoFormatter, which sub-classes loggin.Formatter.

In Python-2, logging.Formatter.format(self, record) uses self._fmt to format the output. In Python-3, self._fmt is not used. Instead, self._style._fmt is consulted, where self._style is one of the styles in logging._STYLES.

The point is: So far, my custom self.format modifies self._fmt and then calls the default .format(). However, in Python-3, when calling the default .format(), the attribute self._fmt plays no role, but this is where I currently put the format into.

Simple way out: Do not call logging.Formatter.format in CohoFormatter.format.

But I wonder: Is there a recommended way to make the logging format change from call to call?

By this, I mean: Make the output of CohoFormatter.format(self, record) depend on record.args[0], in the sense that the output makes visible whether record.args[0] is the same object as in the previous call to CohoFormatter.format.

Or, more radically: What kind of logging approach would you recommend, what do you find most easy to comprehend/parse as a human (I am not interested in automatic parsing of the logging messages)?

I somehow like the current approach, which is: Assume that certain objects (say, ZZ and RR) are involved in a computation, and each object may issue a logging message. Then, stream of messages is arranged in blocks of messages coming from a single object, each block being prepended by an identifier of the object. I.e., if ZZ gives two messages, then RR gives three messages and then again ZZ gives a message, the output would be

Integer Ring:
    message 1
    message 2
Rational Field:
    message 3
    message 4
    message 5
Integer Ring:
    message 6
  1. Do you like this thematic arrangement of messages (which is provided by coho_logger in Python 2, but fails in Python-3)?
  2. If yes: Can you recommend a clean and easy way to achieve it both in Python 2 and Python 3? Ideally, one should still be able to include time information in the logging, which currently is not supported.
  3. If no: What logging format would you recommend?

comment:55 follow-up: Changed 2 years ago by jhpalmieri

Instead of setting self._fmt, can you instead run self.basicConfig(format=FORMAT)? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#user-defined-objects), so maybe you can make use of that somehow.

comment:56 in reply to: ↑ 55 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

Instead of setting self._fmt, can you instead run self.basicConfig(format=FORMAT)? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#user-defined-objects), so maybe you can make use of that somehow.

From these links, I get the impression that each call to the logger results in the creation of a new formatter. Is that really the case?

comment:57 follow-up: Changed 2 years ago by jhpalmieri

I don't know, I only looked at the links, too. It's worth trying.

(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)

comment:58 in reply to: ↑ 57 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

I don't know, I only looked at the links, too. It's worth trying.

(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)

Sure. But I am unsure how a change of formatter could solve the problem without impact on performance.

If logging is in "warn" mode, then there are of course still silent calls to coho_logger.debug. I want that (1) these calls are cheap (i.e., they shouldn't involve creation of Python objects) and (2) the message blocks should not take into account silent messages, i.e., a block shouldn't be closed if a silent message comes from a different object.

Of course, it should be that what I do with the formatter should rather be done with a handler, or maybe a filter. So, I will try to understand what exactly what handlers and filters do.

And, as in point 3. of comment:54: Would you recommend a different formatting of log messages?

comment:59 Changed 2 years ago by SimonKing

Currently I am reading logging HOWTO and logging Cookbook, and at the moment it seems to me that I want to use logging.LoggingAdapter, because it allows to add contextual information (which is what I'd like to use, unless people tell me a nicer way log format).

comment:60 Changed 2 years ago by SimonKing

I think I now know what to do: The customisation should be done in a Formatter, i.e., the existing solution was a good approach. Only the format() method should be different: Before calling super(CohoFormatter, self).format(record), I should add some attribute to the record that is then used in the formatter's format string.

comment:61 Changed 2 years ago by SimonKing

I think I solved the problem. I can now optionally add time information to the log, such as here:

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: CohomologyRing.global_options('time')
sage: H = CohomologyRing(720,763,prime=3)
    2019-09-18 20:54:20,277: Try to compute a Sylow 3-subgroup
    2019-09-18 20:54:20,283: Try to find the SmallGroups address of the Sylow subgroup
    2019-09-18 20:54:20,288: We compute this cohomology ring from scratch
    2019-09-18 20:54:20,289: Group data are rooted at '/home/king/.sage/temp/klap/962/dir_tnhvqeq7/'
    2019-09-18 20:54:20,293: Computing basic setup for Small Group number 1 of order 3
    2019-09-18 20:54:20,405: Computing basic setup for Small Group number 2 of order 9
Resolution of GF(3)[9gp2]:
    2019-09-18 20:54:20,466: > export action matrices
H^*(SmallGroup(9,2); GF(3)):
    2019-09-18 20:54:20,517: Initialising maximal p-elementary abelian subgroups
    2019-09-18 20:54:20,555: Computing intermediate subgroup
    2019-09-18 20:54:20,567: Try to find the SmallGroups address of the intermediate subgroup
    2019-09-18 20:54:20,603: Computing group order
    2019-09-18 20:54:20,604: The group is of order 72
    2019-09-18 20:54:20,609: Computing intermediate subgroup
Resolution of GF(3)[SmallGroup(9,2)]:
    2019-09-18 20:54:20,625: Computing next term
    2019-09-18 20:54:20,644: > rk P_02 =   3
H^*(SmallGroup(9,2); GF(3)):
    2019-09-18 20:54:20,647: We have to choose 2 new generators in degree 1
    2019-09-18 20:54:20,648: > There are 2 nilpotent generators in degree 1
...

However, what I'd find even nicer if the sum of the cputimes of Sage and Singular since start of logging was shown, not just the walltime, as above.

Note that the indentation has changed, which means that I will have to change various doc tests, but that will be doable.

comment:62 Changed 2 years ago by SimonKing

Sigh.

After solving the log problem, I find that meanwhile the code ONLY works in Python-3, not Python-2.

First problem:

class FOO(object):
    def __bool__(self):
        return something
    __nonzero__ = __bool__

works in Py3, but not in Py2, although it was recommended in some stackoverflow discussion. In Py2 it results in NameError: name '__bool__' is not defined. How to fix it without code duplication?

Second problem:

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.py in <module>()
     47 import re,os
     48 
---> 49 import urllib.request, urllib.error
     50 import tarfile
     51 import logging

ImportError: No module named request

How to deal with urllib in a py2/py3-compatible way?

comment:63 follow-up: Changed 2 years ago by jhpalmieri

For the urllib issue: http://python3porting.com/noconv.html#import-errors suggests code like

try:
    from urllib.request import ...
except ImportError:
    from urllib import ...

I can't reproduce the first problem. With Python 2, if I do

sage: class FOO(object):
....:     def __bool__(self):
....:         return something
....:     __nonzero__ = __bool__
....:     
sage: a = FOO()
sage: a.__nonzero__()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-74dab0bd3f82> in <module>()
----> 1 a.__nonzero__()

<ipython-input-1-ec93abb3d539> in __bool__(self)
      1 class FOO(object):
      2     def __bool__(self):
----> 3         return something
      4     __nonzero__ = __bool__
      5 

NameError: global name 'something' is not defined

so a.__nonzero__() is certainly calling __bool__. If I change the return line to something sensible, it works.

comment:64 in reply to: ↑ 63 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

For the urllib issue: http://python3porting.com/noconv.html#import-errors suggests code like

try:
    from urllib.request import ...
except ImportError:
    from urllib import ...

OK, thank you.

I can't reproduce the first problem. With Python 2, if I do

The "something" was of course figuratively. Anyway, the problem seems to be that I am using a cdef class here:. In Sage-with-py2:

sage: cython('''
....: class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')
sage: cython('''
....: cdef class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')
ERROR:root:An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line string', (1, 0))

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-91e48d391924> in <module>()
      4         return True
      5     __nonzero__ = __bool__
----> 6 ''')

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/lazy_import.pyx in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3697)()
    352             True
    353         """
--> 354         return self.get_object()(*args, **kwds)
    355 
    356     def __repr__(self):

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_compile(code, **kwds)
    649     with open(tmpfile,'w') as f:
    650         f.write(code)
--> 651     return cython_import_all(tmpfile, get_globals(), **kwds)

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_import_all(filename, globals, **kwds)
    539       code
    540     """
--> 541     m = cython_import(filename, **kwds)
    542     for k, x in iteritems(m.__dict__):
    543         if k[0] != '_':

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_import(filename, **kwds)
    519     try:
    520         sys.path.append(build_dir)
--> 521         return builtins.__import__(name)
    522     finally:
    523         sys.path = oldpath

/home/king/.sage/temp/klap/23274/spyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0.pyx in init _home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0()
      3     def __bool__(self):
      4         return True
----> 5     __nonzero__ = __bool__

NameError: name '__bool__' is not defined

whereas in Sage-with-py3 I get

sage: cython('''
....: class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')
sage: cython('''
....: cdef class Foo:
....:     def __bool__(self):
....:         return True
....:     __nonzero__ = __bool__
....: ''')

So, what to do?

comment:65 follow-up: Changed 2 years ago by jhpalmieri

I don't know Cython, but replacing __nonzero__ = __bool__ with

def __nonzero__(self):
    return self.__bool__()

seems to work.

comment:66 in reply to: ↑ 65 Changed 2 years ago by SimonKing

Replying to jhpalmieri:

I don't know Cython, but replacing __nonzero__ = __bool__ with

def __nonzero__(self):
    return self.__bool__()

seems to work.

Sure. that's what I basically meant with "code duplication". It is an additional call, thus, if the boolean value of a cochain is tested very often then one will see a slow-down. But I don't know if it actually is tested very often.

Anyway, with this code cuplication, things seem to work. Hooray!

At the moment, it seems that most of the remaining work will be to change the expected output for those tests that use logging.

comment:67 follow-up: Changed 2 years ago by jhpalmieri

My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does __nonzero__ do anything with Python 3? If so, maybe you could do something like

from six import PY2
if PY2:
    def __nonzero__(self):
        ...

to avoid the extra call when using Python 3.

comment:68 in reply to: ↑ 67 ; follow-up: Changed 2 years ago by SimonKing

Replying to jhpalmieri:

My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does __nonzero__ do anything with Python 3? If so, maybe you could do something like

from six import PY2
if PY2:
    def __nonzero__(self):
        ...

to avoid the extra call when using Python 3.

I don't think that __nonzero__ does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current not-yet-published code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for __nonzero__ and for __bool__, but only one of them would be compiled.

comment:69 Changed 2 years ago by SimonKing

One detail: According to unit_test_64, there is no significant speed difference between the py2 and py3 versions. In any case, the cputime obtained as sum of the cputime() and Singular's timer is only half of the walltime. Do I recall correctly that the time used to save data on disk is not measured by cputime()? Is there an easy way to determine that time, too?

comment:70 Changed 2 years ago by dimpase

I don't think it makes much sense measuring time to save to disk, as it varies wildly between different OS's, different hardware, different configuration of buffering, etc etc.

Often saving is completely asyncronous from the computation due to buffering and separate threads run to do buffer flushing, etc.

comment:71 Changed 2 years ago by SimonKing

Now I am totally baffled. Is it not the case that comparison is supposed to be implemented by __richcmp__? Apparently __richcmp__ isn't called anymore.

The following corresponds to the code in pGroupCohomology.dickson, that did work in the past:

sage: cython("""
....: from sage.structure.richcmp import richcmp, op_LT, op_NE, op_GT
....: from sage.all import GF
....: class DICKSON:
....:     def __init__(self,p):
....:         self.K = GF(p)
....:     def __richcmp__(self, other, op):
....:         print("Comparison called")
....:         return richcmp(self.K, other.K, op)
....: """)

Now, rich comparison fails both in py2 and py3:

sage: D = DICKSON(5)
sage: F = loads(dumps(D))
sage: D == F
False
sage: F.K == D.K
True

What has changed, how to fix it?

comment:72 follow-up: Changed 2 years ago by SimonKing

Aha! I see in sage.misc.unknown that a decorator @richcmp_method is used. So, I guess I need to use it, too.

comment:73 in reply to: ↑ 72 Changed 2 years ago by SimonKing

Replying to SimonKing:

Aha! I see in sage.misc.unknown that a decorator @richcmp_method is used. So, I guess I need to use it, too.

Hooray:

sage: cython("""
....: from sage.structure.richcmp import richcmp, richcmp_method
....: from sage.all import GF
....: @richcmp_method
....: class DICKSON(object):
....:     def __init__(self,p):
....:         self.K = GF(p)
....:     def __richcmp__(self, other, op):
....:         print("Comparison called")
....:         return richcmp(self.K, other.K, op)
....: """)
sage: D = DICKSON(5)
sage: F = loads(dumps(D))
sage: D == F
Comparison called
True

comment:74 follow-ups: Changed 2 years ago by SimonKing

At the moment, tests in dickson, auxiliaries, resolution and factory pass both in Py2 and Py3. In isomorphism_test, there are currently two failing tests, and it seems to me that they reveal a Py3-bug in cartesian_product_iterator:

File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 353, in pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms
Failed example:
    T.explore_isomorphisms()
Exception raised:
    Traceback (most recent call last):
      File "sage/misc/cachefunc.pyx", line 1944, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10139)
        return cache[k]
    KeyError: (((3, 5, 4, 6, 7, 1, 2), 0, True), ())

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/mrange.py", line 139, in _xmrange_iter
        curr_elt = [next(i) for i in curr_iters[:-1]] + [None]
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/mrange.py", line 139, in <listcomp>
        curr_elt = [next(i) for i in curr_iters[:-1]] + [None]
    StopIteration

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms[10]>", line 1, in <module>
        T.explore_isomorphisms()
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/isomorphism_test.py", line 402, in explore_isomorphisms
        Cands = self.candidates_of_gens(tuple([p[4] for p in Sizes]),len(self.rigid_generators))
      File "sage/misc/cachefunc.pyx", line 1949, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10273)
        w = self._instance_call(*args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1825, in sage.misc.cachefunc.CachedMethodCaller._instance_call (build/cythonized/sage/misc/cachefunc.c:9758)
        return self.f(self._instance, *args, **kwds)
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/isomorphism_test.py", line 644, in candidates_of_gens
        for Ims1,Ims2 in cartesian_product_iterator([FirstHalf, SecondHalf]):
    RuntimeError: generator raised StopIteration

I will investigate which values of FirstHalf and SecondHalf trigger it.

comment:75 in reply to: ↑ 74 Changed 2 years ago by SimonKing

Replying to SimonKing:

I will investigate which values of FirstHalf and SecondHalf trigger it.

See #28521

EDIT: I am not sure if I'll make #28521 a dependency or simply work around the bug.

Last edited 2 years ago by SimonKing (previous) (diff)

comment:76 in reply to: ↑ 74 Changed 2 years ago by SimonKing

Replying to SimonKing:

At the moment, tests in dickson, auxiliaries, resolution and factory pass both in Py2 and Py3.

When working around the cartesian_product_iterator bug, the tests in isomorphism_test pass both in Py2 and Py3. Yay!

Last edited 2 years ago by SimonKing (previous) (diff)

comment:77 Changed 2 years ago by SimonKing

Two errors in Py3 concern __div__, namely

Running doctests with ID 2019-09-21-07-26-57-20b48344.
Using --optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage
Doctesting 1 file.
sage -t --long --warn-long 47.4 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx", line 3131, in pGroupCohomology.cochain.MODCOCH.__div__
Failed example:
    print(H.1/3) #indirect doctest
Expected:
    (a_6_0)/(3): 6-Cocycle in H^*(SmallGroup(400,206); GF(5))
    defined by
    2*c_2_1*c_2_2*a_1_0*a_1_1
Got:
    (a_6_0)*(2): 6-Cocycle in H^*(SmallGroup(400,206); GF(5))
    defined by
    2*c_2_1*c_2_2*a_1_0*a_1_1
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx", line 3139, in pGroupCohomology.cochain.MODCOCH.__div__
Failed example:
    H.2/H.1
Expected:
    Traceback (most recent call last):
    ...
    TypeError: Can not divide by <class 'pGroupCohomology.cochain.MODCOCH'>
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest pGroupCohomology.cochain.MODCOCH.__div__[6]>", line 1, in <module>
        H.gen(2)/H.gen(1)
      File "sage/structure/element.pyx", line 1718, in sage.structure.element.Element.__truediv__ (build/cythonized/sage/structure/element.c:12780)
        return (<Element>left)._div_(right)
      File "sage/structure/element.pyx", line 2654, in sage.structure.element.RingElement._div_ (build/cythonized/sage/structure/element.c:18184)
        frac = self._parent.fraction_field()
    TypeError: 'NoneType' object is not callable
**********************************************************************
1 item had failures:
   2 of   8 in pGroupCohomology.cochain.MODCOCH.__div__
    [1490 tests, 2 failures, 465.54 s]
----------------------------------------------------------------------
sage -t --long --warn-long 47.4 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx  # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 470.6 seconds
    cpu time: 249.3 seconds
    cumulative wall time: 465.5 seconds

The first error seems to indicate that H.1/3 is automatically translated to H.1*2, which does make sense modulo 5, but which I did not implement like this. And the second error is nothing that I implemented.

In fact I overrode __div__ for COCH and MODCOCH. Is it perhaps the case that overriding __div__ won't work in Py3?

comment:78 follow-up: Changed 2 years ago by chapoton

In py3, there is __truediv__ and __floordiv__

comment:79 in reply to: ↑ 78 ; follow-up: Changed 2 years ago by SimonKing

Replying to chapoton:

In py3, there is __truediv__ and __floordiv__

I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement __truediv__, __floordiv__ and __div__?

On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.

So, given a RingElement, what methods do I need to implement for providing division by an element of the base field, and what do I need to implement for making clear that division of two ring elements doesn't work?

I guess I should implement a fraction_field method for cohomology rings. Of course, since cohomology rings can have zero divisors, a fraction field will not always exist. So, should in that situation fraction_field return NotImplemented, or raise a ValueError, or what else?

Do I need to provide a single-underscore _div_ method, in addition to the fraction_field?

How to implement division by a base field element (I guess _div_ is for division of two ring elements)?

comment:80 in reply to: ↑ 79 Changed 2 years ago by SimonKing

Replying to SimonKing:

Replying to chapoton:

In py3, there is __truediv__ and __floordiv__

I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement __truediv__, __floordiv__ and __div__?

On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.

I think I'll proceed as follows:

  • Implement a fraction_field method for cohomology rings raising a TypeError (note that ZZ.quo(6).fraction_field() raises a TypeError, too)
  • Do NOT implement _div_ -- the attempt to divide two elements of a cohomology ring will result in the TypeError raised by the fraction_field() method
  • Division of a cohomology element by an element of the base field will automatically work, by multiplying the inverse of the field element with the cohomology element, using _lmul_/_rmul_. I should add an example/test in the documentation of the _lmul_/_rmul_ methods

comment:81 in reply to: ↑ 68 Changed 2 years ago by SimonKing

Replying to SimonKing:

I don't think that __nonzero__ does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current not-yet-published code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for __nonzero__ and for __bool__, but only one of them would be compiled.

This did work, to some extent. But not really reliably. Therefore, to be on the safe side, I will implement both methods in both python versions, since the code duplication is only small.

comment:82 Changed 2 years ago by SimonKing

I am currently dealing with different but mathematically equivalent behaviour: Apparently, previously Singular did not always do tail reduction, i.e., some rings in the database do not have a fully reduced representation of the relation ideal. For example, here are relations for the cohomology of SmallGroup(64,14):

sage: H.rels()   # not tail reduced, result obtained with py2
['a_1_0^2',
 'a_1_0*a_1_1',
 'a_1_1^3',
 'a_2_1*a_1_0',
 'a_2_1^2+a_2_1*a_1_1^2',
 'a_1_1*a_3_3+a_2_1^2',
 'a_1_0*a_3_3+a_2_1^2',
 'a_2_1*a_3_3',
 'a_3_3^2']

versus

sage: H.rels()   # tail reduced, obtained with Py3
['a_1_0^2',
 'a_1_0*a_1_1',
 'a_1_1^3',
 'a_2_1*a_1_0',
 'a_2_1^2+a_2_1*a_1_1^2',
 'a_1_0*a_3_3+a_2_1*a_1_1^2',
 'a_1_1*a_3_3+a_2_1*a_1_1^2',
 'a_2_1*a_3_3',
 'a_3_3^2']

As you can see, the relation a_2_1^2+a_2_1*a_1_1^2 could be used to reduce the tail of a_1_1*a_3_3+a_2_1^2. Also, the sorting of the relations is different.

The question is how to deal with it. I guess I could proceed as follows:

sage: if (2, 8) < sys.version_info:
....:     expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0',
....:           'a_2_1^2+a_2_1*a_1_1^2', 'a_1_0*a_3_3+a_2_1*a_1_1^2',
....:           'a_1_1*a_3_3+a_2_1*a_1_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
....: else:
....:     expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0',
....:           'a_2_1^2+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1^2',
....:           'a_1_0*a_3_3+a_2_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
....:     
sage: H.rels() == expected_rels
True

comment:83 Changed 2 years ago by SimonKing

Sigh. I am suffering from "interesting" debugging sessions. At the moment, it seems to me that some errors in the py2-version only occur, when I first install p_group_cohomology in the py2-install of Sage, then install p_group_cohomology in the py3-install of Sage, and only then do tests in the py2-install.

comment:84 follow-up: Changed 2 years ago by dimpase

I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.

comment:85 in reply to: ↑ 84 ; follow-up: Changed 2 years ago by SimonKing

Replying to dimpase:

I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.

You mean by testing the p_group_cohomology package, or something else?

Indeed I had the same impression concerning ~/.sage/. Fortunately most of the trouble could be solved by putting str() around some data (because I got unicode where a string was expected - so, perhaps a rogue py3-pickle got misinterpreted in py2).

comment:86 in reply to: ↑ 85 Changed 2 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.

You mean by testing the p_group_cohomology package, or something else?

Indeed I had the same impression concerning ~/.sage/. Fortunately most of the trouble could be solved by putting str() around some data (because I got unicode where a string was expected - so, perhaps a rogue py3-pickle got misinterpreted in py2).

I was getting different output from a test, IIRC it was releated to unicode, and the problem went away after I removed ~/.sage/. I thought it might have been related to ipython, but who knowns.

comment:87 Changed 2 years ago by SimonKing

I tried to solve it by doing an explicit conversion to str whenever there is data that is expected of this type. And instead of if isinstance(bla, str), I now do if isinstance(bla, (str, unicode)), where I define unicode=str in py3.

So far, this seems to work. And after all, most users will have only one Sage installation anyway.

comment:88 in reply to: ↑ 13 Changed 2 years ago by SimonKing

Replying to SimonKing:

Replying to SimonKing:

If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.

Hooray:

# in Sage-with-python-2
sage: P = CombinatorialFreeModule(ZZ,'x')
sage: save(P,'../../pickle')
# BOTH with python-2 and python-3
sage: B = load('../../pickle.sobj')
sage: B._default_filename
'/home/king/Sage/pickle.sobj'

So, the old trick should still work.

I wonder if I shouldn't improve the trick.

Currently, if the initialisation of an unpickled object can not be finished, it is the __getattr__ method that would trigger the completion of initialisation based on the value of self._default_filename that is set by Sage's loads() after unpickling the object and before returning it.

But what if I turn _default_filename into a property? I could make it so that its setter triggers the completion of initialisation, to the effect that the object returned by loads() will be valid. The trickery would be in the property setter, not in __getattr__.

comment:89 Changed 2 years ago by SimonKing

I am puzzled about the appearance of unicode. For example, consider these two lines in my code:

        root = str(kwds.get('root', COHO.workspace))
        coho_logger.debug("Group data are rooted at %r", None, root)

The logging appears in the following session (in Sage-with-py2, where I first installed the package in Sage-with-py3 and then in Sage-with-py2):

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: from pGroupCohomology.cohomology import COHO
sage: _ = COHO()
<module>:
    Group data are rooted at u'/home/king/.sage/temp/klap/6618/dir_T5Im2o/'

So, apparently the variable root in the code snipped above is a unicode, although I explicitly converted it to a str. Moreover:

sage: COHO.workspace
'/home/king/.sage/temp/klap/6618/dir_T5Im2o/'
sage: type(COHO.workspace)
<type 'str'>

In other words: There is an attribute COHO.workspace of type str, and when I convert it to a str then a unicode results. WTF???

comment:90 follow-up: Changed 2 years ago by dimpase

I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.

It might be that iPython installs some kind of display hook that sets str to convert to unicode. There is code in src/sage/repl/rich_output that might be doing this.

That is to say that unicode you see printed might be happening due to on the fly conversion, set up by DisplayManager.

Last edited 2 years ago by dimpase (previous) (diff)

comment:91 in reply to: ↑ 90 Changed 2 years ago by SimonKing

Replying to dimpase:

I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.

Is there unicode in py3?

The common topic is: In some cases I get a TypeError saying that a str was expected, but got unicode. And a log shows that a variable is of type unicode after an EXPLICIT(!) cast to str.

In all mentioned cases, I proceeded as follows:

  1. Install the package in Sage-with-py2
  2. Install the package in Sage-with-py3
  3. Test in Sage-with-py2

Sage-with-py2 and Sage-with-py3 are of course in different folders. What they share: The git repository (but different git worktrees), and DOT_SAGE.

But it is all very flaky. I just inserted a couple of print statements in my code to investigate more deeply, did the above double-install procedure --- and suddenly the type was str, not unicode.

It might be that iPython installs some kind of display hook that sets str to convert to unicode. There is code in src/sage/repl/rich_output that might be doing this.

It is not only the output but the type inside of the code. In some cases, I get a TypeError.

comment:92 Changed 2 years ago by dimpase

Do you see these errors while running doctests? I gather there is --nodotsage option you can use while running tests, so you may try to see whether you can reproduce this without DOT_SAGE.

Another thing to try is to wipe out DOT_SAGE each time you switch from pyX to py(5-X), and see whether this make the error to go away.

comment:93 Changed 2 years ago by dimpase

another way to work around this might be just to convert all your strings to unicode.

comment:94 Changed 2 years ago by SimonKing

I am afraid that it is currently not possible for anybody else to reproduce the following, because I didn't publish my code yet. But anyway, here is what I did:

  1. Insert type information, so that the mentioned code snippet now is
            root = str(kwds.get('root', COHO.workspace))
            coho_logger.debug("Group data are rooted at %r. Type: %r", None, root, type(root))
    
  2. Install it in py3
  3. Install it in py2
  4. In Py3, run the following:
    sage: from pGroupCohomology import CohomologyRing
    sage: CohomologyRing.doctest_setup()
    sage: H = CohomologyRing(64,14, from_scratch=True)
    sage: H.make()
    
  5. In Py2, run the following:
    sage: from pGroupCohomology import CohomologyRing
    sage: from pGroupCohomology.cohomology import COHO
    sage: CohomologyRing.doctest_setup()
    sage: CohomologyRing.global_options('debug')
    sage: _ = COHO()
    <module>:
        Group data are rooted at u'/home/king/.sage/temp/klap/14917/dir_20mizE/'. Type: <type 'unicode'>
    

So, the type REALLY is unicode, even though I did an EXPLICIT conversion to str.

Then, I continued:

  1. Move the old ~/.sage and create a new one.
  2. Start Sage-with-py2, and repeat the test:
    Setting permissions of DOT_SAGE directory so only you can read and write it.
    sage: from pGroupCohomology import CohomologyRing
    sage: from pGroupCohomology.cohomology import COHO
    sage: CohomologyRing.doctest_setup()
    sage: CohomologyRing.global_options('debug')
    sage: _ = COHO()
    <module>:
        Group data are rooted at u'/home/king/.sage/temp/klap/15112/dir_fYCkwo/'. Type: <type 'unicode'>
    sage: type(COHO.workspace)
    <type 'str'>
    
    

So, removing DOT_SAGE didn't help. For comparison, do the same in Py3:

sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
<module>:
    Group data are rooted at '/home/king/.sage/temp/klap/15493/dir_81mwix3u/'. Type: <class 'str'>

which of course is no surprise, as python-3 has no unicode.

Note that the line _ = COHO() does not involve taking an old pickle from a repository. It merely creates a new instance of COHO and initialises it only to the point where the current workspace is assigned to the attribute .root.

Summary: The class attribute COHO.workspace is of type str. The dictionary kwds is empty. str(kwds.get('root', COHO.workspace))) returns the unicode version of `COHO.workspace. I.e., converting a str to a str results in a unicode.

comment:95 Changed 2 years ago by SimonKing

Just to document that we are talking about a Heisenbug: I wanted to test whether it is perhaps the case that str==unicode. Therefore, I changed the code snippet to

        root = str(kwds.get('root', COHO.workspace))
        coho_logger.debug("Group data are rooted at %r. Type: %r, %r, %r", None, root, type(root), type(COHO.workspace), str)

, installed it in py2 (only), and rerun the test in py2:

sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
<module>:
    Group data are rooted at '/home/king/.sage/temp/klap/16014/dir_M5saF0/'. Type: <type 'str'>, <type 'str'>, <type 'str'>

Hence, no bug. "Hooray".

Start a sage-with-py3-session and do

sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: H = CohomologyRing(64,14, from_scratch=True)
sage: H.make()

The hope was of course that it triggers something. But the test in py2 does not have a unicode problem. "Hooray".

But I am sure that the problem will pop up sooner or later.

comment:96 Changed 2 years ago by SimonKing

To make sure that this bug doesn't pop up again, I inserted a test whether the python version is less than (2,8) and root is of type unicode. In this case, I print some information and then raise a runtime error. I guess it's the best that I can do for now.

After recompiling and rerunning the above tests, they went fine again. But sooner or later it will return, and then the diagnostic tests I inserted will perhaps tell more clearly what is happening.

comment:97 Changed 2 years ago by SimonKing

Or another idea: Since it is Cython code, I could do cdef str root = ....

comment:98 Changed 2 years ago by SimonKing

Hooray!

The inserted test already showed something:

sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
Critical bug in python.
kwds.get('root', COHO.workspace) is <type 'str'>
COHO.workspace is <type 'str'>
str=<type 'unicode'>, unicode=<type 'unicode'>
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-a303ce041f27> in <module>()
----> 1 _ = COHO()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

RuntimeError: An explicit cast to <str> resulted in a unicode object

Aha! For some silly reason, we have str == unicode!!

EDIT: The above was printed like this:

        if (2, 8) > sys.version_info and isinstance(root, unicode):
            print("Critical bug in python.")
            print("kwds.get('root', COHO.workspace) is %r"%type(kwds.get('root', COHO.workspace)))
            print("COHO.workspace is %r"%type(COHO.workspace))
            print("str=%r, unicode=%r"%(str,unicode))
            raise RuntimeError("An explicit cast to <str> resulted in a unicode object")
Last edited 2 years ago by SimonKing (previous) (diff)

comment:99 Changed 2 years ago by SimonKing

How to cope with it? Shall I test upon importing of the module whether str == unicode in Python-2, and raise an error whose message tells to recompile?

comment:100 follow-up: Changed 2 years ago by dimpase

So, would this still pop up if right before running these tests you removed ~/.sage/ ?

comment:101 in reply to: ↑ 100 Changed 2 years ago by SimonKing

Replying to dimpase:

So, would this still pop up if right before running these tests you removed ~/.sage/ ?

See comment:94. There, it didn't help. Now, it doesn't either:

king@klap:~/Sage/git$ mv ~/.sage/ ~/.sageBAK
king@klap:~/Sage/git$ mkdir ~/.sage
king@klap:~/Sage/git$ sage/sage 
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta8, Release Date: 2019-08-25               │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Setting permissions of DOT_SAGE directory so only you can read and write it.
sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
Critical bug in python.
kwds.get('root', COHO.workspace) is <type 'str'>
COHO.workspace is <type 'str'>
str=<type 'unicode'>, unicode=<type 'unicode'>
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-a303ce041f27> in <module>()
----> 1 _ = COHO()

pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()

RuntimeError: An explicit cast to <str> resulted in a unicode object

comment:102 Changed 2 years ago by SimonKing

I wonder if the problem occurs because I am not installing the development versions of my package from an actual "package" (using sage -i p_group_cohomology), but directly from the sources. I.e., in the src folder of my development tree, I do

sage -python setup.py build
sage -pip install . --upgrade

I know that sage -python setup.py build does create .c files in the src folder, but since the changed module is recompiled both in py2 and py3, I thought that it doesn't matter. And for efficiency, I want that only modules are recompiled that have changed.

So, I would like to test if the problem persists when separating the py2- and py3 build folders. When doing sage -python setup.py build, is it somehow possible to prescribe in what folder the .c and .pyc files are created? Then, I could choose separate folders for py2 and py3.

comment:103 Changed 2 years ago by SimonKing

I am now putting cythonize(..., build_dir=os.path.join("build","c_files-{}.{}".format(version_info.major, version_info.minor))) into my setup.py. In that way, the c-files will be created in separate directories for both Sage versions.

Let's see if this is enough to solve the problem!

comment:104 Changed 2 years ago by SimonKing

PS: Even if it solves the problem, I will test upon import whether str is unicode in py2, and if it is, raise a RuntimeError advising to recompile. I hope this error will never pop up...

comment:105 Changed 2 years ago by SimonKing

After changing setup.py so that the .c files will be in different folders for different python versions, tests in cohomology.pyx pass both with py2 and py3 and then py2 again. So, I can now turn to modular_cohomology.pyx and finally init.py (which contains a long doc string, namely the "general overview" of the package).

comment:106 Changed 2 years ago by SimonKing

Hooray! When I ran sage -tp 2 --force-lib --long --warn-long 60.1, all tests passed both in python 2 and python 3. I don't know how significant the run time differences are. In any case, here are the details:

Python-3
========
Running doctests with ID 2019-09-25-00-50-30-78efa273.
Using --optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage
Sorting sources by runtime so that slower doctests are run first....
Doctesting 13 files using 2 threads.
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
    [1481 tests, 350.33 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cohomology.pyx
    [1445 tests, 363.25 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
Warning, slow doctest:
    T.explore_isomorphisms()        # long time
Test ran for 122.79 s
    [213 tests, 221.05 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/modular_cohomology.pyx
    [459 tests, 274.13 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/factory.py
    [229 tests, 126.99 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/__init__.py
    [343 tests, 96.76 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/dickson.pyx
    [28 tests, 1.53 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/auxiliaries.py
    [39 tests, 0.79 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/barcode.py
    [143 tests, 10.31 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution_bindings.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pyx
    [882 tests, 22.95 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 765.7 seconds
    cpu time: 939.6 seconds
    cumulative wall time: 1468.1 seconds

Python-2
========
Running doctests with ID 2019-09-25-01-04-47-fa1de7f4.
Git branch: matrix_implementation_keyword
Using --optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,p_group_cohomology,python2,sage
Sorting sources by runtime so that slower doctests are run first....
Doctesting 13 files using 2 threads.
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
    [1481 tests, 437.44 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cohomology.pyx
    [1445 tests, 465.15 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
Warning, slow doctest:
    T.explore_isomorphisms()        # long time
Test ran for 116.79 s
    [213 tests, 229.85 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/modular_cohomology.pyx
    [459 tests, 312.40 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/factory.py
    [229 tests, 124.88 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/__init__.py
    [343 tests, 96.86 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pyx
    [882 tests, 21.38 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/dickson.pyx
    [28 tests, 1.63 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/auxiliaries.py
    [39 tests, 0.78 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution_bindings.pxd
    [0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/barcode.py
    [143 tests, 14.90 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 873.5 seconds
    cpu time: 996.8 seconds
    cumulative wall time: 1705.3 seconds

Hence, tomorrow (i.e., in central European time, today in a few hours :-)) I will create a new package version and post it here for review.

comment:107 Changed 2 years ago by SimonKing

Something more to do: I think I should add a test showing what happens when moving the directory associated with a cohomology ring. The package can cope with moved data, based on some trickery in the attribute _default_filename that is defined by Sage's unpickler. I've turned _default_filename into a property (with getter, setter and deleter), and its documentation should contain the new test.

comment:108 Changed 2 years ago by SimonKing

  • Description modified (diff)

I'm afraid that another test failure popped up. So, I cannot post the new version yet.

Note: See TracTickets for help on using tickets.