#28414 closed defect (fixed)
p_group_cohomology does not build on Python3
Reported by:  Dima Pasechnik  Owned by:  

Priority:  major  Milestone:  sage9.0 
Component:  packages: optional  Keywords:  py3 
Cc:  Simon King  Merged in:  
Authors:  Simon King  Reviewers:  John Palmieri 
Report Upstream:  N/A  Work issues:  
Branch:  6c5dafc (Commits, GitHub, GitLab)  Commit:  
Dependencies:  #28444  Stopgaps: 
Description (last modified by )
The aim of this ticket is to provide a new version of the optional p_group_cohomology
spkg so that it builds and passes tests both with python2 and python3.
Hopefully the version number 3.3 (two times three) is appropriate.
Attachments (1)
Change History (167)
comment:2 Changed 3 years ago by
Simon, I have made some progress in converting this to Python 3. Let me know and I can email you some diff files.
comment:3 Changed 3 years ago by
Hi John, in fact Python3compatibility was one of the things that I considered for the next package version. So, if you already have a diff, I'd appreciate to see it!
comment:4 followups: 5 36 Changed 3 years ago by
I just sent email to your Jena address. I can get the package to build, but it doesn't work yet. An illustration of one problem:
sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj') /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/src/bin/sageipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead #!/usr/bin/env sagepython23 WARNING: Files on disk have been moved or are not writeable. > Will try to recover later. /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/repl/rich_output/display_manager.py:623: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_') 8gp3: Files on disk have been moved  trying to get things right  UnicodeDecodeError Traceback (most recent call last) <ipythoninput59bd7fbf5ad54> in <module>() > 1 load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj') /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/IPython/core/displayhook.py in __call__(self, result) 244 self.start_displayhook() 245 self.write_output_prompt() > 246 format_dict, md_dict = self.compute_format_data(result) 247 self.update_user_ns(result) 248 self.fill_exec_result(result) /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/IPython/core/displayhook.py in compute_format_data(self, result) 148 149 """ > 150 return self.shell.display_formatter.format(result) 151 152 # This can be set to True by the write_output_prompt method in a subclass /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/repl/display/formatter.py in format(self, obj, include, exclude) 198 """ 199 # First, use Sage rich output if there is any > 200 sage_format, sage_metadata = self.dm.displayhook(obj) 201 assert PLAIN_TEXT in sage_format, 'plain text is always present' 202 if not set(sage_format.keys()).issubset(self.default_mime()): /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/repl/rich_output/display_manager.py in displayhook(self, obj) 809 return 810 self._backend.set_underscore_variable(obj) > 811 plain_text, rich_output = self._rich_output_formatter(obj, dict()) 812 return self._backend.displayhook(plain_text, rich_output) 813 /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/repl/rich_output/display_manager.py in _rich_output_formatter(self, obj, rich_repr_kwds) 621 rich_output = None 622 plain_text = None > 623 has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_') 624 if has_rich_repr: 625 rich_output = self._call_rich_repr(obj, rich_repr_kwds) pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getattr__() /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)() 149 150 ## Load file by absolute filename > 151 with open(filename, 'rb') as fobj: 152 X = loads(fobj.read(), compress=compress) 153 try: /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)() 150 ## Load file by absolute filename 151 with open(filename, 'rb') as fobj: > 152 X = loads(fobj.read(), compress=compress) 153 try: 154 X._default_filename = os.path.abspath(filename) /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)() 967 968 unpickler = SageUnpickler(io.BytesIO(s)) > 969 return unpickler.load() 970 971 UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
comment:5 Changed 3 years ago by
Replying to jhpalmieri:
I just sent email to your Jena address. I can get the package to build, but it doesn't work yet.
Thank you very much, I received the patch.
An illustration of one problem:
sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj') /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage8.9.beta8/src/bin/sageipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead ... UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
That probably is harmless. I.e., I should use "warning", and apparently in some unpickling method I use "str" where it should be "bytes".
comment:6 Changed 3 years ago by
Yes, and I would like to track down the str
vs. bytes
issue. It would be nice if I could get basic functionality like CohomologyRing(8,3)
to work.
comment:7 followup: 8 Changed 3 years ago by
Changing logger.warn
(or coho_logger.warn
) to X.warning
fixes the deprecation message. I don't know about the str/bytes issue.
comment:8 Changed 3 years ago by
Replying to jhpalmieri:
Changing
logger.warn
(orcoho_logger.warn
) toX.warning
fixes the deprecation message. I don't know about the str/bytes issue.
I did change .warn(
into .warning(
everywhere (so, no need to include it in a diff).
Concerning str vs. byte: There is only one occurrence of cdef str
, and this gets assigned to the output of <some singular object>.name()
. Is that output really a string in python3? If not, it should be changed to cdef bytes
. However, so far the error in unpickling stuff looks mysterious to me, as the traceback doesn't mention where the problem actually occurs in my code.
comment:9 followup: 10 Changed 3 years ago by
Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?
comment:10 followup: 13 Changed 3 years ago by
Replying to jhpalmieri:
Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?
You mean Python 3 is severely backwards incompatible and cannot deal with old pickles? That would be a show stopper for Sage, I think.
Note that at some point in the past I inserted code to regenerate very old pickles and translate them automatically to a new format. However, I would certainly not like to regenerate several gigabytes of data in the web repository.
Note that the problem occurs together with a warning:
WARNING: Files on disk have been moved or are not writeable. > Will try to recover later. ... 8gp3: Files on disk have been moved  trying to get things right
Maybe that indicates where the problem comes from. So, let's recall how storing the modular cohomology ring of a prime power group is solved.
Very early in the package development I found that is is terribly slow to pack all data of a ring in a single file. It simply takes so long and requires so much memory (at least in some cases) that it is not feasible to automatically store intermediate results of a lengthy computation (which evidently is a reasonable thing to do).
My solution was to not store a ring in a single file but distribute the data in a folder. So, the actual .sobj pickle file only contains basic information but is not standalone. In particular, the p_group_cohomology spkg needs to know where the data folder is located, the .sobj file has to be in the data folder and also it needs write permission for otherwise it couldn't do a continuation of the cohomology ring's computation. What is stored in the web repository is a compressed tar file of the data folder (i.e., when the web repository is accessed, then the tar file is downloaded, unpacked, and then the cohomology ring is loaded from there).
By default, the data folder's location is composed from the location of the current workspace (which are global variables) and from the identifier of the group (which is stored in the .sobj file).
The warning mentioned above occurs when the data folder's default location cannot be accessed. And in that case, I do some trickery  and perhaps that trickery doesn't work in Python 3? Namely:
 When an object is obtained from reading a pickle file, then Python (at least Python 2) stores the original file location as an attribute of the object. However, that attribute is only available after successful unpickling.
 Therefore, when initially the location of the data folder cannot be determined, an unitialised cohomology ring is returned.
 As soon as the uninitialised cohomology ring is touched (I think constructing its repr is enough), initialisation is triggered. Namely, recall that the location of the .sobj file is stored as an attribute of the uninitialised ring and that the .sobj file is supposed to be located in the data folder  voilà, you can now determine the data folder and finish initialisation of the cohomology ring.
If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.
comment:11 followup: 12 Changed 3 years ago by
Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring)
ok in Python3?
comment:12 followup: 31 Changed 3 years ago by
Replying to SimonKing:
Question: How to test whether something has a type that qualifies it as a filename? Is
isinstance(self._property_dict['_need_new_root'],basestring)
ok in Python3?
Here is what I found in the "internetz" concerning how to test if something is a string in a way that works in python 2 and 3:
import six if isinstance(obj, six.string_types): print('obj is a string!')
So, I think I should fix this.
comment:13 followup: 88 Changed 3 years ago by
Replying to SimonKing:
If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.
Hooray:
# in Sagewithpython2 sage: P = CombinatorialFreeModule(ZZ,'x') sage: save(P,'../../pickle')
# BOTH with python2 and python3 sage: B = load('../../pickle.sobj') sage: B._default_filename '/home/king/Sage/pickle.sobj'
So, the old trick should still work.
comment:14 Changed 3 years ago by
Currently, I get the following behaviour:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_workspace(tmp_dir()) sage: H = CohomologyRing(8,3,from_scratch=True,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/' Computing basic setup for Small Group number 1 of order 2 Computing basic setup for Small Group number 2 of order 4 Computing basic setup for Small Group number 3 of order 8  TypeError Traceback (most recent call last) <ipythoninput3d1c5c11acd7a> in <module>() > 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug') /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False > 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: > 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__() TypeError: expected bytes, str found
and
sage: H = CohomologyRing(8,3,options='debug') Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj Creating symbolic links from /home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3 Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/' The state descriptor of the tobeunpickled ring is expected to be provided at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3/dat/State.sobj'  UnicodeDecodeError Traceback (most recent call last) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds) 1154 coho_options['@use_this_root@'] = root_workspace > 1155 OUT = load(os.path.join(root_workspace,file_name)) # realpath here? 1156 if '@use_this_root@' in coho_options: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)() 150 ## Load file by absolute filename > 151 with open(filename, 'rb') as fobj: 152 X = loads(fobj.read(), compress=compress) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)() 151 with open(filename, 'rb') as fobj: > 152 X = loads(fobj.read(), compress=compress) 153 try: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)() 968 unpickler = SageUnpickler(io.BytesIO(s)) > 969 return unpickler.load() 970 pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO_unpickle() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)() 150 ## Load file by absolute filename > 151 with open(filename, 'rb') as fobj: 152 X = loads(fobj.read(), compress=compress) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)() 151 with open(filename, 'rb') as fobj: > 152 X = loads(fobj.read(), compress=compress) 153 try: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)() 968 unpickler = SageUnpickler(io.BytesIO(s)) > 969 return unpickler.load() 970 UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128) During handling of the above exception, another exception occurred: OSError Traceback (most recent call last) <ipythoninput4927128be23f4> in <module>() > 1 H = CohomologyRing(Integer(8),Integer(3),options='debug') /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False > 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds) 1159 if '@use_this_root@' in coho_options: 1160 del coho_options['@use_this_root@'] > 1161 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_local_sources,file_name), msg)) 1162 ## 4. Search web repository 1163 elif kwds.get('websource')!=False and (not from_scratch): OSError: Saved data at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj are not readable: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
So, that's something I can now try to debug.
comment:15 followup: 16 Changed 3 years ago by
in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3
comment:16 Changed 3 years ago by
Replying to chapoton:
in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3
Thank you! I don't know yet whether or not I'll need them, but certainly it is good to know.
comment:17 Changed 3 years ago by
I found that this fails in RESL.__init__
:
tmprstem = os.path.join(res_folder,rstem) tmpgstem = os.path.join(gps_folder,gstem) self.Data = newResolWithGroupLoaded(tmprstem,tmpgstem,1)
Apparently tmprstem or tmpgstem are strings, but the C function newResolWithGroupLoaded
expects char *
 is there no automatic conversion from py3string to char*
?
comment:18 followup: 19 Changed 3 years ago by
Maybe try
self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)
..with no guarantee..
comment:19 Changed 3 years ago by
Replying to chapoton:
Maybe try
self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)..with no guarantee..
That is likely to work. Another potential solution would be to generally work with bytes. I.e., let self.gstem be a bytes and then do os.path.join(self.res_folder, b'A'+self.gstem+b'.sobj'))
etc.
What would be the preferred way?
comment:20 followup: 21 Changed 3 years ago by
beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes
EDIT:
indeed:
sage: os.path.join('home', b'jojo') ... TypeError: Can't mix strings and bytes in path components
comment:21 Changed 3 years ago by
Replying to chapoton:
beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes
EDIT:
indeed:
sage: os.path.join('home', b'jojo') ... TypeError: Can't mix strings and bytes in path components
This only shows that it cannot mix both types. In fact, if both are bytes then things work:
sage: os.path.join(b'home', b'foo') b'home/foo'
Probably it would be a long and tedious work to change everything to bytes. In principle I would be willing to do that work, but if there is a different solution involving less work and only a slight overhead, I'd prefer that.
Currently, using str_to_bytes
when passing a string to a C function does seem to work.
comment:22 Changed 3 years ago by
WTF? I just see that I have the following argument types of COCH.__init__
:
def __init__(self, PARENT, int n, char *Nick, L, ydeg=None, rdeg=None, is_polyrep=False)
I suppose Nick shouldn't be char*
but (if anything) str. Or better not type it at all.
comment:23 Changed 3 years ago by
I think I should also fix the following compiler warning (there are several similar):
pGroupCohomology/cohomology.c: In function ‘__pyx_pf_16pGroupCohomology_10cohomology_12explore_one_parameter’: pGroupCohomology/cohomology.c:17678:35: warning: comparison between signed and unsigned integer expressions [Wsigncompare] for (__pyx_t_7 = 0; __pyx_t_7 < __pyx_t_16; __pyx_t_7+=1) {
comment:24 Changed 3 years ago by
I guesss getting the following error is a progress, although it seems a bit strange to me:
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/' Computing basic setup for Small Group number 1 of order 2 Computing basic setup for Small Group number 2 of order 4 Computing basic setup for Small Group number 3 of order 8 > export action matrices Initialising maximal pelementary abelian subgroups Inserting SmallGroup(2,1) as a subgroup We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/' > export action matrices Unable to save basic ring setup Traceback (most recent call last): File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627) obj.save(filename=filename, compress=compress, **kwargs) AttributeError: 'tuple' object has no attribute 'save' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636) obj.save(filename=filename, compress=compress, **kwargs) File "sage/structure/sage_object.pyx", line 424, in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328) filename = _base_save(self, filename, compress=compress) File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318) with open(filename, 'wb') as fobj: File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253) fobj.write(_base_dumps(obj, compress=compress)) File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898) gherkin = SagePickler.dumps(obj) File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613) pickler.dump(obj) File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__ File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py", line 85, in safe_save save (obj, path) File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694) _base_save(obj, filename, compress=compress) File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318) with open(filename, 'wb') as fobj: File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253) fobj.write(_base_dumps(obj, compress=compress)) File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898) gherkin = SagePickler.dumps(obj) File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613) pickler.dump(obj) TypeError: can't pickle dict_items objects During handling of the above exception, another exception occurred: Traceback (most recent call last): File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627) obj.save(filename=filename, compress=compress, **kwargs) AttributeError: 'tuple' object has no attribute 'save' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py", line 1254, in _get_p_group_from_scratch safe_save(OUT, OUT.autosave_name()) File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py", line 85, in safe_save save (obj, path) File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694) _base_save(obj, filename, compress=compress) File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318) with open(filename, 'wb') as fobj: File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253) fobj.write(_base_dumps(obj, compress=compress)) File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898) gherkin = SagePickler.dumps(obj) File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613) pickler.dump(obj) File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__ File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py", line 85, in safe_save save (obj, path) File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694) _base_save(obj, filename, compress=compress) File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318) with open(filename, 'wb') as fobj: File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253) fobj.write(_base_dumps(obj, compress=compress)) File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898) gherkin = SagePickler.dumps(obj) File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613) pickler.dump(obj) TypeError: can't pickle dict_items objects Computing next term > rk P_02 = 1 We have to choose 1 new generator in degree 1 > There is 1 Duflot regular generator in degree 1 Summary: 0 relations and 1 generators in degree 1 Ring approximation computed out to degree 1! Storing approximation data  AttributeError Traceback (most recent call last) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)() 239 try: > 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save' During handling of the above exception, another exception occurred: AttributeError Traceback (most recent call last) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)() 239 try: > 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): AttributeError: 'dict_items' object has no attribute 'save' During handling of the above exception, another exception occurred: TypeError Traceback (most recent call last) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)() 239 try: > 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/structure/sage_object.pyx in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)() 423 > 424 filename = _base_save(self, filename, compress=compress) 425 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)() 178 > 179 with open(filename, 'wb') as fobj: 180 fobj.write(_base_dumps(obj, compress=compress)) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)() 179 with open(filename, 'wb') as fobj: > 180 fobj.write(_base_dumps(obj, compress=compress)) 181 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)() 256 > 257 gherkin = SagePickler.dumps(obj) 258 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)() 835 pickler = cls(buf, **kwargs) > 836 pickler.dump(obj) 837 return buf.getvalue() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py in safe_save(obj, path) 84 os.unlink(path) > 85 save (obj, path) 86 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)() 241 except (AttributeError, RuntimeError, TypeError): > 242 _base_save(obj, filename, compress=compress) 243 else: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)() 178 > 179 with open(filename, 'wb') as fobj: 180 fobj.write(_base_dumps(obj, compress=compress)) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)() 179 with open(filename, 'wb') as fobj: > 180 fobj.write(_base_dumps(obj, compress=compress)) 181 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)() 256 > 257 gherkin = SagePickler.dumps(obj) 258 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)() 835 pickler = cls(buf, **kwargs) > 836 pickler.dump(obj) 837 return buf.getvalue() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py in safe_save(obj, path) 84 os.unlink(path) > 85 save (obj, path) 86 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)() 241 except (AttributeError, RuntimeError, TypeError): > 242 _base_save(obj, filename, compress=compress) 243 else: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)() 178 > 179 with open(filename, 'wb') as fobj: 180 fobj.write(_base_dumps(obj, compress=compress)) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)() 179 with open(filename, 'wb') as fobj: > 180 fobj.write(_base_dumps(obj, compress=compress)) 181 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)() 256 > 257 gherkin = SagePickler.dumps(obj) 258 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)() 835 pickler = cls(buf, **kwargs) > 836 pickler.dump(obj) 837 return buf.getvalue() TypeError: can't pickle dict_items objects During handling of the above exception, another exception occurred: AttributeError Traceback (most recent call last) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)() 239 try: > 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save' During handling of the above exception, another exception occurred: AttributeError Traceback (most recent call last) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)() 239 try: > 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): AttributeError: 'dict_items' object has no attribute 'save' During handling of the above exception, another exception occurred: TypeError Traceback (most recent call last) <ipythoninput3d1c5c11acd7a> in <module>() > 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug') /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False > 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: > 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.make() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.next() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py in safe_save(obj, path) 83 if os.path.islink(path): 84 os.unlink(path) > 85 save (obj, path) 86 87 ################################### /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)() 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): > 242 _base_save(obj, filename, compress=compress) 243 else: 244 # Saving an object to an image file. /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)() 177 filename = _normalize_filename(filename) 178 > 179 with open(filename, 'wb') as fobj: 180 fobj.write(_base_dumps(obj, compress=compress)) 181 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)() 178 179 with open(filename, 'wb') as fobj: > 180 fobj.write(_base_dumps(obj, compress=compress)) 181 182 return filename /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)() 255 """ 256 > 257 gherkin = SagePickler.dumps(obj) 258 259 if compress: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)() 834 buf = io.BytesIO() 835 pickler = cls(buf, **kwargs) > 836 pickler.dump(obj) 837 return buf.getvalue() 838 pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py in safe_save(obj, path) 83 if os.path.islink(path): 84 os.unlink(path) > 85 save (obj, path) 86 87 ################################### /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)() 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): > 242 _base_save(obj, filename, compress=compress) 243 else: 244 # Saving an object to an image file. /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)() 177 filename = _normalize_filename(filename) 178 > 179 with open(filename, 'wb') as fobj: 180 fobj.write(_base_dumps(obj, compress=compress)) 181 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)() 178 179 with open(filename, 'wb') as fobj: > 180 fobj.write(_base_dumps(obj, compress=compress)) 181 182 return filename /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)() 255 """ 256 > 257 gherkin = SagePickler.dumps(obj) 258 259 if compress: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)() 834 buf = io.BytesIO() 835 pickler = cls(buf, **kwargs) > 836 pickler.dump(obj) 837 return buf.getvalue() 838 pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/auxiliaries.py in safe_save(obj, path) 83 if os.path.islink(path): 84 os.unlink(path) > 85 save (obj, path) 86 87 ################################### /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)() 240 obj.save(filename=filename, compress=compress, **kwargs) 241 except (AttributeError, RuntimeError, TypeError): > 242 _base_save(obj, filename, compress=compress) 243 else: 244 # Saving an object to an image file. /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)() 177 filename = _normalize_filename(filename) 178 > 179 with open(filename, 'wb') as fobj: 180 fobj.write(_base_dumps(obj, compress=compress)) 181 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)() 178 179 with open(filename, 'wb') as fobj: > 180 fobj.write(_base_dumps(obj, compress=compress)) 181 182 return filename /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)() 255 """ 256 > 257 gherkin = SagePickler.dumps(obj) 258 259 if compress: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)() 834 buf = io.BytesIO() 835 pickler = cls(buf, **kwargs) > 836 pickler.dump(obj) 837 return buf.getvalue() 838 TypeError: can't pickle dict_items objects
What are dict_items
, and why can't they be pickled?
comment:25 followup: 26 Changed 3 years ago by
If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).
You may need to wrap with list(d.items()) for pickling.
comment:26 followups: 28 32 Changed 3 years ago by
Replying to chapoton:
If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).
You may need to wrap with list(d.items()) for pickling.
Wouldn't that considerably slow things down, at least in Python2? After all, d.items()
is a list, and list(...)
applied to a list creates a (shallow) copy of that list.
comment:27 followup: 29 Changed 3 years ago by
There is a considerable progress now (after putting list
around .items()
):
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_workspace(tmp_dir()) sage: H = CohomologyRing(8,3,from_scratch=True,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/' Computing basic setup for Small Group number 1 of order 2 Computing basic setup for Small Group number 2 of order 4 Computing basic setup for Small Group number 3 of order 8 > export action matrices Initialising maximal pelementary abelian subgroups Inserting SmallGroup(2,1) as a subgroup We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/' > export action matrices Computing next term > rk P_02 = 1 We have to choose 1 new generator in degree 1 > There is 1 Duflot regular generator in degree 1 Summary: 0 relations and 1 generators in degree 1 Ring approximation computed out to degree 1! Storing approximation data Determine degree 2 standard monomials We got 1 standard monomials Monomial 0 > Candidate: c_1_0^2 > Express monomial as a Cochain Compute c_1_0*c_1_0 Compose chain maps R_2 > R_1 > R_0 Lift with Urbild Groebner basis in degree 1 load Urbild Groebner basis Decomposable cochain found There is no new generator in degree 2 Summary: 0 relations and 0 generators in degree 2 Ring approximation computed out to degree 2! Storing approximation data Inserting SmallGroup(4,2) as a subgroup We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/' > export action matrices  KeyError Traceback (most recent call last) <ipythoninput3d1c5c11acd7a> in <module>() > 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug') /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False > 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: > 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup() /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False > 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: > 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() KeyError: 'autoliftElAb'
The good news is that for the first time cohomology rings were computed in py3 (namely cohomology rings of elementary abelian groups).
comment:28 Changed 3 years ago by
Replying to SimonKing:
Replying to chapoton:
If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).
You may need to wrap with list(d.items()) for pickling.
Wouldn't that considerably slow things down, at least in Python2? After all,
d.items()
is a list, andlist(...)
applied to a list creates a (shallow) copy of that list.
You could use a try ... except
block to deal with the Python 2 case quickly. Sage will soon (I hope) default to Python 3, so I don't think this is worth it, though.
comment:29 Changed 3 years ago by
Replying to SimonKing:
There is a considerable progress now (after putting
list
around.items()
): ...KeyError: 'autoliftElAb'
Apparently I forgot to create a shallow copy in some places in which I temporarily stored the global options  it doesn't work to recreate a dict from its items in Py3, and that's why I got the key error above:
sage: D = {1:2} sage: I = D.items() sage: D.clear() sage: D.update(I) sage: D {}
In Py2, one would get
sage: D {1: 2}
comment:30 Changed 3 years ago by
Hooray!
I am now in a position to completely compute CohomologyRing(8,3)
from scratch. Unfortunately reading old data doesn't work yet, but certainly it is a progress :)
comment:31 followup: 33 Changed 3 years ago by
Replying to SimonKing:
import six
Please don't do this in Cython files. There is almost never a reason to use six
in Cython since Cython handles most of the Python 2/3 incompatibilities for you.
comment:32 followup: 34 Changed 3 years ago by
comment:33 followup: 35 Changed 3 years ago by
Replying to jdemeyer:
Replying to SimonKing:
import sixPlease don't do this in Cython files. There is almost never a reason to use
six
in Cython since Cython handles most of the Python 2/3 incompatibilities for you.
I am not dealing with incompatibilities in that case. I import six in order to be able to safely test for string types. How to, if I don't use six?
comment:34 Changed 3 years ago by
Replying to jdemeyer:
Use
list(d.iteritems())
in Cython.
Would this also work fast if Cython doesn't know that d is a dict? Should I do
list(<dict>(self._D).iteritems())
, for instance?
comment:35 Changed 3 years ago by
Replying to SimonKing:
I import six in order to be able to safely test for string types. How to, if I don't use six?
First of all, you should check whether you really need to test for "string types". Can you use duck typing instead? If for some reason, duck typing is hard, then it depends what you mean precisely with "string types".
Since you were mentioning filenames: isinstance(s, (bytes, unicode))
are valid types for filenames but in Python 2 and Python 3. Note that Python 3 also has https://docs.python.org/3/library/pathlib.html, a custom class to deal specifically with filenames.
comment:36 Changed 3 years ago by
Replying to jhpalmieri:
I can get the package to build, but it doesn't work yet. An illustration of one problem:
sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj') ... UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
I tried to track down that problem, and currently it seems to me that it is due to a very bad backwards incompatibility in Python. You can reproduce it as follows:
 Download attachment:State.sobj
 Install the optional meataxe spkg (this may involve
sage b
, so that some optional module in the Sage library is built)  Define some unpickle helper:
sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle sage: class unpickle_old_mtx: ....: def __call__(self, *args, **kwds): ....: return mtx_unpickle(*args, **kwds) ....: sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
Now, in SagewithPy2, loading the file works:
sage: X = load('/home/king/Projekte/coho/tests/State.sobj') sage: X ([[1, [(2, 1), [ [1 0 0 0 0 0 0 0] [0 0 0 1 1 1 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_2'] ]]], [2, [(4, 2), [ [1 0 0 0 0 0 0 0] [0 0 0 1 1 1 1 1] [0 1 0 0 0 0 0 0] [0 0 0 0 0 1 0 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_2'] ]]], [3, [(4, 2), [ [1 0 0 0 0 0 0 0] [0 0 0 1 1 1 1 1] [0 0 1 0 0 0 0 0] [0 0 0 0 0 0 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_2'] ]]]], [2, 1, 1], 1, 1, '8gp3/', ['b_1_0*b_1_1'], '8gp3/dat/', [((4, 2), '/home/king/.sage/pGroupCohomology/db/4gp2/H4gp2'), ((2, 1), '/home/king/.sage/pGroupCohomology/db/2gp1/H2gp1')], '8gp3/dat/', [], '8gp3/sgp/', 2, [2, 3], [2, 2], 2, 2, ['b_1_0*b_1_1'], [[2, 'c_2_2', [0 0 1], 1, 0], [1, 'b_1_0', [1 0], 0, 0], [1, 'b_1_1', [0 1], 0, 0]], [[0, [['1', ['1']]]], [1, [['b_1_0', ['b_1_0']], ['b_1_1', ['b_1_1']]]], [2, [['c_2_2', ['c_2_2']], ['b_1_0', ['b_1_0^2']], ['b_1_1', ['b_1_0*b_1_1', 'b_1_1^2']]]]], {}, 0.24, True, 'Monomials', 2, None, '', 0, '8gp3', '8gp3/dat/R8gp3.sobj', 1, ['0'], ['b_1_0 + b_1_1'], None, [('_max_module_deg', 0), ('sgpDickson', [((4, 2), [[1, 'c_1_1', [0 1]]])]), ('useFactorization', True), ('useElimination', False), ('GroupDescr', 'Dihedral group of order 8'), ('_key', ((8, 3), '/home/king/SPKG/database/8gp3/dat/State')), ('auto', 4), ('Restriction_2', ['c_1_0*c_1_1+c_1_0^2', 'c_1_1', '0']), ('GroupName', 'D8'), ('DicksonExp', 3), ('_parameters_do_exist', True), ('KeepBases', None), ('completeGroebner', True), ('_SymondsTestdata', ['c_2_2', 'b_1_1', 'b_1_0']), ('_parameters_for_criterion', ['c_2_2', 'b_1_1', 'b_1_0']), ('root', '/home/king/SPKG/database/'), ('Restriction_3', ['c_1_0*c_1_1+c_1_0^2', '0', 'c_1_1']), ('_method', 'Symonds')], [(('dependent_parameters',), [['c_2_2', 'b_1_1', 'b_1_0'], 2])])
However, SagewithPy3 yields the error found by John:
sage: X = load('/home/king/Projekte/coho/tests/State.sobj')  UnicodeDecodeError Traceback (most recent call last) <ipythoninput474e7e91f9969> in <module>() > 1 X = load('/home/king/Projekte/coho/tests/State.sobj') /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)() 149 150 ## Load file by absolute filename > 151 with open(filename, 'rb') as fobj: 152 X = loads(fobj.read(), compress=compress) 153 try: /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)() 150 ## Load file by absolute filename 151 with open(filename, 'rb') as fobj: > 152 X = loads(fobj.read(), compress=compress) 153 try: 154 X._default_filename = os.path.abspath(filename) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)() 967 968 unpickler = SageUnpickler(io.BytesIO(s)) > 969 return unpickler.load() 970 971 UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
And the point is: Python3 even is unable to read the pickle file as a string! In Python2, reading the file is no problem at all.
king@klap:~$ ~/Sage/git/py3/sage python Python 3.7.3 (default, Aug 27 2019, 23:22:23) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> f = open('/home/king/Projekte/coho/tests/State.sobj') >>> f.read() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/king/Sage/git/py3/local/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 1: invalid start byte >>> king@klap:~$ ~/Sage/git/sage/sage python Python 2.7.15 (default, Jul 26 2019, 11:49:43) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> f = open('/home/king/Projekte/coho/tests/State.sobj') >>> f.read() 'x\x9c\x95U\x8bZ\x1bE\x14&!\x91\x92J\x8b\x82V\xbc\x94x\xedXl\x96\x04\xaah\x95B\x0b\x05]\xc1\x16\x18\xad\x97\xe9:\xd9\x1dv\xb7d3{v7\nU,^\xd8B\xbd\xd6\xfb\xfd\xae\xcf\xd0\xa7\xf0E\t\xcfl\xb2\x08\xf5\xab_I\xbe\x9d\xefd\xe6?\xe7\xfc\xe7?g6kY\xc2 \x83O\x96\xe8\x19\x06\xed\xc4\x0c\xb9J\x81[\xb7\xc3\x92[\x8f\x84\x82\x82\xc7\x97\x84\x91\xfe\x80\x1c\xcdT\xd6\xe7 \xef\xa0QF\xe3\xa6\x18:\x18\xec!\xc4\xf4\xa7\x02\xd9\xf0OJGz\xb2&\xed\x95\x92\x17\x17f\x16\xce\x1a\x8d\xba\xef\x9aK5a\x985\x1e\x86\x05\xe8\x94PX\x85\xbd\xf4\x80a\x84Q\x18\x8b2H1\x98\xd80\xe0\xe6\x8d\xb0J\xf4,~\xf7\xd0\xecZ\xffF4\x07]\x0c\xf6\x11\xa7\xa0v3\xb8\x9bY\xdb\xc4\xcd\xfdt@\xc3lB[B?m\xfe\xb4>\xa5Y<\xe2U\x1e\nm\xc4\xf6\x87\xb4\xd0\xf6\x13#\xb4\xcbF\x19\xbaw\x03\xaf\xc0\x02?\x0cn\xc5\x9c\x0cz\x88*x\x18\x0b\xeeuZ\x12\xdc\x16\xc3\xed\x0c\x0e\xb4X\xe5\x90Un\xad\x7f,\xaf\xd8\xde\xc1\xa0\xef?l\xef\xbc\xf1\xf4\x15d{\xd7n\xe0\x15\xb8\xbb\xc9\xf6\x1e\xa2\xb738\xb8\xc5\xb6?e[\x8c\xe1^\x06\xf7\xedd[lWl\xefg\xf0\x00qvAo\x08\xe9=\xb8\x1bx\x05\x1e\x12M~\x87\x9a\x9adD:?$5\x1e\xa6\xf9\xc4\x8f\xc1a\xba\xb7j\x94\x8d\xc1\xc3j\xc3\x00\xa7\x9d\xc9\t\xc6\xc7\xd3G\x88\xd3\xeb\xa0\xf6G\xa8\xb6\x8d@I\r\xaev\xed\x08jVU\x1b\xb6\xfd\x8a6\xad\xd6\x18JN\xde\xc1\x81\xd5n\xd0\xb5b\xfbemZ\xad1\x0c\x8a\xed,Z\xb6\xaaQMG\x99\xa42W\x941\x84\xc6\x10\x16;\xbc\xb5}45\x1e\x15\xceQ\xe5\xf1\xd8\xce\x1aG8\x83\xc7\xf1\x1e>\x81\xf2\xd0\xbci\xa8\x8e\x1eK\x07\xa8\x9df\x8aj\x80\x9eD\xbb\r\xc3>\x85\x97\x95\xe6\x13w\x18MA\xd9\xd6\x94\x1d\xd7\xdb\x12\xd0\xd8\x16\xa8\x0c\xe3\xdb@c\nt"\x01!\xea$\xe6\x9c z\x1b\x83I\x06\xa7\x08v\x82\xc1\x14\xae\\p<\x9dN\xde\nO#\xe6\x19\xe2\x8c2\xd0\xd3\xac\xcf\xaa\xd3\x19\xe2\x8c3\x98M\x93<\xc7\x93\x06\x9fN\xae\xcb\x19\xf4\x99#\xce1\x06\xf3i9\x0b\xca\x87\x924\xc4\xf3\x0c^\xa0\x1d\x89}\xae\x02g\xd5\xe1\x8b$\x8d\xf5\x12\x83\x97\xc9N\x85^i\x82\xcb\x08f\xc9,\xad\xc2\xb9\xa9\xe3\x7f]=\xf3\xf7\xfa\xc1\xab\x1b\xb4sF\xd6\xa5\xe7\xf2Z\x08F\xaa\xf5\xab\xb3\x14\xcb\xa49\xd5*\xe0\xb4g\xab}s\xca*\x85\xb2z^\xd5W\xa5\x99A\x94\xdf\xa4]I\xbe\xe2@\xb1\x99\xd1\xe2\xb3\x0c\x04\xa1\xfb\r\x8f/\x1b\x9e\xb4\x1a\xf8\n\xb3\x84\r\x8b*\xc1 &\xb0cph\x01\x87`\x02\xdf]\xa1\xac\x83\xcb\xe0\xbcs\x84\xc1\x12\x83Z\xa2\xbf\x99D\xf2\xae\xd5\xbf.x\x0c\x12\x1f\x9fv7Bq\x8a\x9b\x91\x0c\xdc\x0b<r1\x08l\xc4\x10\xd0}\xb8?Ys=\xb7\xde\xdc\r7c\x88h!\x99\xd2\t\x11\x9a\x014h\xdf\x84\xeb\x08+\xe0\xb5\xa2\xad\xb6\x8br\xb1(\x03K\x04\xc5\x11x\x86\xd7i\xceX\x12+\xb0\xac\xe8\x8e \xdd\x95t6/\xc4\xf0\x06=\xf4\xff\xd7W\xe94\x1f\xf1H\xc4\xf0f\x0c\xab4\xc7\x1b\x91\x84\xb7\xf4\\\x0c\x17i\xd7\x9c\x08\xa3\xc05\x153l\xed\x1a\x83\xb7\t\xed1\x93v%%\x0f\x98\xad\xc6\xbe\x93j\xf0.J\x86\xb1\xde\xa3\x9dI\t\xb3\xdc\x13\xb0N\xb3\x13#\x10\xc7p\x89\x16Z\x12N.\xfb\xb0\xa1\xb7\xc7\xb0I{\r\x9f\x07\x08\x8bD\x10\x1a\x964\xc4\xb2\x1bFp\x19\xd5y\x9fv\xeaB\xf8\'\x90l\x08\x1f\xcc\xc6\xf0!\xed6\xa5\xe7\xd7\x10\x8b\xd1E\xb5.\x02\xf8\x08\x81\x1f\xd3nc~\xc5\x93u+\\@\xc6\xaaB\xf8\x84\xc1\x15\x9cLg\xdc\x19E>\x9f\xd2\xbe\xedi\xd4_\x91\x19\xb8h+\xcd?c\xf0\xf9\xbf\xd0/h.\x902\x82/i\xdfu\xa5\x83\xafb\xf8z\xa7<C\xf0\r\x83o\xaf#\xcfw\xa8J\xaa\xd0\xf7\x98\xe3\x07\xdaa \x15GZ\xf0#\xedhQ\x87\x9fb\xf8\x19\xef\xc7/\xb4\xd7\x12\xbe\xa8[\xa2\x1em#\r\xbf\xae\xc3o\x0c~\'\xce\x15=\x8b1\xfe\xe0\x11\xfcY\xfa\x07\x14\x1f\x95\xac'
comment:37 followup: 38 Changed 3 years ago by
This works fine in python3:
sage: with open('State.sobj', 'rb') as f: ....: f.read()
comment:38 Changed 3 years ago by
Replying to chapoton:
This works fine in python3:
sage: with open('State.sobj', 'rb') as f: ....: f.read()
Do I understand that it works because b
means "binary"? But of course the real question is why Python 3 cannot unpickle from that file.
I tried to dig a bit deeper into Sage's load
function (I guess the error is hidden there), and indeed load()
uses with open(filename, 'rb') as fobj:
. So, the unpickling problem is something else:
sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle sage: filename = '/home/king/Projekte/coho/tests/State.sobj' sage: compress = True sage: class unpickle_old_mtx: ....: def __call__(self, *args, **kwds): ....: return mtx_unpickle(*args, **kwds) ....: sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx) # The following is copied from Sage's load() function. sage: ## Load file by absolute filename ....: with open(filename, 'rb') as fobj: ....: X = loads(fobj.read(), compress=compress) ....: try: ....: X._default_filename = os.path.abspath(filename) ....: except AttributeError: ....: pass ....:  UnicodeDecodeError Traceback (most recent call last) <ipythoninput9654643a51ef6> in <module>() 1 ## Load file by absolute filename 2 with open(filename, 'rb') as fobj: > 3 X = loads(fobj.read(), compress=compress) 4 try: 5 X._default_filename = os.path.abspath(filename) /home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)() 967 968 unpickler = SageUnpickler(io.BytesIO(s)) > 969 return unpickler.load() 970 971 UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
So, now we see that the error occurs in SageUnpickler(io.BytesIO(x)).load()
.
comment:39 followup: 40 Changed 3 years ago by
I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx
when one forgets to register_unpickle_override(...)
seems to indicate that the error occurs when an mtx matrix is unpickled.
So, I am now trying to find if this is really the case or not.
comment:40 followup: 41 Changed 3 years ago by
Replying to SimonKing:
I find it very difficult to track down. But the fact that one does see a mentioning of
pGroupCohomology.mtx
when one forgets toregister_unpickle_override(...)
seems to indicate that the error occurs when an mtx matrix is unpickled.So, I am now trying to find if this is really the case or not.
It isn't. I inserted a print statement at the beginning of mtx_unpickle
, but it isn't shown before the error triggered by load('/home/king/Projekte/coho/tests/State.sobj')
is raised.
I'll open a ticket for that problem.
comment:41 Changed 3 years ago by
comment:42 Changed 3 years ago by
Dependencies:  → #28444 

comment:43 Changed 3 years ago by
It seems that #28444 fixes the problem for p_group_cohomology on Python3. With the current unpublished code for the package, I get
$ ./sage ┌────────────────────────────────────────────────────────────────────┐ │ SageMath version 8.9.beta9, Release Date: 20190902 │ │ Using Python 3.7.3. Type "help()" for help. │ └────────────────────────────────────────────────────────────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Warning: this is a prerelease version, and it may be unstable. ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_workspace(tmp_dir()) sage: H = CohomologyRing(8,3,options='debug') Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj Creating symbolic links from /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3 Get original root and GStem /home/king/SPKG/database 8gp3 now, StateFile is /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj Try to get (8, 3) from cache Group data are rooted at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/' The state descriptor of the tobeunpickled ring is expected to be provided at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj' Setting state > export action matrices Differential reloaded > rk P_02 = 3 Differential reloaded > rk P_03 = 4 Import monomials Checking compatibility of SmallGroups library and stored cohomology ring sage: print(H) Cohomology ring of Dihedral group of order 8 with coefficients in GF(2) Computation complete Minimal list of generators: [c_2_2: 2Cocycle in H^*(D8; GF(2)), b_1_0: 1Cocycle in H^*(D8; GF(2)), b_1_1: 1Cocycle in H^*(D8; GF(2))] Minimal list of algebraic relations: [b_1_0*b_1_1]
Hooray...
comment:44 Changed 3 years ago by
Strangely, while I can compute some cohomology rings, the following computation fails in py3:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: H = CohomologyRing(32,4) sage: H.make()
Reason: Automatic saving does not work, and apparently it is because H.__getstate__()
returns Singular elements, which cannot be pickled. It seems I did take care of it, as H.__getstate__()
in py2 only returns the class sage.interfaces.singular.SingularElement
and not the instance, but for some reasons it doesn't in py3.
comment:45 Changed 3 years ago by
Question: Has the mechanism of tab completion changed between py2 and py3? I notice that tab completion on a cohomology ring works in py2, but doesn't work (no error, but no completion either) in py3.
comment:46 Changed 3 years ago by
tab completion is done by ipython, which should be upgraded (they've switched to Python 3 only, so we're using an old version, which in turn might lag in its Python 3 support).
I suggest not to worry about it now.
comment:47 Changed 3 years ago by
Aha! The culprit for pickle problem is pGroupCohomology.cohomology.pickle_gap_data
, which inspite of its name also takes care of singular elements. In Py2:
sage: self = CohomologyRing(32,4) sage: self.make() sage: G = self._decorator_cache['_parameter_restrictions',frozenset({'c_2_2'}),('radical', True)] sage: G [[(polynomial ring, over a field, global ordering // coefficients: ZZ/2 // number of vars : 2 // block 1 : ordering M // : names c_1_0 c_1_1 // : weights 1 1 // : weights 1 0 // block 2 : ordering C, <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca690>) failed: AttributeError: 'NoneType' object has no attribute 'group'>, <repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca500>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)], 2] sage: pickle_gap_data(G) [[(<class 'sage.interfaces.singular.SingularElement'>, <class 'sage.interfaces.singular.SingularElement'>, <class 'sage.interfaces.singular.SingularElement'>)], 2]
In Py3:
sage: pickle_gap_data(G) [[(polynomial ring, over a field, global ordering // coefficients: ZZ/2 // number of vars : 2 // block 1 : ordering M // : names c_1_0 c_1_1 // : weights 1 1 // : weights 1 0 // block 2 : ordering C, <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86a20>) failed: AttributeError: 'NoneType' object has no attribute 'group'>, <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86dc8>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)], 2]
comment:48 Changed 3 years ago by
Ouch. A nasty thing.
In pickle_gap_data
, I test if the input is an iterable class, and if it is and if the class is a builtin, then the pickle_gap_data
traverses the items of the input and creates from the individual results a modified copy of the input. In code:
try: I = iter(G) except: return G if isinstance(G,dict): return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items()) if getattr(type(G), '__module__', None) == '__builtin__': return type(G)(pickle_gap_data(X) for X in I) return G
The problem is that in Py3 we get
sage: getattr(list, '__module__', None) 'builtins'
Question: How to test in a py2/py3 invariant way whether a type is a builtin?
That asked, are there more iterable builtins than list, tuple, str/bytes and dict? I guess it is safer to just explicitly test for list and tuple. EDIT Like this:
if isinstance(G, dict): return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items()) if isinstance(G, (list, tuple)): return type(G)(pickle_gap_data(X) for X in G) return G
comment:49 followup: 50 Changed 3 years ago by
An iterable is an object with an __iter__
method or/and __getitem__
method defined.
So perhaps you can test for their presense. (These methods may be broken, but OK...)
comment:50 Changed 3 years ago by
Replying to dimpase:
An iterable is an object with an
__iter__
method or/and__getitem__
method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)
You misunderstood. The problem was not how to test if something is iterable but how to test if that iterable is a builtin. But, as I said in my previous post, probably this isn't the actual problem anyway. What I want is to deal with strings, lists, tuples and dicts. Period. So, I could be more explicit (and I already tested that the error from comment:44 can be fixed in that way).
comment:51 Changed 3 years ago by
In Python2, one implemented c.__nonzero__()
and then bool(c)
would work. It seems that in Python3, something else needs to be done. Is it perhaps the case that I need to implement __bool__
?
What should I do to make stuff work both in py2 and py3? Implement both __bool__
and __nonzero__
? Or would it be enough to replace __nonzero__
with __bool__
?
comment:52 Changed 3 years ago by
comment:53 followup: 54 Changed 3 years ago by
There is a problem with the logging module: In Python2, I get
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: P = CohomologyRing(9,2,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/19448/dir_0MUJxK/' Computing basic setup for Small Group number 1 of order 3 Computing basic setup for Small Group number 2 of order 9 Resolution of GF(3)[9gp2]: > export action matrices H^*(SmallGroup(9,2); GF(3)): Initialising maximal pelementary abelian subgroups
whereas in Python3 I get
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: P = CohomologyRing(9,2,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/18649/dir_9bz1_9bh/' Computing basic setup for Small Group number 1 of order 3 Computing basic setup for Small Group number 2 of order 9 > export action matrices Initialising maximal pelementary abelian subgroups
So, the indentation is gone, and (whats worse) the logging doesn't indicate what object the log comes from (which I found very useful for debugging, as in the computation of a single cohomology ring often many other rings are involved).
Can someone please give me a pointer to a page explaining the differences between py2 and py3 in the logging module?
comment:54 Changed 3 years ago by
Replying to SimonKing:
Can someone please give me a pointer to a page explaining the differences between py2 and py3 in the logging module?
By looking at the code of the logging module in Python2 and Python3, I see where the problem comes from. In the following, self
denotes an instance of pGroupCohomology.auxiliary.CohoFormatter
, which subclasses loggin.Formatter
.
In Python2, logging.Formatter.format(self, record)
uses self._fmt
to format the output. In Python3, self._fmt
is not used. Instead, self._style._fmt
is consulted, where self._style
is one of the styles in logging._STYLES
.
The point is: So far, my custom self.format
modifies self._fmt
and then calls the default .format()
. However, in Python3, when calling the default .format()
, the attribute self._fmt
plays no role, but this is where I currently put the format into.
Simple way out: Do not call logging.Formatter.format
in CohoFormatter.format
.
But I wonder: Is there a recommended way to make the logging format change from call to call?
By this, I mean: Make the output of CohoFormatter.format(self, record)
depend on record.args[0]
, in the sense that the output makes visible whether record.args[0]
is the same object as in the previous call to CohoFormatter.format
.
Or, more radically: What kind of logging approach would you recommend, what do you find most easy to comprehend/parse as a human (I am not interested in automatic parsing of the logging messages)?
I somehow like the current approach, which is: Assume that certain objects (say, ZZ
and RR
) are involved in a computation, and each object may issue a logging message. Then, stream of messages is arranged in blocks of messages coming from a single object, each block being prepended by an identifier of the object. I.e., if ZZ
gives two messages, then RR
gives three messages and then again ZZ
gives a message, the output would be
Integer Ring: message 1 message 2 Rational Field: message 3 message 4 message 5 Integer Ring: message 6
 Do you like this thematic arrangement of messages (which is provided by
coho_logger
in Python 2, but fails in Python3)?  If yes: Can you recommend a clean and easy way to achieve it both in Python 2 and Python 3? Ideally, one should still be able to include time information in the logging, which currently is not supported.
 If no: What logging format would you recommend?
comment:55 followup: 56 Changed 3 years ago by
Instead of setting self._fmt
, can you instead run self.basicConfig(format=FORMAT)
? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#userdefinedobjects), so maybe you can make use of that somehow.
comment:56 Changed 3 years ago by
Replying to jhpalmieri:
Instead of setting
self._fmt
, can you instead runself.basicConfig(format=FORMAT)
? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#userdefinedobjects), so maybe you can make use of that somehow.
From these links, I get the impression that each call to the logger results in the creation of a new formatter. Is that really the case?
comment:57 followup: 58 Changed 3 years ago by
I don't know, I only looked at the links, too. It's worth trying.
(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)
comment:58 Changed 3 years ago by
Replying to jhpalmieri:
I don't know, I only looked at the links, too. It's worth trying.
(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)
Sure. But I am unsure how a change of formatter could solve the problem without impact on performance.
If logging is in "warn" mode, then there are of course still silent calls to coho_logger.debug
. I want that (1) these calls are cheap (i.e., they shouldn't involve creation of Python objects) and (2) the message blocks should not take into account silent messages, i.e., a block shouldn't be closed if a silent message comes from a different object.
Of course, it should be that what I do with the formatter should rather be done with a handler, or maybe a filter. So, I will try to understand what exactly what handlers and filters do.
And, as in point 3. of comment:54: Would you recommend a different formatting of log messages?
comment:59 Changed 3 years ago by
Currently I am reading logging HOWTO and logging Cookbook, and at the moment it seems to me that I want to use logging.LoggingAdapter
, because it allows to add contextual information (which is what I'd like to use, unless people tell me a nicer way log format).
comment:60 Changed 3 years ago by
I think I now know what to do: The customisation should be done in a Formatter, i.e., the existing solution was a good approach. Only the format()
method should be different: Before calling super(CohoFormatter, self).format(record)
, I should add some attribute to the record
that is then used in the formatter's format string.
comment:61 Changed 3 years ago by
I think I solved the problem. I can now optionally add time information to the log, such as here:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: CohomologyRing.global_options('time') sage: H = CohomologyRing(720,763,prime=3) 20190918 20:54:20,277: Try to compute a Sylow 3subgroup 20190918 20:54:20,283: Try to find the SmallGroups address of the Sylow subgroup 20190918 20:54:20,288: We compute this cohomology ring from scratch 20190918 20:54:20,289: Group data are rooted at '/home/king/.sage/temp/klap/962/dir_tnhvqeq7/' 20190918 20:54:20,293: Computing basic setup for Small Group number 1 of order 3 20190918 20:54:20,405: Computing basic setup for Small Group number 2 of order 9 Resolution of GF(3)[9gp2]: 20190918 20:54:20,466: > export action matrices H^*(SmallGroup(9,2); GF(3)): 20190918 20:54:20,517: Initialising maximal pelementary abelian subgroups 20190918 20:54:20,555: Computing intermediate subgroup 20190918 20:54:20,567: Try to find the SmallGroups address of the intermediate subgroup 20190918 20:54:20,603: Computing group order 20190918 20:54:20,604: The group is of order 72 20190918 20:54:20,609: Computing intermediate subgroup Resolution of GF(3)[SmallGroup(9,2)]: 20190918 20:54:20,625: Computing next term 20190918 20:54:20,644: > rk P_02 = 3 H^*(SmallGroup(9,2); GF(3)): 20190918 20:54:20,647: We have to choose 2 new generators in degree 1 20190918 20:54:20,648: > There are 2 nilpotent generators in degree 1 ...
However, what I'd find even nicer if the sum of the cputimes of Sage and Singular since start of logging was shown, not just the walltime, as above.
Note that the indentation has changed, which means that I will have to change various doc tests, but that will be doable.
comment:62 Changed 3 years ago by
Sigh.
After solving the log problem, I find that meanwhile the code ONLY works in Python3, not Python2.
First problem:
class FOO(object): def __bool__(self): return something __nonzero__ = __bool__
works in Py3, but not in Py2, although it was recommended in some stackoverflow discussion. In Py2 it results in NameError: name '__bool__' is not defined
. How to fix it without code duplication?
Second problem:
/home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.py in <module>() 47 import re,os 48 > 49 import urllib.request, urllib.error 50 import tarfile 51 import logging ImportError: No module named request
How to deal with urllib in a py2/py3compatible way?
comment:63 followup: 64 Changed 3 years ago by
For the urllib
issue: http://python3porting.com/noconv.html#importerrors suggests code like
try: from urllib.request import ... except ImportError: from urllib import ...
I can't reproduce the first problem. With Python 2, if I do
sage: class FOO(object): ....: def __bool__(self): ....: return something ....: __nonzero__ = __bool__ ....: sage: a = FOO() sage: a.__nonzero__()  NameError Traceback (most recent call last) <ipythoninput374dab0bd3f82> in <module>() > 1 a.__nonzero__() <ipythoninput1ec93abb3d539> in __bool__(self) 1 class FOO(object): 2 def __bool__(self): > 3 return something 4 __nonzero__ = __bool__ 5 NameError: global name 'something' is not defined
so a.__nonzero__()
is certainly calling __bool__
. If I change the return
line to something sensible, it works.
comment:64 Changed 3 years ago by
Replying to jhpalmieri:
For the
urllib
issue: http://python3porting.com/noconv.html#importerrors suggests code liketry: from urllib.request import ... except ImportError: from urllib import ...
OK, thank you.
I can't reproduce the first problem. With Python 2, if I do
The "something" was of course figuratively. Anyway, the problem seems to be that I am using a cdef class here:. In Sagewithpy2:
sage: cython(''' ....: class Foo: ....: def __bool__(self): ....: return True ....: __nonzero__ = __bool__ ....: ''') sage: cython(''' ....: cdef class Foo: ....: def __bool__(self): ....: return True ....: __nonzero__ = __bool__ ....: ''') ERROR:root:An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multiline string', (1, 0))  NameError Traceback (most recent call last) <ipythoninput291e48d391924> in <module>() 4 return True 5 __nonzero__ = __bool__ > 6 ''') /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/lazy_import.pyx in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3697)() 352 True 353 """ > 354 return self.get_object()(*args, **kwds) 355 356 def __repr__(self): /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/cython.pyc in cython_compile(code, **kwds) 649 with open(tmpfile,'w') as f: 650 f.write(code) > 651 return cython_import_all(tmpfile, get_globals(), **kwds) /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/cython.pyc in cython_import_all(filename, globals, **kwds) 539 code 540 """ > 541 m = cython_import(filename, **kwds) 542 for k, x in iteritems(m.__dict__): 543 if k[0] != '_': /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/cython.pyc in cython_import(filename, **kwds) 519 try: 520 sys.path.append(build_dir) > 521 return builtins.__import__(name) 522 finally: 523 sys.path = oldpath /home/king/.sage/temp/klap/23274/spyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0.pyx in init _home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0() 3 def __bool__(self): 4 return True > 5 __nonzero__ = __bool__ NameError: name '__bool__' is not defined
whereas in Sagewithpy3 I get
sage: cython(''' ....: class Foo: ....: def __bool__(self): ....: return True ....: __nonzero__ = __bool__ ....: ''') sage: cython(''' ....: cdef class Foo: ....: def __bool__(self): ....: return True ....: __nonzero__ = __bool__ ....: ''')
So, what to do?
comment:65 followup: 66 Changed 3 years ago by
I don't know Cython, but replacing __nonzero__ = __bool__
with
def __nonzero__(self): return self.__bool__()
seems to work.
comment:66 Changed 3 years ago by
Replying to jhpalmieri:
I don't know Cython, but replacing
__nonzero__ = __bool__
withdef __nonzero__(self): return self.__bool__()seems to work.
Sure. that's what I basically meant with "code duplication". It is an additional call, thus, if the boolean value of a cochain is tested very often then one will see a slowdown. But I don't know if it actually is tested very often.
Anyway, with this code cuplication, things seem to work. Hooray!
At the moment, it seems that most of the remaining work will be to change the expected output for those tests that use logging.
comment:67 followup: 68 Changed 3 years ago by
My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does __nonzero__
do anything with Python 3? If so, maybe you could do something like
from six import PY2 if PY2: def __nonzero__(self): ...
to avoid the extra call when using Python 3.
comment:68 followup: 81 Changed 3 years ago by
Replying to jhpalmieri:
My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does
__nonzero__
do anything with Python 3? If so, maybe you could do something likefrom six import PY2 if PY2: def __nonzero__(self): ...to avoid the extra call when using Python 3.
I don't think that __nonzero__
does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current notyetpublished code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for __nonzero__
and for __bool__
, but only one of them would be compiled.
comment:69 Changed 3 years ago by
One detail: According to unit_test_64, there is no significant speed difference between the py2 and py3 versions. In any case, the cputime obtained as sum of the cputime() and Singular's timer is only half of the walltime. Do I recall correctly that the time used to save data on disk is not measured by cputime()? Is there an easy way to determine that time, too?
comment:70 Changed 3 years ago by
I don't think it makes much sense measuring time to save to disk, as it varies wildly between different OS's, different hardware, different configuration of buffering, etc etc.
Often saving is completely asyncronous from the computation due to buffering and separate threads run to do buffer flushing, etc.
comment:71 Changed 3 years ago by
Now I am totally baffled. Is it not the case that comparison is supposed to be implemented by __richcmp__
? Apparently __richcmp__
isn't called anymore.
The following corresponds to the code in pGroupCohomology.dickson
, that did work in the past:
sage: cython(""" ....: from sage.structure.richcmp import richcmp, op_LT, op_NE, op_GT ....: from sage.all import GF ....: class DICKSON: ....: def __init__(self,p): ....: self.K = GF(p) ....: def __richcmp__(self, other, op): ....: print("Comparison called") ....: return richcmp(self.K, other.K, op) ....: """)
Now, rich comparison fails both in py2 and py3:
sage: D = DICKSON(5) sage: F = loads(dumps(D)) sage: D == F False sage: F.K == D.K True
What has changed, how to fix it?
comment:72 followup: 73 Changed 3 years ago by
Aha! I see in sage.misc.unknown that a decorator @richcmp_method
is used. So, I guess I need to use it, too.
comment:73 Changed 3 years ago by
Replying to SimonKing:
Aha! I see in sage.misc.unknown that a decorator
@richcmp_method
is used. So, I guess I need to use it, too.
Hooray:
sage: cython(""" ....: from sage.structure.richcmp import richcmp, richcmp_method ....: from sage.all import GF ....: @richcmp_method ....: class DICKSON(object): ....: def __init__(self,p): ....: self.K = GF(p) ....: def __richcmp__(self, other, op): ....: print("Comparison called") ....: return richcmp(self.K, other.K, op) ....: """) sage: D = DICKSON(5) sage: F = loads(dumps(D)) sage: D == F Comparison called True
comment:74 followups: 75 76 Changed 3 years ago by
At the moment, tests in dickson
, auxiliaries
, resolution
and factory
pass both in Py2 and Py3. In isomorphism_test
, there are currently two failing tests, and it seems to me that they reveal a Py3bug in cartesian_product_iterator
:
File "/home/king/Projekte/coho/cohodevel/src/pGroupCohomology/isomorphism_test.py", line 353, in pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms Failed example: T.explore_isomorphisms() Exception raised: Traceback (most recent call last): File "sage/misc/cachefunc.pyx", line 1944, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10139) return cache[k] KeyError: (((3, 5, 4, 6, 7, 1, 2), 0, True), ()) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/mrange.py", line 139, in _xmrange_iter curr_elt = [next(i) for i in curr_iters[:1]] + [None] File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/misc/mrange.py", line 139, in <listcomp> curr_elt = [next(i) for i in curr_iters[:1]] + [None] StopIteration The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms[10]>", line 1, in <module> T.explore_isomorphisms() File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/isomorphism_test.py", line 402, in explore_isomorphisms Cands = self.candidates_of_gens(tuple([p[4] for p in Sizes]),len(self.rigid_generators)) File "sage/misc/cachefunc.pyx", line 1949, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10273) w = self._instance_call(*args, **kwds) File "sage/misc/cachefunc.pyx", line 1825, in sage.misc.cachefunc.CachedMethodCaller._instance_call (build/cythonized/sage/misc/cachefunc.c:9758) return self.f(self._instance, *args, **kwds) File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/pGroupCohomology/isomorphism_test.py", line 644, in candidates_of_gens for Ims1,Ims2 in cartesian_product_iterator([FirstHalf, SecondHalf]): RuntimeError: generator raised StopIteration
I will investigate which values of FirstHalf
and SecondHalf
trigger it.
comment:75 Changed 3 years ago by
comment:76 Changed 3 years ago by
Replying to SimonKing:
At the moment, tests in
dickson
,auxiliaries
,resolution
andfactory
pass both in Py2 and Py3.
When working around the cartesian_product_iterator bug, the tests in isomorphism_test
pass both in Py2 and Py3. Yay!
comment:77 Changed 3 years ago by
Two errors in Py3 concern __div__
, namely
Running doctests with ID 2019092107265720b48344. Using optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage Doctesting 1 file. sage t long warnlong 47.4 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pyx ********************************************************************** File "/home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pyx", line 3131, in pGroupCohomology.cochain.MODCOCH.__div__ Failed example: print(H.1/3) #indirect doctest Expected: (a_6_0)/(3): 6Cocycle in H^*(SmallGroup(400,206); GF(5)) defined by 2*c_2_1*c_2_2*a_1_0*a_1_1 Got: (a_6_0)*(2): 6Cocycle in H^*(SmallGroup(400,206); GF(5)) defined by 2*c_2_1*c_2_2*a_1_0*a_1_1 ********************************************************************** File "/home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pyx", line 3139, in pGroupCohomology.cochain.MODCOCH.__div__ Failed example: H.2/H.1 Expected: Traceback (most recent call last): ... TypeError: Can not divide by <class 'pGroupCohomology.cochain.MODCOCH'> Got: <BLANKLINE> Traceback (most recent call last): File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/home/king/Sage/git/py3/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest pGroupCohomology.cochain.MODCOCH.__div__[6]>", line 1, in <module> H.gen(2)/H.gen(1) File "sage/structure/element.pyx", line 1718, in sage.structure.element.Element.__truediv__ (build/cythonized/sage/structure/element.c:12780) return (<Element>left)._div_(right) File "sage/structure/element.pyx", line 2654, in sage.structure.element.RingElement._div_ (build/cythonized/sage/structure/element.c:18184) frac = self._parent.fraction_field() TypeError: 'NoneType' object is not callable ********************************************************************** 1 item had failures: 2 of 8 in pGroupCohomology.cochain.MODCOCH.__div__ [1490 tests, 2 failures, 465.54 s]  sage t long warnlong 47.4 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pyx # 2 doctests failed  Total time for all tests: 470.6 seconds cpu time: 249.3 seconds cumulative wall time: 465.5 seconds
The first error seems to indicate that H.1/3
is automatically translated to H.1*2
, which does make sense modulo 5, but which I did not implement like this. And the second error is nothing that I implemented.
In fact I overrode __div__
for COCH
and MODCOCH
. Is it perhaps the case that overriding __div__
won't work in Py3?
comment:79 followup: 80 Changed 3 years ago by
Replying to chapoton:
In py3, there is
__truediv__
and__floordiv__
I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement __truediv__
, __floordiv__
and __div__
?
On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.
So, given a RingElement
, what methods do I need to implement for providing division by an element of the base field, and what do I need to implement for making clear that division of two ring elements doesn't work?
I guess I should implement a fraction_field
method for cohomology rings. Of course, since cohomology rings can have zero divisors, a fraction field will not always exist. So, should in that situation fraction_field
return NotImplemented
, or raise a ValueError
, or what else?
Do I need to provide a singleunderscore _div_
method, in addition to the fraction_field
?
How to implement division by a base field element (I guess _div_
is for division of two ring elements)?
comment:80 Changed 3 years ago by
Replying to SimonKing:
Replying to chapoton:
In py3, there is
__truediv__
and__floordiv__
I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement
__truediv__
,__floordiv__
and__div__
?On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.
I think I'll proceed as follows:
 Implement a
fraction_field
method for cohomology rings raising a TypeError (note thatZZ.quo(6).fraction_field()
raises a TypeError, too)  Do NOT implement
_div_
 the attempt to divide two elements of a cohomology ring will result in the TypeError raised by thefraction_field()
method  Division of a cohomology element by an element of the base field will automatically work, by multiplying the inverse of the field element with the cohomology element, using
_lmul_/_rmul_
. I should add an example/test in the documentation of the_lmul_/_rmul_
methods
comment:81 Changed 3 years ago by
Replying to SimonKing:
I don't think that
__nonzero__
does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current notyetpublished code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for__nonzero__
and for__bool__
, but only one of them would be compiled.
This did work, to some extent. But not really reliably. Therefore, to be on the safe side, I will implement both methods in both python versions, since the code duplication is only small.
comment:82 Changed 3 years ago by
I am currently dealing with different but mathematically equivalent behaviour: Apparently, previously Singular did not always do tail reduction, i.e., some rings in the database do not have a fully reduced representation of the relation ideal. For example, here are relations for the cohomology of SmallGroup(64,14)
:
sage: H.rels() # not tail reduced, result obtained with py2 ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1^2', 'a_1_0*a_3_3+a_2_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
versus
sage: H.rels() # tail reduced, obtained with Py3 ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_0*a_3_3+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1*a_1_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
As you can see, the relation a_2_1^2+a_2_1*a_1_1^2
could be used to reduce the tail of a_1_1*a_3_3+a_2_1^2
. Also, the sorting of the relations is different.
The question is how to deal with it. I guess I could proceed as follows:
sage: if (2, 8) < sys.version_info: ....: expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', ....: 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_0*a_3_3+a_2_1*a_1_1^2', ....: 'a_1_1*a_3_3+a_2_1*a_1_1^2', 'a_2_1*a_3_3', 'a_3_3^2'] ....: else: ....: expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', ....: 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1^2', ....: 'a_1_0*a_3_3+a_2_1^2', 'a_2_1*a_3_3', 'a_3_3^2'] ....: sage: H.rels() == expected_rels True
comment:83 Changed 3 years ago by
Sigh. I am suffering from "interesting" debugging sessions. At the moment, it seems to me that some errors in the py2version only occur, when I first install p_group_cohomology
in the py2install of Sage, then install p_group_cohomology
in the py3install of Sage, and only then do tests in the py2install.
comment:84 followup: 85 Changed 3 years ago by
I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/
.
comment:85 followup: 86 Changed 3 years ago by
Replying to dimpase:
I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of
~/.sage/
.
You mean by testing the p_group_cohomology package, or something else?
Indeed I had the same impression concerning ~/.sage/
. Fortunately most of the trouble could be solved by putting str()
around some data (because I got unicode where a string was expected  so, perhaps a rogue py3pickle got misinterpreted in py2).
comment:86 Changed 3 years ago by
Replying to SimonKing:
Replying to dimpase:
I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of
~/.sage/
.You mean by testing the p_group_cohomology package, or something else?
Indeed I had the same impression concerning
~/.sage/
. Fortunately most of the trouble could be solved by puttingstr()
around some data (because I got unicode where a string was expected  so, perhaps a rogue py3pickle got misinterpreted in py2).
I was getting different output from a test, IIRC it was releated to unicode, and the problem went away after I removed ~/.sage/
. I thought it might have been related to ipython, but who knowns.
comment:87 Changed 3 years ago by
I tried to solve it by doing an explicit conversion to str whenever there is data that is expected of this type. And instead of if isinstance(bla, str)
, I now do if isinstance(bla, (str, unicode))
, where I define unicode=str
in py3.
So far, this seems to work. And after all, most users will have only one Sage installation anyway.
comment:88 Changed 3 years ago by
Replying to SimonKing:
Replying to SimonKing:
If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.
Hooray:
# in Sagewithpython2 sage: P = CombinatorialFreeModule(ZZ,'x') sage: save(P,'../../pickle')# BOTH with python2 and python3 sage: B = load('../../pickle.sobj') sage: B._default_filename '/home/king/Sage/pickle.sobj'So, the old trick should still work.
I wonder if I shouldn't improve the trick.
Currently, if the initialisation of an unpickled object can not be finished, it is the __getattr__
method that would trigger the completion of initialisation based on the value of self._default_filename
that is set by Sage's loads()
after unpickling the object and before returning it.
But what if I turn _default_filename
into a property? I could make it so that its setter triggers the completion of initialisation, to the effect that the object returned by loads()
will be valid. The trickery would be in the property setter, not in __getattr__
.
comment:89 Changed 3 years ago by
I am puzzled about the appearance of unicode. For example, consider these two lines in my code:
root = str(kwds.get('root', COHO.workspace)) coho_logger.debug("Group data are rooted at %r", None, root)
The logging appears in the following session (in Sagewithpy2, where I first installed the package in Sagewithpy3 and then in Sagewithpy2):
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: from pGroupCohomology.cohomology import COHO sage: _ = COHO() <module>: Group data are rooted at u'/home/king/.sage/temp/klap/6618/dir_T5Im2o/'
So, apparently the variable root
in the code snipped above is a unicode
, although I explicitly converted it to a str
. Moreover:
sage: COHO.workspace '/home/king/.sage/temp/klap/6618/dir_T5Im2o/' sage: type(COHO.workspace) <type 'str'>
In other words: There is an attribute COHO.workspace
of type str
, and when I convert it to a str
then a unicode
results. WTF???
comment:90 followup: 91 Changed 3 years ago by
I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.
It might be that iPython installs some kind of display hook that sets str
to convert to unicode. There is code in src/sage/repl/rich_output
that might be doing this.
That is to say that unicode you see printed might be happening due to on the fly conversion, set up by DisplayManager
.
comment:91 Changed 3 years ago by
Replying to dimpase:
I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.
Is there unicode in py3?
The common topic is: In some cases I get a TypeError saying that a str was expected, but got unicode. And a log shows that a variable is of type unicode after an EXPLICIT(!) cast to str.
In all mentioned cases, I proceeded as follows:
 Install the package in Sagewithpy2
 Install the package in Sagewithpy3
 Test in Sagewithpy2
Sagewithpy2 and Sagewithpy3 are of course in different folders. What they share: The git repository (but different git worktrees), and DOT_SAGE.
But it is all very flaky. I just inserted a couple of print statements in my code to investigate more deeply, did the above doubleinstall procedure  and suddenly the type was str, not unicode.
It might be that iPython installs some kind of display hook that sets
str
to convert to unicode. There is code insrc/sage/repl/rich_output
that might be doing this.
It is not only the output but the type inside of the code. In some cases, I get a TypeError.
comment:92 Changed 3 years ago by
Do you see these errors while running doctests? I gather there is nodotsage
option you can use while running tests, so you may try to see whether you can reproduce this without DOT_SAGE.
Another thing to try is to wipe out DOT_SAGE each time you switch from pyX to py(5X), and see whether this make the error to go away.
comment:93 Changed 3 years ago by
another way to work around this might be just to convert all your strings to unicode.
comment:94 Changed 3 years ago by
I am afraid that it is currently not possible for anybody else to reproduce the following, because I didn't publish my code yet. But anyway, here is what I did:
 Insert type information, so that the mentioned code snippet now is
root = str(kwds.get('root', COHO.workspace)) coho_logger.debug("Group data are rooted at %r. Type: %r", None, root, type(root))
 Install it in py3
 Install it in py2
 In Py3, run the following:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: H = CohomologyRing(64,14, from_scratch=True) sage: H.make()
 In Py2, run the following:
sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() <module>: Group data are rooted at u'/home/king/.sage/temp/klap/14917/dir_20mizE/'. Type: <type 'unicode'>
So, the type REALLY is unicode, even though I did an EXPLICIT conversion to str.
Then, I continued:
 Move the old
~/.sage
and create a new one.  Start Sagewithpy2, and repeat the test:
Setting permissions of DOT_SAGE directory so only you can read and write it. sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() <module>: Group data are rooted at u'/home/king/.sage/temp/klap/15112/dir_fYCkwo/'. Type: <type 'unicode'> sage: type(COHO.workspace) <type 'str'>
So, removing DOT_SAGE didn't help. For comparison, do the same in Py3:
sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() <module>: Group data are rooted at '/home/king/.sage/temp/klap/15493/dir_81mwix3u/'. Type: <class 'str'>
which of course is no surprise, as python3 has no unicode
.
Note that the line _ = COHO()
does not involve taking an old pickle from a repository. It merely creates a new instance of COHO
and initialises it only to the point where the current workspace is assigned to the attribute .root
.
Summary: The class attribute COHO.workspace
is of type str. The dictionary kwds
is empty. str(kwds.get('root', COHO.workspace)))
returns the unicode version of `COHO.workspace. I.e., converting a str to a str results in a unicode.
comment:95 Changed 3 years ago by
Just to document that we are talking about a Heisenbug: I wanted to test whether it is perhaps the case that str==unicode
. Therefore, I changed the code snippet to
root = str(kwds.get('root', COHO.workspace)) coho_logger.debug("Group data are rooted at %r. Type: %r, %r, %r", None, root, type(root), type(COHO.workspace), str)
, installed it in py2 (only), and rerun the test in py2:
sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() <module>: Group data are rooted at '/home/king/.sage/temp/klap/16014/dir_M5saF0/'. Type: <type 'str'>, <type 'str'>, <type 'str'>
Hence, no bug. "Hooray".
Start a sagewithpy3session and do
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: H = CohomologyRing(64,14, from_scratch=True) sage: H.make()
The hope was of course that it triggers something. But the test in py2 does not have a unicode problem. "Hooray".
But I am sure that the problem will pop up sooner or later.
comment:96 Changed 3 years ago by
To make sure that this bug doesn't pop up again, I inserted a test whether the python version is less than (2,8) and root
is of type unicode. In this case, I print some information and then raise a runtime error. I guess it's the best that I can do for now.
After recompiling and rerunning the above tests, they went fine again. But sooner or later it will return, and then the diagnostic tests I inserted will perhaps tell more clearly what is happening.
comment:97 Changed 3 years ago by
Or another idea: Since it is Cython code, I could do cdef str root = ...
.
comment:98 Changed 3 years ago by
Hooray!
The inserted test already showed something:
sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() Critical bug in python. kwds.get('root', COHO.workspace) is <type 'str'> COHO.workspace is <type 'str'> str=<type 'unicode'>, unicode=<type 'unicode'>  RuntimeError Traceback (most recent call last) <ipythoninput5a303ce041f27> in <module>() > 1 _ = COHO() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() RuntimeError: An explicit cast to <str> resulted in a unicode object
Aha! For some silly reason, we have str == unicode
!!
EDIT: The above was printed like this:
if (2, 8) > sys.version_info and isinstance(root, unicode): print("Critical bug in python.") print("kwds.get('root', COHO.workspace) is %r"%type(kwds.get('root', COHO.workspace))) print("COHO.workspace is %r"%type(COHO.workspace)) print("str=%r, unicode=%r"%(str,unicode)) raise RuntimeError("An explicit cast to <str> resulted in a unicode object")
comment:99 Changed 3 years ago by
How to cope with it? Shall I test upon importing of the module whether str == unicode
in Python2, and raise an error whose message tells to recompile?
comment:100 followup: 101 Changed 3 years ago by
So, would this still pop up if right before running these tests you removed ~/.sage/
?
comment:101 Changed 3 years ago by
Replying to dimpase:
So, would this still pop up if right before running these tests you removed
~/.sage/
?
See comment:94. There, it didn't help. Now, it doesn't either:
king@klap:~/Sage/git$ mv ~/.sage/ ~/.sageBAK king@klap:~/Sage/git$ mkdir ~/.sage king@klap:~/Sage/git$ sage/sage ┌────────────────────────────────────────────────────────────────────┐ │ SageMath version 8.9.beta8, Release Date: 20190825 │ │ Using Python 2.7.15. Type "help()" for help. │ └────────────────────────────────────────────────────────────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Warning: this is a prerelease version, and it may be unstable. ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ Setting permissions of DOT_SAGE directory so only you can read and write it. sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() Critical bug in python. kwds.get('root', COHO.workspace) is <type 'str'> COHO.workspace is <type 'str'> str=<type 'unicode'>, unicode=<type 'unicode'>  RuntimeError Traceback (most recent call last) <ipythoninput5a303ce041f27> in <module>() > 1 _ = COHO() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() RuntimeError: An explicit cast to <str> resulted in a unicode object
comment:102 Changed 3 years ago by
I wonder if the problem occurs because I am not installing the development versions of my package from an actual "package" (using sage i p_group_cohomology
), but directly from the sources. I.e., in the src
folder of my development tree, I do
sage python setup.py build sage pip install . upgrade
I know that sage python setup.py build
does create .c files in the src folder, but since the changed module is recompiled both in py2 and py3, I thought that it doesn't matter. And for efficiency, I want that only modules are recompiled that have changed.
So, I would like to test if the problem persists when separating the py2 and py3 build folders. When doing sage python setup.py build
, is it somehow possible to prescribe in what folder the .c
and .pyc
files are created? Then, I could choose separate folders for py2 and py3.
comment:103 Changed 3 years ago by
I am now putting cythonize(..., build_dir=os.path.join("build","c_files{}.{}".format(version_info.major, version_info.minor)))
into my setup.py
. In that way, the cfiles will be created in separate directories for both Sage versions.
Let's see if this is enough to solve the problem!
comment:104 Changed 3 years ago by
PS: Even if it solves the problem, I will test upon import whether str is unicode in py2, and if it is, raise a RuntimeError
advising to recompile. I hope this error will never pop up...
comment:105 Changed 3 years ago by
After changing setup.py so that the .c files will be in different folders for different python versions, tests in cohomology.pyx pass both with py2 and py3 and then py2 again. So, I can now turn to modular_cohomology.pyx and finally init.py (which contains a long doc string, namely the "general overview" of the package).
comment:106 Changed 3 years ago by
Hooray! When I ran sage tp 2 forcelib long warnlong 60.1
, all tests passed both in python 2 and python 3. I don't know how significant the run time differences are. In any case, here are the details:
Python3 ======== Running doctests with ID 2019092500503078efa273. Using optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage Sorting sources by runtime so that slower doctests are run first.... Doctesting 13 files using 2 threads. sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pxd [0 tests, 0.00 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pyx [1481 tests, 350.33 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cohomology.pyx [1445 tests, 363.25 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/isomorphism_test.py ********************************************************************** File "/home/king/Projekte/coho/cohodevel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest Warning, slow doctest: T.explore_isomorphisms() # long time Test ran for 122.79 s [213 tests, 221.05 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/modular_cohomology.pyx [459 tests, 274.13 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/factory.py [229 tests, 126.99 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/__init__.py [343 tests, 96.76 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/dickson.pyx [28 tests, 1.53 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/auxiliaries.py [39 tests, 0.79 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/barcode.py [143 tests, 10.31 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/resolution.pxd [0 tests, 0.00 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/resolution_bindings.pxd [0 tests, 0.00 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/resolution.pyx [882 tests, 22.95 s]  All tests passed!  Total time for all tests: 765.7 seconds cpu time: 939.6 seconds cumulative wall time: 1468.1 seconds Python2 ======== Running doctests with ID 20190925010447fa1de7f4. Git branch: matrix_implementation_keyword Using optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,p_group_cohomology,python2,sage Sorting sources by runtime so that slower doctests are run first.... Doctesting 13 files using 2 threads. sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pyx [1481 tests, 437.44 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cochain.pxd [0 tests, 0.00 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/cohomology.pyx [1445 tests, 465.15 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/isomorphism_test.py ********************************************************************** File "/home/king/Projekte/coho/cohodevel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest Warning, slow doctest: T.explore_isomorphisms() # long time Test ran for 116.79 s [213 tests, 229.85 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/modular_cohomology.pyx [459 tests, 312.40 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/factory.py [229 tests, 124.88 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/__init__.py [343 tests, 96.86 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/resolution.pxd [0 tests, 0.00 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/resolution.pyx [882 tests, 21.38 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/dickson.pyx [28 tests, 1.63 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/auxiliaries.py [39 tests, 0.78 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/resolution_bindings.pxd [0 tests, 0.00 s] sage t long warnlong 60.1 /home/king/Projekte/coho/cohodevel/src/pGroupCohomology/barcode.py [143 tests, 14.90 s]  All tests passed!  Total time for all tests: 873.5 seconds cpu time: 996.8 seconds cumulative wall time: 1705.3 seconds
Hence, tomorrow (i.e., in central European time, today in a few hours :)
) I will create a new package version and post it here for review.
comment:107 Changed 3 years ago by
Something more to do: I think I should add a test showing what happens when moving the directory associated with a cohomology ring. The package can cope with moved data, based on some trickery in the attribute _default_filename
that is defined by Sage's unpickler. I've turned _default_filename
into a property (with getter, setter and deleter), and its documentation should contain the new test.
comment:108 Changed 3 years ago by
Description:  modified (diff) 

I'm afraid that another test failure popped up. So, I cannot post the new version yet.
comment:109 Changed 3 years ago by
After fixing the failed test (in which I forgot to save something before loading it), I added a new feature to unit_test_64
: Previously it was only tested whether the generator degrees and the Poincaré series of a freshly computed cohomology ring is consistent with what is stored in the database; now I test whether the new ring and the stored ring are isomorphic.
Indeed they are, for all groups of order 64. More precisely: In many cases the generators of the relation ideal are not identical. But in all cases, mapping the ith generator of the new ring to the ith generator of the old ring results in an isomorphism.
There is one remaining py3 issue, though: When I try to build the documentation, I get the error "No module named sphinx". The documentation does build, when I use py2. What can I do to fix it in py3?
EDIT: Is it not the case that sphinx is installed in Sage for building its documentation, both in py2 and py3? I am not sure if I ever tried to build the documentation of Sagewithpy3, though. Will sphinx be installed when I do a plain make
in Sage (i.e., when I build the docs)?
comment:110 followup: 111 Changed 3 years ago by
Sphinx should be installed, yes. Look for SAGE_ROOT/local/lib/python3.7/sitepackages/sphinx/
. If you run Sage, can you import sphinx
?
Edit: this is for a Python 3 build of Sage.
comment:111 Changed 3 years ago by
Replying to jhpalmieri:
Sphinx should be installed, yes. Look for
SAGE_ROOT/local/lib/python3.7/sitepackages/sphinx/
. If you run Sage, can youimport sphinx
?
It was a trivial mistake: In the doc's Makefile, I had SPHINXBUILD = python msphinx
, but since python means python2 even in a py3install of Sage, I have to use SPHINXBUILD = sage python msphinx
comment:112 Changed 3 years ago by
Branch:  → u/SimonKing/p_group_cohomology_in_py_3 

comment:113 Changed 3 years ago by
Authors:  → Simon King 

Commit:  → 463806c6cebb3120e20df0f19a216dd529553741 
Description:  modified (diff) 
Status:  new → needs_review 
I checked that the package builds and passes tests in Sagewithpy3. I checked that when running the tests directly on the sources (not during package installation) the tests pass in Sagewithpy2. I recommend that whoever reviews this tries the package both in py2 and in py3.
New commits:
2e0d11e  Pass unpickling options to pickle.load, default encoding 'latin1'. Accept both str and bytes in mtx_unpickle

8c105cd  Make str_to_bytes/bytes_to_str accept both str and bytes input.

c8a0748  Add tests for #28444

89ac77a  Fix keyword for py3only test

3693024  Fix doc strings in sage.misc.persist

f0828ee  Add a comment regarding the expected data type for an unpickle helper

ba41ebe  Fix two typos in a comment

463806c  Upgrade p_group_cohomology to version 3.3

comment:114 followup: 115 Changed 3 years ago by
Hooray! In Py2, sage f c p_group_cohomology
results in
... [p_group_cohomology3.3] Running the test suite for p_group_cohomology3.3... [p_group_cohomology3.3] make[2]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1' [p_group_cohomology3.3] Making check in src [p_group_cohomology3.3] make[3]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] make mnttest.sh [p_group_cohomology3.3] make[4]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] test r ./test.reg  cp ./test.reg . [p_group_cohomology3.3] make[4]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] make checkTESTS [p_group_cohomology3.3] make[4]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] make[5]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] PASS: mnttest.sh [p_group_cohomology3.3] ============================================================================ [p_group_cohomology3.3] Testsuite summary for modular_resolution 1.1 [p_group_cohomology3.3] ============================================================================ [p_group_cohomology3.3] # TOTAL: 1 [p_group_cohomology3.3] # PASS: 1 [p_group_cohomology3.3] # SKIP: 0 [p_group_cohomology3.3] # XFAIL: 0 [p_group_cohomology3.3] # FAIL: 0 [p_group_cohomology3.3] # XPASS: 0 [p_group_cohomology3.3] # ERROR: 0 [p_group_cohomology3.3] ============================================================================ [p_group_cohomology3.3] make[5]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] make[4]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] make[3]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1/src' [p_group_cohomology3.3] make[3]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1' [p_group_cohomology3.3] make[3]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1' [p_group_cohomology3.3] make[2]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3/src/modular_resolution1.1' [p_group_cohomology3.3] Running doctests with ID 20190926111911b01ece2b. [p_group_cohomology3.3] Git branch: t/28414/p_group_cohomology_in_py_3 [p_group_cohomology3.3] Using optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,python2,sage [p_group_cohomology3.3] Sorting sources by runtime so that slower doctests are run first.... [p_group_cohomology3.3] Doctesting 13 files using 3 threads. [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/modular_cohomology.pyx [p_group_cohomology3.3] [459 tests, 456.02 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/cochain.pxd [p_group_cohomology3.3] [0 tests, 0.00 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/cochain.pyx [p_group_cohomology3.3] [1481 tests, 599.35 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/cohomology.pyx [p_group_cohomology3.3] [1465 tests, 606.86 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/isomorphism_test.py [p_group_cohomology3.3] ********************************************************************** [p_group_cohomology3.3] File "pGroupCohomology3.3/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest [p_group_cohomology3.3] Warning, slow doctest: [p_group_cohomology3.3] T.explore_isomorphisms() # long time [p_group_cohomology3.3] Test ran for 161.47 s [p_group_cohomology3.3] [213 tests, 307.72 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/__init__.py [p_group_cohomology3.3] [343 tests, 148.33 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/factory.py [p_group_cohomology3.3] ********************************************************************** [p_group_cohomology3.3] File "pGroupCohomology3.3/pGroupCohomology/factory.py", line 133, in pGroupCohomology.factory.unit_test_64 [p_group_cohomology3.3] Warning, slow doctest: [p_group_cohomology3.3] L, t = unit_test_64(from_scratch=False) # long time [p_group_cohomology3.3] Test ran for 99.46 s [p_group_cohomology3.3] [229 tests, 209.64 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/resolution.pxd [p_group_cohomology3.3] [0 tests, 0.00 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/resolution.pyx [p_group_cohomology3.3] [882 tests, 26.85 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/dickson.pyx [p_group_cohomology3.3] [28 tests, 1.41 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/barcode.py [p_group_cohomology3.3] [143 tests, 21.04 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/resolution_bindings.pxd [p_group_cohomology3.3] [0 tests, 0.00 s] [p_group_cohomology3.3] sage t long warnlong 80.0 pGroupCohomology3.3/pGroupCohomology/auxiliaries.py [p_group_cohomology3.3] [39 tests, 0.92 s] [p_group_cohomology3.3]  [p_group_cohomology3.3] All tests passed! [p_group_cohomology3.3]  [p_group_cohomology3.3] Total time for all tests: 815.2 seconds [p_group_cohomology3.3] cpu time: 1261.3 seconds [p_group_cohomology3.3] cumulative wall time: 2378.1 seconds [p_group_cohomology3.3] [p_group_cohomology3.3] real 13m52.051s [p_group_cohomology3.3] user 22m48.984s [p_group_cohomology3.3] sys 4m1.098s [p_group_cohomology3.3] Successfully installed p_group_cohomology3.3 [p_group_cohomology3.3] Deleting temporary build directory [p_group_cohomology3.3] /home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology3.3 [p_group_cohomology3.3] Finished installing p_group_cohomology3.3.spkg make[1]: Leaving directory '/home/king/Sage/git/sage/build/make' real 27m19.468s user 30m7.398s sys 4m10.930s Sage build/upgrade complete!
In Py3, I had more warnings about slow tests (six instead of two, if I recall correctly), but perhaps my laptop (or its harddisk) was busy with other things.
Now I am running unit_test_64
in Python2. Note that in Python3 I got about 20 (or even more) examples where the freshly computed relation ideal was presented by a different set of generators than what was stored in the database. But, as I have mentioned in comment:82, this seems to be Singular not doing all tail reduction in py2.
comment:115 Changed 3 years ago by
Replying to SimonKing:
Now I am running
unit_test_64
in Python2. Note that in Python3 I got about 20 (or even more) examples where the freshly computed relation ideal was presented by a different set of generators than what was stored in the database.
Just for the record: In the py2 version, the freshly computed results were identical (not only equal up to generator choice) to the stored results.
comment:116 Changed 3 years ago by
Description:  modified (diff) 

By the way, at the recent Sage Days in Bonn, I learnt how to host the package repository at github and use Travis CI to some extent. I have posted the link to the ticket description.
Note, however, that testing it on Travis CI with python3 will be difficult: I was told that the current Sagewithpy3 docker image is rather outdated, and also I don't know how I should write my .travis.yml
so that the automatic tests involve both py2 and py3. The answer on sagedevel didn't help me much. Is someone reading this who can tell me how I should modify .travis.yml
?
comment:117 followup: 118 Changed 3 years ago by
Suppose I delete .sage/pGroupCohomology/
, to force the package to reinstall those files. Then
 if I build with Python 2 first and then Python 3, all tests pass.
 if I build with Python 3 first and then Python 2, I get failures:
[p_group_cohomology3.3] IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/4gp2/H4gp2.sobj are not readable: expected bytes, unicode found
I guess I have a few questions:
 is it possible (and not too much work to implement) for the Python 3 version to build
.sobj
files which are readable by Python 2?  if not, should the package discard the existing
.sobj
file and rebuild in this case?  more generally, if there is any error reading the
.sobj
file — maybe somehow the file got corrupted — should it just rebuild?
This particular problem should not happen much: people won't be switching from Python 3 to Python 2 very much. So don't work too hard to get the Python 3 version to produce .sobj
files which are readable by Python 2.
comment:118 followup: 119 Changed 3 years ago by
Replying to jhpalmieri:
Suppose I delete
.sage/pGroupCohomology/
, to force the package to reinstall those files. Then
 if I build with Python 2 first and then Python 3, all tests pass.
 if I build with Python 3 first and then Python 2, I get failures:
[p_group_cohomology3.3] IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/4gp2/H4gp2.sobj are not readable: expected bytes, unicode found
It was a good idea to do such tests. And I am surprised about the error message: There is no use of bytes in the whole cohomology package. Perhaps the error is about unpickling a MeatAxe matrix (sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense
)? But this was supposed to be fixed in #28444.
I guess I have a few questions:
 is it possible (and not too much work to implement) for the Python 3 version to build
.sobj
files which are readable by Python 2?
I thought they were.
 if not, should the package discard the existing
.sobj
file and rebuild in this case? more generally, if there is any error reading the
.sobj
file — maybe somehow the file got corrupted — should it just rebuild?
It is already the case that if some data source provides data in legacy format, it will automatically be overwritten by a new pickle.
This particular problem should not happen much: people won't be switching from Python 3 to Python 2 very much. So don't work too hard to get the Python 3 version to produce
.sobj
files which are readable by Python 2.
But it should still work and needs to be investigated.
I'll follow your instructions (remove .sage/pGroupCOhomology
, install (and also test?) with Python 3 and then install with Python 2 and see what is happening.
comment:119 Changed 3 years ago by
Replying to SimonKing:
I'll follow your instructions (remove
.sage/pGroupCOhomology
, install (and also test?) with Python 3 and then install with Python 2 and see what is happening.
I installed and tested when I did this, but just now I hit the same error with:
 delete `.sage/pGroupCohomology'
 in Python 3, install the package
 then run Sage:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing(16, 2) H^*(SmallGroup(16,2); GF(2))
 in Python 2, install the package
 then run Sage:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing(16, 2)  IOError Traceback (most recent call last) <ipythoninput2e1924d996e12> in <module>() > 1 CohomologyRing(Integer(16), Integer(2)) /Users/jpalmier/Desktop/Sage/sage_builds/py2/sage8.9.rc1/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds) 1778 if q < 128: 1779 extras['websource'] = False > 1780 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1781 return OUT 1782 /Users/jpalmier/Desktop/Sage/sage_builds/py2/sage8.9.rc1/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds) 1209 if '@use_this_root@' in coho_options: 1210 del coho_options['@use_this_root@'] > 1211 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_workspace,file_name), msg)) 1212 ## 3. Link with local sources and load from there 1213 elif root_local_sources and os.access(os.path.join(root_local_sources,file_name), os.R_OK) and not from_scratch: IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj are not readable: NEWOBJ class argument isn't a type object
comment:120 followup: 123 Changed 3 years ago by
Probably the following would work, too (so that it is not always needed to reinstall):
 Have the package both in py2 and py3
 Create a permanent workspace and store a cohomology ring created with py3
 Try to load the cohomology ring from that permanent workspace with py2
Anyway, after doing the py3 test suit, I can confirm what you reported: The stuff in .sage/pGroupCohomology/db/16gp2
cannot be unpickled with py2. In particular, the state descriptor cannot be loaded:
sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')  TypeError Traceback (most recent call last) <ipythoninput32796f9b01dc5> in <module>() > 1 S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj') /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2845)() 157 158 ## Load file by absolute filename > 159 with open(filename, 'rb') as fobj: 160 X = loads(fobj.read(), compress=compress, **kwargs) 161 try: /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2795)() 158 ## Load file by absolute filename 159 with open(filename, 'rb') as fobj: > 160 X = loads(fobj.read(), compress=compress, **kwargs) 161 try: 162 X._default_filename = os.path.abspath(filename) /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:6929)() 1043 1044 unpickler = SageUnpickler(io.BytesIO(s), **kwargs) > 1045 return unpickler.load() 1046 1047 /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)() 655 656 def load(self): > 657 return self._unpickler.load() 658 659 def noload(self): /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/structure/factory.pyx in sage.structure.factory.lookup_global (build/cythonized/sage/structure/factory.c:4615)() 748 Integer Ring 749 """ > 750 name = bytes_to_str(name, encoding='ASCII') 751 try: 752 return factory_unpickles[name] /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/structure/factory.c:5038)() 55 return b 56 if type(b) is not bytes: > 57 raise TypeError(f"expected bytes, {type(b).__name__} found") 58 59 if PY_MAJOR_VERSION <= 2: TypeError: expected bytes, unicode found
And I guess this can be fixed in bytes_to_str
: After the changes in #28444, bytes_to_str
is more tolerant about the input type (it may be str), but it should be even more so. It should (in py2) accept unicode.
comment:121 followup: 122 Changed 3 years ago by
I think it's good enough if py2created pickles work with py3.
Require the opposite compatibility seems to be an overkill.
comment:122 Changed 3 years ago by
Replying to dimpase:
I think it's good enough if py2created pickles work with py3.
Require the opposite compatibility seems to be an overkill.
I believe the fix (in #28540) would be relatively easy.
If you think that we shouldn't worry so much about reading py3pickles in py2, then of course the group cohomology package can be reviewed as it is now. If I am right about #28640, then #28540 will fix the unpickling of py3cohomology in py2 without any change in the package.
comment:123 followup: 124 Changed 3 years ago by
Replying to SimonKing:
And I guess this can be fixed in
bytes_to_str
: After the changes in #28444,bytes_to_str
is more tolerant about the input type (it may be str), but it should be even more so. It should (in py2) accept unicode.
I think that at this point it's becoming pretty clear you should use a routine that is not named bytes_to_str
, but a routine "ensure_str", which in py3 accepts unicode=str and return the result unchanged, and also accepts py2's "bytes=str" objects and assumes it's "latin1" encoded, because that's what it would have been as a "str" in py2.
You are clearly NOT converting a "bytes" object to a "str" here; you are ensuring something is a "str", doing some conversions if necessary. You should use a routine that reflects that in its name if you need specific behaviour.
The reason why I care is because you are changing routines in sage.cpython.string, which people will expect to contain widely valid; welldefined routines. Stretching them up with hacks for specific purposes will lead to silent erroneous behaviour further down the line, when these routines are used in other contexts because they sound like the right tool (but aren't).
If it gets a py3 "unicode" object in py2, ideally it could just leave it as unicode, because that should be valid in str contexts in py2 for the most time as well. But we may be far from the ideal in py2. A safe choice would be str(<unicode>)
to ensure you actually get a py2 str. That uses "ascii" encoding, so it may fail. You could allow for "UTF8" but that would not roundtrip correctly with the "latin1" on the other side. Using "latin1" for encoding unicodetostr on py2 may also fail, because not all unicode code points have an encoding in "latin1".
comment:124 Changed 3 years ago by
Replying to nbruin:
The reason why I care is because you are changing routines in sage.cpython.string, which people will expect to contain widely valid; welldefined routines. Stretching them up with hacks for specific purposes will lead to silent erroneous behaviour further down the line, when these routines are used in other contexts because they sound like the right tool (but aren't).
Agreed. I have already changed the topic of #28540.
If it gets a py3 "unicode" object in py2, ideally it could just leave it as unicode, because that should be valid in str contexts in py2 for the most time as well. But we may be far from the ideal in py2. A safe choice would be
str(<unicode>)
to ensure you actually get a py2 str. That uses "ascii" encoding, so it may fail.
The application (and the place where #28540 should be changed) is sage.structure.factory.lookup_global
. This function used to only accept str. By #28444, it also accepts bytes, which makes it possible to unpickle a py2pickle in py3.
I suggest in #28540 that it also accepts a unicode. This makes it possible to unpickle a py3pickle in py2.
Again, I won't make this a dependency for the cohomology package.
comment:125 Changed 3 years ago by
#28540 brought an improvement:
sage: from pGroupCohomology import CohomologyRing sage: H = CohomologyRing(16,2,options='debug') _get_p_group_from_cache_or_db: Data found at /home/king/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj Group data are rooted at '/home/king/.sage/pGroupCohomology/db'. Loading a cohomology ring: The state descriptor of the tobeunpickled ring is expected to be provided at u'/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj'  IOError Traceback (most recent call last) <ipythoninput2a62afc150e39> in <module>() > 1 H = CohomologyRing(Integer(16),Integer(2),options='debug') /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds) 1778 if q < 128: 1779 extras['websource'] = False > 1780 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1781 return OUT 1782 /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds) 1209 if '@use_this_root@' in coho_options: 1210 del coho_options['@use_this_root@'] > 1211 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_workspace,file_name), msg)) 1212 ## 3. Link with local sources and load from there 1213 elif root_local_sources and os.access(os.path.join(root_local_sources,file_name), os.R_OK) and not from_scratch: IOError: Saved data at /home/king/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj are not readable: NEWOBJ class argument isn't a type object
So, the previous error arising in lookup_global has gone.
I am afraid the new error looks more like an error that needs to be fixed in the package, not in the Sage library. We will see...
comment:126 Changed 3 years ago by
Next step of debugging.
In Py2, where we still have py3pickles in DOT_SAGE
:
sage: from pGroupCohomology.cohomology import COHO sage: H = COHO() sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj') sage: H.__setstate__(S)  IOError Traceback (most recent call last) <ipythoninput49c79e2f6e0a6> in <module>() > 1 H.__setstate__(S) pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__setstate__() IOError: Unable to read resolution saved at /home/king/Sage/git/sage/local/share/pGroupCohomology/16gp2/dat/R16gp2.sobj
The same in py3:
sage: from pGroupCohomology.cohomology import COHO sage: H = COHO() sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj') sage: H.__setstate__(S) sage: print(H) Cohomology ring of Small Group number 2 of order 16 with coefficients in GF(2) Computation complete Minimal list of generators: [c_2_1: 2Cocycle in H^*(SmallGroup(16,2); GF(2)), c_2_2: 2Cocycle in H^*(SmallGroup(16,2); GF(2)), c_1_0: 1Cocycle in H^*(SmallGroup(16,2); GF(2)), c_1_1: 1Cocycle in H^*(SmallGroup(16,2); GF(2))] Minimal list of algebraic relations: [c_1_0^2, c_1_1^2]
So, the error is now moved to __setstate__
. I guess I'll fix it there.
comment:127 followup: 128 Changed 3 years ago by
Meanwhile I know at what point the error mentioned in comment:125 arises:
sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class') sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())  UnpicklingError Traceback (most recent call last) <ipythoninput2855d0a850e70> in <module>() > 1 si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ()) /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/explain_pickle.pyc in unpickle_newobj(klass, args) 2438 return pers[int(id)] 2439 > 2440 return SageUnpickler.loads(pickle, persistent_load=pers_load) 2441 2442 /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.SageUnpickler.loads (build/cythonized/sage/misc/persist.c:6467)() 962 """ 963 > 964 return cls(io.BytesIO(data), **kwargs).load() 965 966 /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)() 655 656 def load(self): > 657 return self._unpickler.load() 658 659 def noload(self): UnpicklingError: NEWOBJ class argument isn't a type object
But I do not have the faintest idea yet what this error means. For comparison, in py3 I get
sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class') sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ()) sage: si1 <pGroupCohomology.resolution.RESL_sparse_unpickle_class object at 0x7f2ee987abe0>
I'm guessing that a class in Python3 is a type, but in Python2 it generally isn't. Is that the meaning of the error? But isn't it the case that a class defined in Cython (here: resolution.pyx) automatically is a type?
Anyway, I'll try to turn class RESL_sparse_unpickle_class
into class RESL_sparse_unpickle_class(object)
and see if it helps.
comment:128 Changed 3 years ago by
Replying to SimonKing:
Anyway, I'll try to turn
class RESL_sparse_unpickle_class
intoclass RESL_sparse_unpickle_class(object)
and see if it helps.
Nope.
sage: from pGroupCohomology.resolution import RESL_sparse_unpickle_class sage: RESL_sparse_unpickle_class <class 'pGroupCohomology.resolution.RESL_sparse_unpickle_class'> sage: isinstance(_, type) True
but still
sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class') sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ()) Traceback (most recent call last): ... UnpicklingError: NEWOBJ class argument isn't a type object
comment:129 Changed 3 years ago by
Odd. After trying again, the error is gone. In Py2:
sage: R = load("/home/king/.sage/pGroupCohomology/db/16gp2/dat/R16gp2.sobj")  OSError Traceback (most recent call last) <ipythoninput1c703eea54ef8> in <module>() > 1 R = load("/home/king/.sage/pGroupCohomology/db/16gp2/dat/R16gp2.sobj") /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2845)() 157 158 ## Load file by absolute filename > 159 with open(filename, 'rb') as fobj: 160 X = loads(fobj.read(), compress=compress, **kwargs) 161 try: /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2795)() 158 ## Load file by absolute filename 159 with open(filename, 'rb') as fobj: > 160 X = loads(fobj.read(), compress=compress, **kwargs) 161 try: 162 X._default_filename = os.path.abspath(filename) /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:6929)() 1043 1044 unpickler = SageUnpickler(io.BytesIO(s), **kwargs) > 1045 return unpickler.load() 1046 1047 /home/king/Sage/git/sage/local/lib/python2.7/sitepackages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)() 655 656 def load(self): > 657 return self._unpickler.load() 658 659 def noload(self): pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL_sparse_unpickle_class.__call__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__() OSError: Bad file format in file pgroup.c (line 404)
So, now complain about NEWOBJ any more.
comment:130 Changed 3 years ago by
Spoke too soon. Immediately after this different error (i.e., in the same session), I again got the "NEWOBJ" error.
comment:131 Changed 3 years ago by
sage: from pGroupCohomology.resolution import RESL_sparse_unpickle_class sage: pg_RESL_sparse_unpickle_class is RESL_sparse_unpickle_class False
In a different session, I got the answer "True".
Anyway, the error about "NEWOBJ" comes from python. It seems that python decided to kill data persistency. I tried to work around to the extent that py2pickles of cohomology rings seem to be legible in py3. But I think I will not work further on making py3pickles legible in py2.
Another question: Why is the branch of this ticket shown red?
comment:132 followup: 133 Changed 3 years ago by
The oddity is: Sometimes pg_RESL_sparce_unpickle_class
is RESL_sparse_unpickle_class
and sometimes it isn't. When the error occurs, I get
sage: pg_RESL_sparse_unpickle_class <class pGroupCohomology.auxiliaries.unpickle_old_resl at 0x7fb8d4f549a8>
It is not nice that unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sometimes returns something else. But perhaps it is worth trying to make this class inherit from object
.
comment:133 Changed 3 years ago by
Replying to SimonKing:
But perhaps it is worth trying to make this class inherit from
object
.
Nope, didn't help. As there doesn't seem to be an easy fix, I do not plan to work further on it.
Do I understand correctly that I have to rebase on the current develop branch?
comment:134 Changed 3 years ago by
Commit:  463806c6cebb3120e20df0f19a216dd529553741 → 7ad6a4a35c9af94d1692a17e1b8855f49d60a38e 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
7ad6a4a  Upgrade p_group_cohomology to version 3.3

comment:135 Changed 3 years ago by
Maybe it would be simpler, if any error arises reading a pickle, to just delete and reproduce the pickle file. Compatibility going from Python 3 to Python 2 may be annoying to debug and to fix, and besides, there may be other errors that we haven't foreseen.
comment:136 Changed 3 years ago by
Or as you say, to just not work on the Py3 > Py2 conversion. That's fine with me.
comment:137 Changed 3 years ago by
I am using unpickle_override for RESL_sparse_unpickle_class
, but I am not sure if that's needed at all.
comment:138 Changed 3 years ago by
Bummer. Apparently I have destroyed the git repository for my py3 installation. It says:
king@klap:~/Sage/git/py3$ git lg fatal: your current branch 't/28444/fix_backwards_incompatibility_of_unpickling_in_python_3' does not have any commits yet
and ALL files in the Sage tree are untracked. How has that happened, and how to fix it?
comment:139 followup: 142 Changed 3 years ago by
Hopefully John has a version of this branch just before you did that fateful forced push.
There could be copies left on patchbots, maybe LinuxMint?/18.2/x86_64/4.8.053generic/pc78math patchbot does have it...
comment:140 Changed 3 years ago by
Unfortunately I don't, but there are the commits in comment:113 which are still accessible.
comment:141 Changed 3 years ago by
yes, indeed, these commits are available in the full "git fetch trac"  I have them locally, in case they get gc'd.
comment:142 Changed 3 years ago by
Replying to dimpase:
Hopefully John has a version of this branch just before you did that fateful forced push.
 What's wrong with the forced push? IMHO, that has nothing to to with the problem whatsoever. The forced push is nothing more than a rebasing of the previously committed branch. After pushing it, I could still work.
 The branch is still present in my git repository. Perhaps I didn't make it clear. I have a single git repository with two worktrees, one for py2, one for py3. The py2 worktree is fine (AFAIC), in particular I can checkout any branch. However, it seems that I deleted the branch on which the py3 worktree was living, with
git branch d
. As a result, all files in the py3 worktree are now untracked, as is shown bygit status
, andgit log
says that there has never been a commit. Again, I am talking about only one of my two worktrees.  The deleted branch is the branch from #28444, not from here. That branch is merged in "develop", that's why I did
git branch d
on it.
So, from my perspective, "solving the problem" means "allowing me to do git checkout <branchname>
in the py3worktree"  this currently is refused, because git claims that I should first commit or stash the whole Sage library.
comment:143 Changed 3 years ago by
Would git checkout f <branch_name>
possibly help? Or would I possibly destroy even more?
comment:144 followup: 145 Changed 3 years ago by
You could try git stash
and then git checkout BRANCH
and then git stash COMMAND
for some value of COMMAND
: git stash pop
? git stash apply
? git stash branch NEW
?
comment:145 Changed 3 years ago by
Replying to jhpalmieri:
You could try
git stash
and thengit checkout BRANCH
and thengit stash COMMAND
for some value ofCOMMAND
:git stash pop
?git stash apply
?git stash branch NEW
?
Certainly not. I do in fact not have any uncommitted changes! Instead, git believes that I have uncommitted changes (indeed it believes that the whole sage library is completely new), because git is convinced that I am on a branch with an empty commit history  namely, it has allowed me to delete the branch on which I was checked out.
comment:146 followup: 147 Changed 3 years ago by
You could first also make a copy of the whole directory, in case git stash
etc. screws things up even more. How much can you recover by doing diff r /path/to/py2/worktree /path/to/py3/worktree
, or maybe git diff /path/to/py2/worktree /path/to/py3/worktree
?
comment:147 followup: 148 Changed 3 years ago by
Replying to jhpalmieri:
You could first also make a copy of the whole directory,
Hopefully the other worktree (that is still working!) is the whole directory.
in case
git stash
etc. screws things up even more. How much can you recover by doingdiff r /path/to/py2/worktree /path/to/py3/worktree
, or maybegit diff /path/to/py2/worktree /path/to/py3/worktree
?
That would be a megapatch. It would be the diff of an empty folder and the complete Sage sources.
comment:148 followup: 149 Changed 3 years ago by
Replying to SimonKing:
Replying to jhpalmieri:
You could first also make a copy of the whole directory,
Hopefully the other worktree (that is still working!) is the whole directory.
in case
git stash
etc. screws things up even more. How much can you recover by doingdiff r /path/to/py2/worktree /path/to/py3/worktree
, or maybegit diff /path/to/py2/worktree /path/to/py3/worktree
?That would be a megapatch. It would be the diff of an empty folder and the complete Sage sources.
Why? diff r ...
should just compare the files, and the files exist in both places, regardless of whether they're tracked by git.
By the way, if the branch still exists, can you create a new workspace using that branch?
comment:149 Changed 3 years ago by
Replying to jhpalmieri:
Why?
diff r ...
should just compare the files, and the files exist in both places, regardless of whether they're tracked by git.
I see. Then the diff is empty.
By the way, if the branch still exists, can you create a new workspace using that branch?
That's my question. The branch for this ticket still exists, in fact it is the branch that is currently checked out in my py2 worktree. However, git won't allow me to do any normal (nonforced) operation on my py3 worktree, as the branch that was checked out there was deleted, thus, git assumes that my py3 worktree has no commit history and has a couple of thousands new files.
comment:150 Changed 3 years ago by
So if the two worktrees aren't any different from the content point of view, maybe you should just create an entirely new worktree, in a new directory, using the existing branch, and delete the old one. It means you will have to rebuild Sage in the new worktree. I guess that's what you're trying to avoid?
Alternatively, you could rename the old directory (OLD > TEMP
), create a new worktree where the old one used to be (OLD
). Then in the shell, but without telling git, you could delete that newly created directory and rename TEMP
back to OLD
. Then see what git says. I think git keeps track of its worktree information in the original folder, not in the newly created one. By the way, is there anything in .git/worktrees/
? If so, you might need to remove that before you go through this procedure.
comment:151 followup: 153 Changed 3 years ago by
I'm getting a few doctest failures with a Python 3 build of Sage, version 9.0.beta0. All failures have the form
File "pGroupCohomology3.3/pGroupCohomology/cohomology.pyx", line 4591, in pGroupCohomology.cohomology.COHO.__dir__ Failed example: [a for a in dir(H) if a.startswith('f')] #indirect doctest Expected: ['filter_degree_type', 'filter_regular_gready_parameters', 'filter_regular_parameters', 'find_dickson', 'find_dickson_in_subgroup', 'find_small_last_parameter', 'firstOdd', 'fraction_field', 'from_base_ring'] Got: ['filter_degree_type', 'filter_regular_gready_parameters', 'filter_regular_parameters', 'find_dickson', 'find_dickson_in_subgroup', 'find_small_last_parameter', 'firstOdd', 'fraction_field', 'free_module', 'from_base_ring']
comment:153 Changed 3 years ago by
Replying to jhpalmieri:
'free_module',
So, if I understand correctly, the problem is both in py2 and py3., and it is about an additional entry in the dir. That should be solvable...
comment:154 followup: 155 Changed 3 years ago by
Is that kind of doctest too fragile? How important is it? You could certainly change it to startswith('fi')
or add 'free_module'
to the list, but maybe there are more robust options. 'fraction_field' in dir(H)
? I guess I don't know what it's supposed to be testing.
comment:155 Changed 3 years ago by
Replying to jhpalmieri:
Is that kind of doctest too fragile? How important is it? You could certainly change it to
startswith('fi')
or add'free_module'
to the list, but maybe there are more robust options.'fraction_field' in dir(H)
? I guess I don't know what it's supposed to be testing.
It is supposed to test introspection and autocompletion (which doesn't work in Py3, if I understand correctly).
On the other hand, it is true that all methods in that list that do not start with "fi" are inherited and are thus easily changed. Perhaps it is best to insert an ellipsis.
comment:156 Changed 3 years ago by
Commit:  7ad6a4a35c9af94d1692a17e1b8855f49d60a38e → c247f13fd6e6ae3d3feeef41ebba1a78346744da 

Branch pushed to git repo; I updated commit sha1. New commits:
c247f13  Fix some introspection tests

comment:157 Changed 3 years ago by
comment:158 Changed 3 years ago by
Hooray, with some long tests:
$ ./sage f c p_group_cohomology ... [p_group_cohomology3.3]  [p_group_cohomology3.3] All tests passed! [p_group_cohomology3.3]  ... [sagelib8.9.rc1] real 0m18.161s [sagelib8.9.rc1] user 0m5.983s [sagelib8.9.rc1] sys 0m1.083s make[2]: Leaving directory '/home/king/Sage/git/py3/build/make' make[1]: Leaving directory '/home/king/Sage/git/py3/build/make' real 0m18.438s user 0m6.252s sys 0m1.126s Sage build/upgrade complete!
So, it seems to work!
comment:159 Changed 3 years ago by
I still get one failure:
File "pGroupCohomology3.3/pGroupCohomology/cohomology.pyx", line 4814, in pGroupCohomology.cohomology.COHO.__getattr__ Failed example: tc.completions('H.f',globals()) Expected: ['H.filter_degree_type', 'H.filter_regular_gready_parameters', 'H.filter_regular_parameters', 'H.find_dickson', 'H.find_dickson_in_subgroup', 'H.find_small_last_parameter', 'H.firstOdd', 'H.fraction_field', 'H.from_base_ring'] Got: ['H.filter_degree_type', 'H.filter_regular_gready_parameters', 'H.filter_regular_parameters', 'H.find_dickson', 'H.find_dickson_in_subgroup', 'H.find_small_last_parameter', 'H.firstOdd', 'H.fraction_field', 'H.free_module', 'H.from_base_ring']
comment:160 Changed 3 years ago by
Too bad that those tests work for me, i.e., it is easy to miss one of those tests.
Trying again (in a few minutes...).
comment:161 Changed 3 years ago by
Commit:  c247f13fd6e6ae3d3feeef41ebba1a78346744da → 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 

Branch pushed to git repo; I updated commit sha1. New commits:
6c5dafc  Silence another flaky doctest

comment:162 Changed 3 years ago by
comment:163 Changed 3 years ago by
Reviewers:  → John Palmieri 

Status:  needs_review → positive_review 
All tests passed!
comment:164 Changed 3 years ago by
Milestone:  sage8.9 → sage9.0 

comment:165 Changed 3 years ago by
Branch:  u/SimonKing/p_group_cohomology_in_py_3 → 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 

Resolution:  → fixed 
Status:  positive_review → closed 
comment:166 Changed 3 years ago by
Commit:  6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 

sdh_X
scripts call sdh_die
themselves, so doing
sdh_X  sdh_die ...
is equivalent to simply sdh_X
.
The intention of calling sdh_die
to produce a meaninful error message is commendable, but these after 
just won't be called.
It's better to modify sdh_X
to allow for custiom error messages, I suppose.
cmp
is also not supported in Python 3, and for a dictionaryd
, you should dox in d
rather thand.has_key(x)
.